logseq/pages/@A Model Is Not Built By A Single Prompt%3A LLM-Based Conceptual Modeling With Question Decomposition.md

tags:: #zotero date:: 2024 title:: @A Model Is Not Built By A Single Prompt: LLM-Based Conceptual Modeling With Question Decomposition item-type:: journalArticle original-title:: A Model Is Not Built By A Single Prompt: LLM-Based Conceptual Modeling With Question Decomposition language:: en authors:: Anonymous Author library-catalog:: Zotero links:: Local library, Web library

• Abstract
  • Conceptual modeling plays a crucial role in model-driven engineering. When the system description is highly complicated or the modelers lack sufficient domain knowledge, the task can become particularly challenging. Large language models (LLMs) can facilitate the task by automatically generating an initial conceptual model from the system description. Nevertheless, model generation is more complex than code generation, and a single prompt to LLMs cannot solve the problem well. This paper proposes an LLMbased conceptual modeling approach via question decomposition. Following conventional modeling guidelines, we divide the model generation task into several sub-problems, i.e., class generation, association and aggregation generation, and inheritance generation. For each sub-problem, we carefully design the prompt by choosing more efficient query words and providing essential modeling knowledge to unlock the modeling potential of LLMs. Finally, we sum up the answers of all the sub-problems and programmatically create a conceptual model to avoid trivial syntactic model errors. We evaluate our approach with 10 systems from different application domains. The preliminary results show that our approach outperforms the singe-prompt-based baseline by improving recall values and F1 scores in most systems.

• Attachments

  • Author - 2024 - A Model Is Not Built By A Single Prompt LLM-Based Conceptual Modeling With Question Decomposition.pdf {{zotero-imported-file 7RYW5ZW3, "Author - 2024 - A Model Is Not Built By A Single Prompt LLM-Based Conceptual Modeling With Question Decomposition.pdf"}}

• Notes

  • Annotazioni

    (19/4/2024, 22:17:28)

    • “Large language models (LLMs) can facilitate the task by automatically generating an initial conceptual model from the system description.” (Author, 2024, p. 1) #00b036

    • “singe-prompt-” (Author, 2024, p. 1) #ff4400

    • “Step3. Semantically check and remove the associations relationships···” (Author, 2024, p. 4) #f0ff00 This is not clear.

    • “choose a default type.” (Author, 2024, p. 5) #f0ff00 Can this be a bias or source of error?

    • “default multiplicity,” (Author, 2024, p. 5) #f0ff00 See my previous comment.

    • “4” (Author, 2024, p. 6) #f0ff00 Maybe it is too short to justify a separate section. You can merge it with previous section.

    • “with a 1-gram similarity higher than 0.9)” (Author, 2024, p. 6) #f0ff00 What's the encoding that you have used?

    • “their types are equal,” (Author, 2024, p. 6) #f0ff00 Can you also identify those that have been killed due to the default type?

    • “-shot version” (Author, 2024, p. 7) #f0ff00 What's the difference with the baseline?

    • “For our approach, we choose temperature 0.6 for class/attribute generation and 0.3 for relationship generation (see Section 5.6 for the details about temperature selection).” (Author, 2024, p. 7) #f0ff00 It is necessary to discuss the temperature definition

    • “manually examined the relationships generated by the baseline and strictly followed R3 to fill the” (Author, 2024, p. 7) #f0ff00 This is a threat to validity. Check if it has been discussed.

    • “Table” (Author, 2024, p. 8) #f0ff00 Please put in bold the highest values for each metric/approach pair.

    • “Answer.” (Author, 2024, p. 8) #f0ff00

    • “two parallel tasks enhance” (Author, 2024, p. 8) #f0ff00 Why parallel?

    • “baseline” (Author, 2024, p. 9) #f0ff00 The baseline is your approach here, isn't it?

    • “temperature 0.3” (Author, 2024, p. 9) #f0ff00 How is it defined?

    • “send the classes in Oracle models to our prompts. We” (Author, 2024, p. 9) #f0ff00 What does it mean?

    • “11a” (Author, 2024, p. 9) #f0ff00

    • “we ignore the internal validity.” (Author, 2024, p. 10) #f0ff00 I'm not sure this makes sense.

    • “refers to the degree to which the metrics used in a study measures the performance of our approach.” (Author, 2024, p. 10) #f0ff00 Not sure

    • “Conclusion validity refers to the reliability and accuracy of the conclusions drawn from a study. To avoid a threat to conclusion validity, all results and answers to the research questions in our evaluation were thoroughly discussed until the authors reached an agreement.” (Author, 2024, p. 10) #f0ff00 Not sure