76 lines
4.9 KiB
Markdown
76 lines
4.9 KiB
Markdown
tags:: [[#zotero]]
|
|
date:: 2024
|
|
title:: @A Model Is Not Built By A Single Prompt: LLM-Based Conceptual Modeling With Question Decomposition
|
|
item-type:: [[journalArticle]]
|
|
original-title:: A Model Is Not Built By A Single Prompt: LLM-Based Conceptual Modeling With Question Decomposition
|
|
language:: en
|
|
authors:: [[Anonymous Author]]
|
|
library-catalog:: Zotero
|
|
links:: [Local library](zotero://select/library/items/CJQAHLZ5), [Web library](https://www.zotero.org/users/1039502/items/CJQAHLZ5)
|
|
|
|
- [[Abstract]]
|
|
- Conceptual modeling plays a crucial role in model-driven engineering. When the system description is highly complicated or the modelers lack sufficient domain knowledge, the task can become particularly challenging. Large language models (LLMs) can facilitate the task by automatically generating an initial conceptual model from the system description. Nevertheless, model generation is more complex than code generation, and a single prompt to LLMs cannot solve the problem well. This paper proposes an LLMbased conceptual modeling approach via question decomposition. Following conventional modeling guidelines, we divide the model generation task into several sub-problems, i.e., class generation, association and aggregation generation, and inheritance generation. For each sub-problem, we carefully design the prompt by choosing more efficient query words and providing essential modeling knowledge to unlock the modeling potential of LLMs. Finally, we sum up the answers of all the sub-problems and programmatically create a conceptual model to avoid trivial syntactic model errors. We evaluate our approach with 10 systems from different application domains. The preliminary results show that our approach outperforms the singe-prompt-based baseline by improving recall values and F1 scores in most systems.
|
|
- ### Attachments
|
|
- [Author - 2024 - A Model Is Not Built By A Single Prompt LLM-Based Conceptual Modeling With Question Decomposition.pdf](zotero://select/library/items/7RYW5ZW3) {{zotero-imported-file 7RYW5ZW3, "Author - 2024 - A Model Is Not Built By A Single Prompt LLM-Based Conceptual Modeling With Question Decomposition.pdf"}}
|
|
- ### Notes
|
|
- # Annotazioni
|
|
(19/4/2024, 22:17:28)
|
|
|
|
- “Large language models (LLMs) can facilitate the task by automatically generating an initial conceptual model from the system description.” (Author, 2024, p. 1) #00b036
|
|
|
|
- “singe-prompt-” (Author, 2024, p. 1) #ff4400
|
|
|
|
- “Step3. Semantically check and remove the associations relationships···” (Author, 2024, p. 4) #f0ff00
|
|
*This is not clear.*
|
|
|
|
- “choose a default type.” (Author, 2024, p. 5) #f0ff00
|
|
*Can this be a bias or source of error?*
|
|
|
|
- “default multiplicity,” (Author, 2024, p. 5) #f0ff00
|
|
*See my previous comment.*
|
|
|
|
- “4” (Author, 2024, p. 6) #f0ff00
|
|
*Maybe it is too short to justify a separate section. You can merge it with previous section.*
|
|
|
|
- “with a 1-gram similarity higher than 0.9)” (Author, 2024, p. 6) #f0ff00
|
|
*What's the encoding that you have used?*
|
|
|
|
- “their types are equal,” (Author, 2024, p. 6) #f0ff00
|
|
*Can you also identify those that have been killed due to the default type?*
|
|
|
|
- “-shot version” (Author, 2024, p. 7) #f0ff00
|
|
*What's the difference with the baseline?*
|
|
|
|
- “For our approach, we choose temperature 0.6 for class/attribute generation and 0.3 for relationship generation (see Section 5.6 for the details about temperature selection).” (Author, 2024, p. 7) #f0ff00
|
|
*It is necessary to discuss the temperature definition*
|
|
|
|
- “manually examined the relationships generated by the baseline and strictly followed R3 to fill the” (Author, 2024, p. 7) #f0ff00
|
|
*This is a threat to validity. Check if it has been discussed.*
|
|
|
|
- “Table” (Author, 2024, p. 8) #f0ff00
|
|
*Please put in bold the highest values for each metric/approach pair.*
|
|
|
|
- “Answer.” (Author, 2024, p. 8) #f0ff00
|
|
|
|
- “two parallel tasks enhance” (Author, 2024, p. 8) #f0ff00
|
|
*Why parallel?*
|
|
|
|
- “baseline” (Author, 2024, p. 9) #f0ff00
|
|
*The baseline is your approach here, isn't it?*
|
|
|
|
- “temperature 0.3” (Author, 2024, p. 9) #f0ff00
|
|
*How is it defined?*
|
|
|
|
- “send the classes in Oracle models to our prompts. We” (Author, 2024, p. 9) #f0ff00
|
|
*What does it mean?*
|
|
|
|
- “11a” (Author, 2024, p. 9) #f0ff00
|
|
|
|
- “we ignore the internal validity.” (Author, 2024, p. 10) #f0ff00
|
|
*I'm not sure this makes sense.*
|
|
|
|
- “refers to the degree to which the metrics used in a study measures the performance of our approach.” (Author, 2024, p. 10) #f0ff00
|
|
*Not sure*
|
|
|
|
- “Conclusion validity refers to the reliability and accuracy of the conclusions drawn from a study. To avoid a threat to conclusion validity, all results and answers to the research questions in our evaluation were thoroughly discussed until the authors reached an agreement.” (Author, 2024, p. 10) #f0ff00
|
|
*Not sure* |