logseq/pages/@Multi-step Iterative Automated Domain Modeling with Large Language Models.md

tags:: #zotero date:: 2018 title:: @Multi-step Iterative Automated Domain Modeling with Large Language Models item-type:: journalArticle original-title:: Multi-step Iterative Automated Domain Modeling with Large Language Models language:: en authors:: Yujing Yang, Boqi Chen, Kua Chen, Gunter Mussbacher, Dániel Varró library-catalog:: Zotero links:: Local library, Web library

• Abstract
  • Domain modeling, which represents the concepts and relationships in a problem domain, is an essential part of software engineering. As large language models (LLMs) have recently exhibited remarkable ability in language understanding and generation, many approaches are designed to automate domain modeling with LLMs. However, these approaches usually formulate all input information to the LLM in a single step, resulting in many missing modeling elements and advanced patterns. This paper presents a novel framework designed to improve the performance of fully automated domain model generation leveraging LLMs. The multi-step automated domain modeling approach extracts model elements (e.g., classes, attributes, and relationships) from problem descriptions. Improving from earlier work, the proposed approach includes instructions and human knowledge in each step and uses an iterative process to identify complex patterns, repeatedly extracting the pattern from various instances and then synthesizing these extractions into a summarized overview. Furthermore, the framework incorporates a self-reflection mechanism. This mechanism assesses each generated model element, offering feedback for necessary modifications or removals, and integrates the domain model with the generated feedback. The proposed approach is assessed in experiments, comparing it with a baseline single-step approach. It improves the 𝐹1-score of identifying classes by 22.71% and relationships by 75.18%, while the 𝐹1-score is similar for attributes. Furthermore, it improves the 𝐹1-score by 10.39% for identifying complex design patterns. Our approach, dataset, and evaluation provide valuable insight for future research in automated LLM-based domain modeling.

• Attachments

  • File PDF {{zotero-imported-file M5WTEFPX, "Yang et al. - 2018 - Multi-step Iterative Automated Domain Modeling with Large Language Models.pdf"}}

• Notes

  • Annotazioni

    (30/7/2024, 11:52:09)

    • “Multi-step Iterative Automated Domain Modeling with Large Language Models” (Yang et al., 2018, p. 1) #5fb236

  - “language understanding and generation, many approaches are designed to automate domain modeling with LLMs” (Yang et al., 2018, p. 1) #5fb236
  * *
  
   
  
  - “This paper presents a novel framework designed to improve the performance of fully automated domain model generation leveraging LLMs” (Yang et al., 2018, p. 1) #5fb236
  * *
  
   
  
  - “proposed approach includes instructions and human knowledge in each step and uses an iterative process to identify complex patterns” (Yang et al., 2018, p. 1) #5fb236
  * *
  
   
  
  - “self-reflection mechanism” (Yang et al., 2018, p. 1) #a28ae5
  * *
  
   
  
  - “integrates the domain model with the generated feedback.” (Yang et al., 2018, p. 1) #5fb236
  * *
  
   
  
  - “baseline single-step approach” (Yang et al., 2018, p. 1) #a28ae5
  * *
  
   
  
  - “Recent advances in large language models (LLMs) have shown remarkable generalizability to tasks beyond natural language processing” (Yang et al., 2018, p. 1) #5fb236
  * *
  
   
  
  - “With different prompt designs, LLMs can achieve impressive performance on various tasks by only using a few labeled examples in the prompt.” (Yang et al., 2018, p. 1) #a28ae5
  * *
  
   
  
  - “Previous work [7] f” (Yang et al., 2018, p. 1) #5fb236
  * *
  
   
  
  - “the recall of identifying classes, attributes, and relationships is, in general, much lower than the precision” (Yang et al., 2018, p. 1) #a28ae5
  * *
  
   
  
  - “o modeling patterns (that capture modeling best practices) have been identified i” (Yang et al., 2018, p. 1) #a28ae5
  * *
  
   
  
  - “task into more simple sub-tasks and including human knowledge to solve these sub-tasks” (Yang et al., 2018, p. 2) #a28ae5
  * *
  
   
  
  - “a feedback mechanism can be included in automated domain modeling to further enhance the quality of output models” (Yang et al., 2018, p. 2) #ffd400
  *Kind or reinforcement learning? *
  
   
  
  - “novel approach for fully automated domain modeling using LLMs” (Yang et al., 2018, p. 2) #ffd400
  *Fully automated, thus no human intervention is required??? *
  
   
  
  - “automated domain modeling approach” (Yang et al., 2018, p. 2) #5fb236
  * *
  
   
  
  - “iterative process to identify best-practice patterns” (Yang et al., 2018, p. 2) #5fb236
  * *
  
   
  
  - “self-reflection mechanism to generate internal feedback according to human knowledge and then integrate the generated feedback into the domain model for improvement” (Yang et al., 2018, p. 2) #a28ae5
  * *
  
   
  
  - “text generation problem, where a generative function f directly converts a domain specification d into a domain model M” (Yang et al., 2018, p. 2) #5fb236
  * *
  
   
  
  - “player-role pattern [11] and abstraction-occurrence pattern” (Yang et al., 2018, p. 2) #2ea8e5
  * *
  
   
  
  - “player-role pattern because it is one of the most used patterns while LLMs perform poorly in identifying it.” (Yang et al., 2018, p. 2) #ffd400
  *Can you give a supporting reference for this statement? *
  
   
  
  - “This method involves a single prompt, including a problem description, task description, and output format description, to generate the domain model with an LLM.” (Yang et al., 2018, p. 2) #5fb236
  * *
  
   
  
  - “this approach often misses elements in the domain model (low recall)” (Yang et al., 2018, p. 2) #5fb236
  * *
  
   
  
  - “achieves the functionality” (Yang et al., 2018, p. 2) #ffd400
  *What does it mean in general? Human in the loop is required, right? *
  
   
  
  - “if x is in exact match or semantically equivalent” (Yang et al., 2018, p. 2) #ffd400
  *I would have distinguished these. Indeed exact match is also semantically equivalent, but not the other way around. *
  
   
  
  - “standard statistical metrics of precision, recall, and F1-score over the classes, relationships, and attributes” (Yang et al., 2018, p. 2) #5fb236
  * *
  
   
  
  - “multi-step automated domain modeling” (Yang et al., 2018, p. 3) #5fb236
  * *
  
   
  
  - “task description to describe the overall domain model generation task” (Yang et al., 2018, p. 3) #a28ae5
  * *
  
   
  
  - “modeling problem description written in natural language to describe a problem domain” (Yang et al., 2018, p. 3) #a28ae5
  * *
  
   
  
  - “example store that contains few-shot examples explaining the definition of model elements and patterns” (Yang et al., 2018, p. 3) #a28ae5
  * *
  
   
  
  - “partial model” (Yang et al., 2018, p. 3) #ffd400
  *How are the different partial models linked together? Or is it a monothonic approach where models can be only incrementally increased? (from each iteration, elements can be only added to the domain model being created), *
  
   
  
  - “sequence of these outputs can be combined” (Yang et al., 2018, p. 3) #ffd400
  *Such a combination operation requires details. *
  
   
  
  - “iterative” (Yang et al., 2018, p. 3) #ffd400
  *See my previous comments. It's not clear if domain models can be only enlarged? *
  
   
  
  - “The post-processor uses rule-based methods to parse and extract relevant results from the LLM’s response to create the final domain model.” (Yang et al., 2018, p. 3) #ffd400
  *Does the LLM's response need to adhere some syntax to facilitate post-processing / parsing? *
  
   
  
  - “evaluation” (Yang et al., 2018, p. 3) #ffd400
  *So models are not compared directly with the ground truth, but instead the are indirectly compared by considering their precision, recall, and F1-score. *
  
   
  
  - “single-step process into smaller, pre-defined subprocesses.” (Yang et al., 2018, p. 3) #5fb236
  * *
  
   
  
  - “human instructions and examples can be included in the individual process to improve the performance of identifying modeling elements” (Yang et al., 2018, p. 3) #ffd400
  *When such instructions are given? *
  
   
  
  - “k times” (Yang et al., 2018, p. 4) #ffd400
  *How is k identified? *
  
   
  
  - “non-zero temperature” (Yang et al., 2018, p. 4) #ffd400
  *Same here, how is temperature value determined? *
  
   
  
  - “integrates the player-role pattern into the previous partial model from Step 1 with the problem description and the extracted patterns” (Yang et al., 2018, p. 4) #ffd400
  *How is this integration done? *
  
   
  
  - “Self-reflection” (Yang et al., 2018, p. 5) #5fb236
  * *
  
   
  
  - “generate feedback for the partial domain model” (Yang et al., 2018, p. 5) #ffd400
  * *
  
   
  
  - “improve the partial domain model according to the feedback” (Yang et al., 2018, p. 5) #ffd400
  *How is this feedback given and automatically processed? *
  
   
  
  - “self-refinement [13] and reflexion [18]” (Yang et al., 2018, p. 5) #a28ae5
  * *
  
   
  
  - “good understanding of where the model made mistakes as well as the ability to generate a summary containing actionable insights for improvement” (Yang et al., 2018, p. 5) #ffd400
  *So, human intervention seems to be required here. If this is the case, it is need to smooth the "automated" aspect of the approach (even in the title) *
  
   
  
  - “problem description” (Yang et al., 2018, p. 5) #5fb236
  * *
  
   
  
  - “partial model revised from previous step” (Yang et al., 2018, p. 5) #5fb236
  * *
  
   
  
  - “One example of the generated feedback for the class Dimension (String length) is shown below: It is unnecessary as the dimension can be an attribute of the Item class.” (Yang et al., 2018, p. 5) #ffd400
  *It is not clear how feedback is generated automatically! *
  
   
  
  - “Integrate feedback into the partial model.” (Yang et al., 2018, p. 5) #5fb236
  * *
  
   
  
  - “feedback for each class as input.” (Yang et al., 2018, p. 5) #ffd400
  *It is not clear how this gets generated for each class. It should be also for structural features (individually and not induced by class feedback), isn't it? *
  
   
  
  - “Table 1: Comparison of precision, recall, and F1-score for single-step and MIG approaches for each model element type” (Yang et al., 2018, p. 6) #ffd400
  *What are the values that are used for k and temperature? *
  
   
  
  - “we carry out a more fine-grained analysis by highlighting which modeling aspects the LLMs struggle with.” (Yang et al., 2018, p. 6) #5fb236
  * *
  
   
  
  - “For the single-step approach, the precision is higher than the recall for almost all model elements, indicating that the identified elements are accurate, but there are many missing elements.” (Yang et al., 2018, p. 6) #ffd400
  *SO this means that it is not a clear win, isn't it? *
  
   
  
  - “This indicates that our generated model covers many more elements in the solution model.” (Yang et al., 2018, p. 6) #ffd400
  *yes, but with reduced precision... *
  
   
  
  - “Table 2: Precision, recall, and F1-score of three approaches of identifying player-role patterns: zero-shot single-step, twoshot single-step, and multi-step” (Yang et al., 2018, p. 6) #ffd400
  *Same comment for the previous Table. *
  
   
  
  - “This RQ shows that our proposed approach overall improves the performance of automating domain modeling compared to the single-step approach” (Yang et al., 2018, p. 6) #ffd400
  *It's not clear like this. *
  
   
  
  - “In the research question, we further investigate the performance of identifying the playerrole pattern and compare it with the single-step approach.” (Yang et al., 2018, p. 6) #5fb236
  * *
  
   
  
  - “One of the potential future directions is that we can design components to identify other advanced patterns, for example, the abstraction-occurrence pattern.” (Yang et al., 2018, p. 7) #ffd400
  *It's not clear how the identified patterns are checked if they are correct or not. I think humans should be involved here this it not mentioned in the paper. *
  
   
  
  - “Furthermore, we address this variation by experimenting with a set of eight diverse domain examples.” (Yang et al., 2018, p. 7) #5fb236
  * *
  
   
  
  - “without any human interaction or supervised training” (Yang et al., 2018, p. 8) #ffd400
  *This is not clear in the paper. *
  
   
  
  - “generate and integrate feedback without any external interaction.” (Yang et al., 2018, p. 8) #ffd400
  *Can be the case that feedback given in not correct or accurate? *