type:: [[REVIEWS]] tags:: year:: 2024 venue:: [[MODELS]] full-title:: A Model Is Not Built By A Single Prompt: LLM-Based Conceptual Modeling With Question Decomposition date-start:: [[22-04-2024]] - 10:32 date-submitted:: [[23-04-2024]] external-links:: [Reviews and Comments on Submission 9204 (easychair.org)](https://easychair.org/conferences/submission_reviews?submission=6849996&a=32229753#{fr:VlTkxRpsq92c}) status:: [[DONE]] deadline-submission:: [[22-04-2024]] file:: [[@A Model Is Not Built By A Single Prompt: LLM-Based Conceptual Modeling With Question Decomposition]] parent:: todoist:: https://app.todoist.com/app/task/9204-a-model-is-not-built-by-a-single-prompt-llm-based-conceptual-modeling-with-7858701468 - ### [[Highlights]] - # Annotazioni collapsed:: true - (19/4/2024, 22:17:28) - - “Large language models (LLMs) can facilitate the task by automatically generating an initial conceptual model from the system description.” (Author, 2024, p. 1) #00b036 - - “singe-prompt-” (Author, 2024, p. 1) #ff4400 - - “Step3. Semantically check and remove the associations relationships···” (Author, 2024, p. 4) #f0ff00 - _This is not clear._ - - “choose a default type.” (Author, 2024, p. 5) #f0ff00 - _Can this be a bias or source of error?_ - - “default multiplicity,” (Author, 2024, p. 5) #f0ff00 - _See my previous comment._ - - “4” (Author, 2024, p. 6) #f0ff00 - _Maybe it is too short to justify a separate section. You can merge it with previous section._ - - “with a 1-gram similarity higher than 0.9)” (Author, 2024, p. 6) #f0ff00 - _What's the encoding that you have used?_ - - “their types are equal,” (Author, 2024, p. 6) #f0ff00 - _Can you also identify those that have been killed due to the default type?_ - - “-shot version” (Author, 2024, p. 7) #f0ff00 - _What's the difference with the baseline?_ - - “For our approach, we choose temperature 0.6 for class/attribute generation and 0.3 for relationship generation (see Section 5.6 for the details about temperature selection).” (Author, 2024, p. 7) #f0ff00 - _It is necessary to discuss the temperature definition_ - - “manually examined the relationships generated by the baseline and strictly followed R3 to fill the” (Author, 2024, p. 7) #f0ff00 - _This is a threat to validity. Check if it has been discussed._ - - “Table” (Author, 2024, p. 8) #f0ff00 - _Please put in bold the highest values for each metric/approach pair._ - - “Answer.” (Author, 2024, p. 8) #f0ff00 - - “two parallel tasks enhance” (Author, 2024, p. 8) #f0ff00 - _Why parallel?_ - - “baseline” (Author, 2024, p. 9) #f0ff00 - _The baseline is your approach here, isn't it?_ - - “temperature 0.3” (Author, 2024, p. 9) #f0ff00 - _How is it defined?_ - - “send the classes in Oracle models to our prompts. We” (Author, 2024, p. 9) #f0ff00 - _What does it mean?_ - - “11a” (Author, 2024, p. 9) #f0ff00 - - “we ignore the internal validity.” (Author, 2024, p. 10) #f0ff00 - _I'm not sure this makes sense._ - - “refers to the degree to which the metrics used in a study measures the performance of our approach.” (Author, 2024, p. 10) #f0ff00 - _Not sure_ - - “Conclusion validity refers to the reliability and accuracy of the conclusions drawn from a study. To avoid a threat to conclusion validity, all results and answers to the research questions in our evaluation were thoroughly discussed until the authors reached an agreement.” (Author, 2024, p. 10) #f0ff00 - _Not sure_ - ### [[Comments]] - #.tabular - Summary of the paper - This paper explores the use of LLMs to automatically generate domain models from textual descriptions. It introduces a decomposition approach to divide the model generation process into multiple steps: first generating classes, then associations and aggregations, and finally inheritance relations. Each step utilizes specific prompts to facilitate targeted model creation. The approach is evaluated against a baseline technique. The paper presents the adoption of LLMs to automatically create domain models starting from a textual description of the system at hand. The paper proposes the adoption of a decomposition approach with the aim of creating the target conceptual model by means of different steps and formulating the generation task into several sub-problems. First classes are generated, then association and aggregations are produced, finally inheritance relations are generation. For each step, precise prompts are proposed. The proposed approach is compared with a baseline technique. - _Summary of main points for/against the paper_ - - + Interesting problem - + Comparison of the proposed approach with a baseline - - Presentation issues affecting the quality of the whole paper - - The experiments are not always easy to read - Detailed evaluation - The paper is about an interesting problem. Investigating the usage of LLMs to support modelling tasks is indeed relevant. I have some minor concerns about the presentation of the paper that are related to the following issues: - - Page 4: In Fig. 3.a, how are semantic checks performed in Step 3, and why do associations need to be removed? - - Page 5: The rationale behind choosing default types during model creation should be addressed more thoroughly to avoid bias or errors. Similarly, the justification for the default multiplicity needs a more precise explanation. - - Page 6: The encoding method used to support a 1-gram similarity needs to be clarified. A detailed explanation of the encoding choices and their implications is also needed. Moreover, the authors should specify whether the introduction of default types and cardinalities can influence the pairing process of model elements as described by the rules given in Sec. 5.3. - - Page 6: Section 4 is too short; I suggest merging it with the previous section. - - Page 6: When comparing generated models with those in the Oracle, why not use existing model differencing techniques and tools? - - Page 8: To interpret the data more clearly, emphasize the highest values in Table 2 using bold text. This will help the reader quickly identify significant results. - - The presentation of the experiments is not always clear, as detailed below: - -- Experiment 1: When presenting the process, the authors state: "We choose the approach proposed by Chen et al. [3] as the baseline in this experiment. For a fair comparison, we only try their 0-shot version because our approach uses 0-shot prompts." The main point of the proposed approach is on the decomposition of the system description in terms of a sequence of different prompts, each devoted to the generation of various parts of the wanted model. How can the decomposition idea be consistent with the "0-shot prompts" concept? - -- Experiment 3: RQ3—The rationale behind choosing parallel tasks should be explained, particularly how they enhance the study's outcomes. What tasks have been parallelized, and how? - -- Experiment 4: The authors state "To evaluate relationship generation, we send the classes in Oracle models to our prompts. We compare the results with Oracle models, and calculate the average F1 scores for each system under each temperature." It is not clear what the authors mean by sending Oracle models to prompts and, thus, the subsequent phase of the process. - -- Moreover, when referring to the concept of "baseline", the reader can get confused because sometimes it seems that authors refer to the approach presented by Chen et al. [3]. In other cases, it appears that the term "baseline" refers to particular "configurations" of the approach proposed by the authors. This inconsistency complicates the reading. - The motivation for why authors have ignored the internal validity of the performed experiments is neither clear nor convincing. As I mentioned previously, another threat to validity to discuss is the usage of default types when generating model elements. - _Questions to authors (rebuttal phase)_ - Can you clarify why you have decided to develop a specific process for comparing generated models with those in the Oracle, rather then using existing techniques and tools for model comparison/similarity? - What can you say about the generalizability of the discussed prompt decompositions? - ### [[REVIEWS/Notes]] - ### Clarity and Justification - 1. **Semantically Checking Associations** (p. 4) - The process for checking and removing associations is not clearly explained. Clarification on what constitutes an association relationship and how they are evaluated would improve understanding. - background-color:: green 2. **Choice of Default Type and Multiplicity** (p. 5) - The rationale behind choosing a default type should be addressed more thoroughly to avoid potential bias or errors. How the default type might affect results should be explored. background-color:: green - Similarly, the justification for the default multiplicity needs clearer explanation, especially considering previous concerns about biases introduced by defaults. background-color:: green - 3. **Encoding and Model Types** (p. 6) - It's unclear what encoding method was used for achieving a 1-gram similarity higher than 0.9. A detailed explanation of the encoding choices and their implications is needed. background-color:: green - The document should specify if the default type has influenced the elimination of certain model elements. background-color:: green - 4. **Section Length and Structure** (p. 6) - Consider merging shorter sections (e.g., section "4") with adjacent ones if they provide insufficient standalone content. background-color:: green - ### Methodological Concerns - 5. **Temperature Settings for Generation Tasks** (p. 7) - The choice of temperature values for class/attribute and relationship generation lacks a detailed explanation. Discuss the impact of these values on the results and why these specific values were chosen. background-color:: yellow - 6. **Validation and Bias in Generated Relationships** (p. 7) - The manual examination of relationships and adherence to Rule 3 (R3) may pose a threat to validity. Ensure that any potential biases introduced by this method are discussed. - ### Presentation and Data Reporting - 7. **Data Representation** (p. 8) - For clearer data interpretation, emphasize the highest values in tables by using bold text. This will help in quickly identifying significant results. background-color:: green - 8. **Parallel Task Execution** (p. 8) - The rationale behind choosing parallel tasks should be explained, particularly how they enhance the study's outcomes. background-color:: green - ### Technical Descriptions and Definitions - 9. **Clarification of Technical Terms and Processes** (p. 9-10) - Terms like "baseline" and "temperature 0.3" need precise definitions to ensure they are understood in the context of the study. - The method of integrating classes from Oracle models into the prompts requires further explanation to avoid confusion. - ### Validity and Conclusion Accuracy - 10. **Internal and Conclusion Validity** (p. 10) - The statement about ignoring internal validity is concerning. Discuss the implications of this choice and how it might affect the study's integrity. - Enhance the explanation of conclusion validity to illustrate how the results' reliability and accuracy are ensured throughout the study. - ### General Suggestions - **Consistent Review and Feedback:** Ensure that each section and result discussed has been critically reviewed and agreed upon by all authors to maintain a high standard of research integrity. background-color:: yellow - **Enhanced Justification and Rationalization:** Wherever choices are made, particularly in methodological approaches, provide comprehensive justifications to support these decisions, thus strengthening the research framework. background-color:: yellow - ### YELLOW CONCERNS background-color:: yellow collapsed:: true - {{query (and [[f0ff00]] [[models-2024-9204]] )}} query-table:: true - ### ❓️Questions - {{query (and [[question]] [[models-2024-9204]] )[[question]]}} query-table:: true query-properties:: [:block]