210 lines
17 KiB
Markdown
210 lines
17 KiB
Markdown
tags:: [[#zotero]]
|
||
title:: @icse2025-paper1115
|
||
item-type:: [[document]]
|
||
original-title:: icse2025-paper1115
|
||
links:: [Local library](zotero://select/library/items/AC9SS9YP), [Web library](https://www.zotero.org/users/1039502/items/AC9SS9YP)
|
||
|
||
- ### Attachments
|
||
- [PDF](zotero://select/library/items/I9R3EFI5) {{zotero-imported-file I9R3EFI5, "icse2025-paper1115.pdf"}}
|
||
- ### Notes
|
||
- # Annotazioni
|
||
(30/9/2024, 22:59:45)
|
||
|
||
- “generate accurate and 11 comprehensive UML models described through the PlantUML 12 syntax.” (“icse2025-paper1115”, p. 1) #5fb236
|
||
|
||
- “accuracy, completeness, and ease 18 of understanding” (“icse2025-paper1115”, p. 1) #5fb236
|
||
|
||
- “31 for capturing system architectures. However, manual creation 32 of these models can be time-consuming and error-prone, 33” (“icse2025-paper1115”, p. 1) #00b036
|
||
|
||
- “LLMs have witnessed a meteoric rise in 40 popularity and utility across various domains” (“icse2025-paper1115”, p. 1) #5fb236
|
||
|
||
- “41 characterized by their ability to comprehend and generate 42 human-like contents, have demonstrated remarkable efficacy 43” (“icse2025-paper1115”, p. 1) #00b036
|
||
|
||
- “in creating artifacts from scratch, ranging from articles and 44 stories” (“icse2025-paper1115”, p. 1) #00b036
|
||
|
||
- “The goal of is paper is to perform a comprehensive 49 investigation into the ability of LLMs to automatically 50 generate UML class diagrams that capture informal,” (“icse2025-paper1115”, p. 1) #00b036
|
||
|
||
- “descriptions” (“icse2025-paper1115”, p. 1) #f0ff00
|
||
*If we have more at different levels of abstraction, how to link them?*
|
||
|
||
- “developers to harness the power of LLMs 69 for automating UML diagram generation tasks.” (“icse2025-paper1115”, p. 1) #00b036
|
||
|
||
- “The idea of automatically generating models and other 93 software engineering artefacts from descriptions expressed in 94 natural language has been around for years before the advent 95 of LLMs.” (“icse2025-paper1115”, p. 1) #5fb236
|
||
|
||
- “ChatGPT struggled with accurately representing complex 138 diagrams involving classes and advanced UML concepts such 139 as association classes and multiple inheritance.” (“icse2025-paper1115”, p. 2) #00b036
|
||
|
||
- “into an educational context where they demonstrated that 144 utilizing ChatGPT in software modeling courses achieves 145 improved outcomes in formative assessments compared to 146 those using traditional methods.” (“icse2025-paper1115”, p. 2) #00b036
|
||
|
||
- “Hence, we decided to focus our 178 attention on the presence or absence of the most important 179 elements in a class diagram: classes, attributes, associations, 180 cardinalities, operations.” (“icse2025-paper1115”, p. 2) #5fb236
|
||
|
||
- “The ultimate goal of this research is to evaluate the ability 187 of LLMss to automatically generate UML models described 188 through class diagrams and to compare their performance 189 based on the quality of the obtained results.” (“icse2025-paper1115”, p. 2) #00b036
|
||
|
||
- “The ultimate goal of this research is to evaluate the ability 187 of LLMss to automatically generate UML models described 188 through class diagrams and to compare their performance 189 based on the quality of the obtained results.” (“icse2025-paper1115”, p. 2) #a28ae5
|
||
|
||
- “The textual descriptions were 193 originally created as exercises to be developed by students of 194 a Software Engineering class to evaluate their proficiency in 195 UML modeling.” (“icse2025-paper1115”, p. 2) #5fb236
|
||
|
||
- “We used the textual descriptions 200 as basis to prompt LLMs and the corresponding UML models 201 as ground truth against which to compare the models generated 202 by the LLMs.1” (“icse2025-paper1115”, p. 2) #a28ae5
|
||
|
||
- “(i) Experiment Preparation” (“icse2025-paper1115”, p. 3) #5fb236
|
||
|
||
- “(ii) Experiment 208 Execution” (“icse2025-paper1115”, p. 3) #5fb236
|
||
|
||
- “(iii) Data Analysis” (“icse2025-paper1115”, p. 3) #5fb236
|
||
|
||
- “1) LLM selection:” (“icse2025-paper1115”, p. 3) #2ea8e5
|
||
|
||
- “ChatGPT-4 and ChatGPT-3.5 have demonstrated excep232 tional performance across a wide range of NLP tasks, in233 cluding text generation, translation, and summarization” (“icse2025-paper1115”, p. 3) #5fb236
|
||
|
||
- “excep232” (“icse2025-paper1115”, p. 3) #00b036
|
||
|
||
- “a deep understanding of” (“icse2025-paper1115”, p. 3) #f0ff00
|
||
*I have some doults.*
|
||
|
||
- “semantic nuances,” (“icse2025-paper1115”, p. 3) #f0ff00
|
||
*Are we sure?*
|
||
|
||
- “LLama 3 has been fine-tuned on 250 diverse datasets, making it adept at handling a wide range 251 of inputs and producing coherent and contextually relevant 252 outputs [21]” (“icse2025-paper1115”, p. 3) #5fb236
|
||
|
||
- “Its effi- 255 ciency and capability in capturing structural details make it an 256 important LLM to evaluate in our study” (“icse2025-paper1115”, p. 3) #5fb236
|
||
|
||
- “Through this comparative study, we sought to determine 258 whether each LLM shows specific characteristics in the gen- 259 eration of UML class diagrams and whether it is possible to 260 identify one that outperforms the others.” (“icse2025-paper1115”, p. 3) #a28ae5
|
||
|
||
- “2) Preparation of descriptions and prompt engineering” (“icse2025-paper1115”, p. 3) #2ea8e5
|
||
|
||
- “he dataset we used in this study consists of UML class 263 diagrams generated from informal descriptions for which an 264 expert-provided corresponding UML model was available” (“icse2025-paper1115”, p. 3) #5fb236
|
||
|
||
- “identifying and 266 collecting the informal descriptions” (“icse2025-paper1115”, p. 3) #e56eee
|
||
|
||
- “generating the dataset 267 by feeding the informal descriptions, though a suitable prompt, 268 to the selected LLMs.” (“icse2025-paper1115”, p. 3) #e56eee
|
||
|
||
- “a suitable” (“icse2025-paper1115”, p. 3) #f0ff00
|
||
*Check if suitability has been defined in the context of this work.*
|
||
|
||
- “they described realistic—if not real—applications” (“icse2025-paper1115”, p. 3) #ffd400
|
||
*What does it mean in practice?*
|
||
|
||
- “realistic—if not real—” (“icse2025-paper1115”, p. 3) #f0ff00
|
||
*What's the difference in the context of this work?*
|
||
|
||
- “Each exercise starts with a textual description of an 279 application, and asks students to model it through a UML class 280 diagram.” (“icse2025-paper1115”, p. 4) #a28ae5
|
||
|
||
- “The exercises were inspired by different sources, 285 including real-life applications available to the general public 286 and systems being developed at INSTITUTION.” (“icse2025-paper1115”, p. 4) #5fb236
|
||
|
||
- “the course instructor,” (“icse2025-paper1115”, p. 4) #f0ff00
|
||
*Only one? In my opinion this can be a bias related to the construction of the ground truth*
|
||
|
||
- “prompt 292 engineering, wherein tailored prompts were crafted to elicit 293 the most effective responses from each LLM for UML class 294 diagram generation from informal textual descriptions” (“icse2025-paper1115”, p. 4) #5fb236
|
||
|
||
- “trial-and-error process,” (“icse2025-paper1115”, p. 4) #2ea8e5
|
||
|
||
- “each LLM 296 was tested with different prompts to determine the optimal 297 phrasing that produced the best results.” (“icse2025-paper1115”, p. 4) #ffd400
|
||
*A notion of optimality is needed here.*
|
||
|
||
- “One key resource was 300 the work done by White et al. [22] that outlined 16 distinct 301 prompt patterns, offering valuable insights into various strate302 gies for crafting effective prompts.” (“icse2025-paper1115”, p. 4) #5fb236
|
||
|
||
- “White et al. [22] that outlined 16 distinct 301 prompt patterns,” (“icse2025-paper1115”, p. 4) #f0ff00
|
||
*To be downloaded.*
|
||
|
||
- “We tested prospective prompts 306 on 2-3 examples of textual descriptions, until we settled on a 307 final formulation.” (“icse2025-paper1115”, p. 4) #5fb236
|
||
|
||
- “We tested prospective prompts 306 on 2-3 examples of textual descriptions, until we settled on a 307 final formulation.” (“icse2025-paper1115”, p. 4) #f0ff00
|
||
*Which LLM ha been used in this process of creating the "best" prompt formulation? Seem next comment, this phase an represent a potential bias.*
|
||
|
||
- “We initially used prompts that were more 308 abstract and general, and that did not include any specific 309 keywords.” (“icse2025-paper1115”, p. 4) #5fb236
|
||
|
||
- “these initial versions did not yield sat310 isfactory results, hence, we refined the prompt by introducing 311 specific keywords, until we arrived at our desired version.” (“icse2025-paper1115”, p. 4) #5fb236
|
||
|
||
- “Specifying 317 “PlantUML” directed the LLMs to produce output in a specific, 318 recognized format, which enhanced the clarity and usability 319 of the generated diagrams” (“icse2025-paper1115”, p. 4) #5fb236
|
||
|
||
- “question goes here” (“icse2025-paper1115”, p. 4) #ffd400
|
||
*i guess also the question should be properly crafted, beyond the previous fragment of the prompt about the supposed role of the LLM.*
|
||
|
||
- “Fig. 2.” (“icse2025-paper1115”, p. 4) #f0ff00
|
||
*This is another bias because it is not sure that one prompt formulation that works best for a given LLM cn achieve the same accuracy for other LLMs. Potentially., the authors should have defined different best prompts, one for each LLM.*
|
||
|
||
- “different LLMs, the same 326 prompt” (“icse2025-paper1115”, p. 4) #f0ff00
|
||
|
||
- “the same 326 prompt formulation was used for each LLM” (“icse2025-paper1115”, p. 4) #ffd400
|
||
*So, all the LLMs have been used with the same prompt formulation? This can represent a bias, isn't it?*
|
||
|
||
- “3) Selecting the evaluation criteria:” (“icse2025-paper1115”, p. 4) #2ea8e5
|
||
|
||
- “there is no agreement in 332 the literature regarding the best approach to assess the quality 333 of UML models” (“icse2025-paper1115”, p. 4) #e56eee
|
||
|
||
- “we decided to base the evaluation of 334 LLMs’ results on the same criteria used to asses the solutions 335 of the students of the Software Engineering class from which 336 the textual descriptions originate” (“icse2025-paper1115”, p. 4) #5fb236
|
||
|
||
- “presence of all needed classe” (“icse2025-paper1115”, p. 4) #a28ae5
|
||
|
||
- “correct identification of 339 attributes and their assignment to classe” (“icse2025-paper1115”, p. 4) #a28ae5
|
||
|
||
- “orrect identification 340 of associations” (“icse2025-paper1115”, p. 4) #a28ae5
|
||
|
||
- “correct- 341 ness of association cardinalities” (“icse2025-paper1115”, p. 4) #2ea8e5
|
||
|
||
- “correct identification of 342 operations and their association with classes.” (“icse2025-paper1115”, p. 4) #5fb236
|
||
|
||
- “readability and 347 clarity” (“icse2025-paper1115”, p. 4) #ffd400
|
||
*how to measure them in an objective manner? Moreover, a given problem can be rightly modeled in different manners. How have you addressed such critical point for the evaluation?*
|
||
|
||
- “clarity of the PlantUML” (“icse2025-paper1115”, p. 4) #f0ff00
|
||
*How?*
|
||
|
||
- “deciphered” (“icse2025-paper1115”, p. 4) #f0ff00
|
||
*I'm not sure this is the correct item. Textual descriptions were no cyphred I guess.*
|
||
|
||
- “Model generation was performed for each 358 textual input, with the generated PlantUML code stored for 359 subsequent analysis.” (“icse2025-paper1115”, p. 4) #5fb236
|
||
|
||
- “360 Finally, the PlantUML code generated by the LLMs” (“icse2025-paper1115”, p. 4) #f0ff00
|
||
*Was this done straightaway or some post process of generated code was needed?*
|
||
|
||
- “Its functionality 365 facilitated the seamless conversion of textual representations 366 into concrete visual artifacts, thereby bridging the gap between 367 abstract textual descriptions and tangible UML diagrams” (“icse2025-paper1115”, p. 4) #5fb236
|
||
|
||
- “a dataset of 188 UML class dia- 369 grams” (“icse2025-paper1115”, p. 4) #5fb236
|
||
|
||
- “was produced across all LLMs. 371” (“icse2025-paper1115”, p. 4) #5fb236
|
||
|
||
- “assigning a score to each of the characteristics iden- 374 tified in the experiment preparation phase” (“icse2025-paper1115”, p. 4) #2ea8e5
|
||
|
||
- “were generated from 47 textual descriptions. The generated 379 graphical representations were manually evaluated taking into 380 account the expert-provided solutions, which served as our 381 “ground truth”. The comparison was done by expert software” (“icse2025-paper1115”, p. 4) #f0ff00
|
||
*Also in this phase, we have the same potential bias related to the ground truth creation. An agreement phase is needed both for the creation of diagrams in the ground truth and for evaluating/judging/score the diagrams generated by the considered LLMs.*
|
||
|
||
- “we considered and ranked also higher-level aspects, that is, the 389 ability of LLMs to show a good understanding of the domain, 390 the clarity and readability of the generated diagrams, and the 391 length of the generation process.” (“icse2025-paper1115”, p. 5) #ffd400
|
||
*TO CHECK THE METRICS THAT HAVE BEEN USED TO MEASURE AND RANK WRT SUCH ASPECTS.*
|
||
|
||
- “three. Cohen’s Kappa Score [23] was computed to evaluate 414 the level of inter-rater agreement and ensure the reliability of 415 the assessment process. The results of the whole evaluation 416 process are presented in the next section.” (“icse2025-paper1115”, p. 5) #00b036
|
||
*Good!*
|
||
|
||
- “We took into account the possibility of 442 having synonyms and, in cases of missing classes, we assigned 443 a penalty based on the importance of these classes in the 444 original textual description and the extent to which they were 445 strictly needed or could have been omitted.” (“icse2025-paper1115”, p. 5) #5fb236
|
||
|
||
- “we took into account synonyms and the importance of 469 specific attributes in the textual description.” (“icse2025-paper1115”, p. 5) #5fb236
|
||
|
||
- “main operations for managing events and notifications 487 are present in the LLM-generated solution and are quite 488 aligned with the ground truth.” (“icse2025-paper1115”, p. 5) #e56eee
|
||
|
||
- “host” (“icse2025-paper1115”, p. 6) #5fb236
|
||
|
||
- “some participants.” (“icse2025-paper1115”, p. 6) #5fb236
|
||
|
||
- “configuration of events. When a new participant is added to an event, the Event Manager uses another component, the Notification Handler,” (“icse2025-paper1115”, p. 6) #00b036
|
||
|
||
- “The evaluation of UML models generated by various LLMs 621 across multiple criteria reveals distinct strengths and weak622 nesses.” (“icse2025-paper1115”, p. 8) #5fb236
|
||
|
||
- “Table II provides a comprehensive summary 625 of the performance metrics” (“icse2025-paper1115”, p. 8) #a28ae5
|
||
|
||
- “In contrast, LLaMA 3 struggles with class identification and 640 association definition, showing lower accuracy in cardinalities 641 and clarity.” (“icse2025-paper1115”, p. 8) #5fb236
|
||
|
||
- “188 generated UML models were evaluated by three of the 699 co-authors.” (“icse2025-paper1115”, p. 9) #5fb236
|
||
|
||
- “These 14 models were then also evaluated by the other 702 two coauthors, both of whom have over 20 years of experience 703 in UML modeling.” (“icse2025-paper1115”, p. 9) #5fb236
|
||
|
||
- “the process of evaluating UML models is in- 726 herently subjective.” (“icse2025-paper1115”, p. 9) #ffd400
|
||
*Exactly, I agree with this!*
|
||
|
||
- “they were all produced 739 by a single instructor, so to mitigate this issue future studies 740 could incorporate descriptions from diverse authors” (“icse2025-paper1115”, p. 9) #ffd400
|
||
*This is a problem in my opinion. Even the solutions for the considered cases, should have been discussed among the different authors for creating the ground truth, that currently is defined by a single instructor.*
|
||
|
||
- “By scoring and comparing the 770 generated diagrams to expert-defined solutions, this systematic 771 evaluation framework provides a robust basis for assessing 772 the quality of UML models generated by LLMs.” (“icse2025-paper1115”, p. 10) #5fb236
|
||
|
||
- “By evaluating a broader range of LLMs, 803 we can better understand their strengths and limitations, ulti- 804 mately identifying those that excel at generating high-quality 805 UML diagrams. This will not only help in advancing the field, 806 but also provide valuable insights for practitioners seeking the 807 best tools for their software modeling needs.” (“icse2025-paper1115”, p. 10) #5fb236
|
||
|
||
- “18” (“icse2025-paper1115”, p. 11) #f0ff00
|
||
*To be checked.* |