logseq/pages/STAF_2025_paper_39.md

type:: REVIEWS tags:: year:: 2025 venue:: STAF full-title:: Leveraging LLMs to support co-evolution between definitions and instances of textual DSLs date-start:: 25-02-2025 - 16:57 date-submitted:: external-links:: status:: DONE deadline-submission:: file:: @Leveraging LLMs to support co-evolution between definitions and instances of textual DSLs parent:: todoist::

- ### [[Highlights]]
	- # Annotazioni
	- “However, these techniques are not geared to support DSLs with a textual syntax — applying them to textual language definitions and instances may lead to the loss of information from the original instances, such as comments and layout information, which are valuable for software comprehension and maintenance.” ([Zhang et al., p. 1](zotero://select/library/items/Q3TEL7CL)) ([pdf](zotero://open-pdf/library/items/XUFLKKRA?page=1&annotation=5VMLAAEK)) #5fb236
	- “(LLM)-based solutions in achieving grammar and instance co-evolution” ([Zhang et al., p. 1](zotero://select/library/items/Q3TEL7CL)) ([pdf](zotero://open-pdf/library/items/XUFLKKRA?page=1&annotation=2BDNFMG2)) #2ea8e5
	- “ability to preserve auxiliary information when directly processing textual instances” ([Zhang et al., p. 1](zotero://select/library/items/Q3TEL7CL)) ([pdf](zotero://open-pdf/library/items/XUFLKKRA?page=1&annotation=S27BNSKP)) #5fb236
	- “Claude-3.5 and GPT-4o” ([Zhang et al., p. 1](zotero://select/library/items/Q3TEL7CL)) ([pdf](zotero://open-pdf/library/items/XUFLKKRA?page=1&annotation=TGG45Z4T)) #a28ae5
	- “migrating textual instances in small-scale cases” ([Zhang et al., p. 1](zotero://select/library/items/Q3TEL7CL)) ([pdf](zotero://open-pdf/library/items/XUFLKKRA?page=1&annotation=L4LJFWU6)) #5fb236
	- “significant challenges with the scalability of LLM-based solutions to larger languages and instances, leading to insights that are useful for informing future research.” ([Zhang et al., p. 1](zotero://select/library/items/Q3TEL7CL)) ([pdf](zotero://open-pdf/library/items/XUFLKKRA?page=1&annotation=KHASJBRH)) #a28ae5
	- “To that end, the original instance needs to be parsed into the form of a model and transformed back into textual form after the model is co-evolved.” ([Zhang et al., p. 2](zotero://select/library/items/Q3TEL7CL)) ([pdf](zotero://open-pdf/library/items/XUFLKKRA?page=2&annotation=HN8WNRWJ)) #5fb236
	- “it serves a critical purpose during tasks such as code maintenance, debugging, and understanding design intent” ([Zhang et al., p. 2](zotero://select/library/items/Q3TEL7CL)) ([pdf](zotero://open-pdf/library/items/XUFLKKRA?page=2&annotation=SMQM82P5)) #5fb236
	- “We harness our recently published dataset on Xtext-based language evolution cases retrieved from GitHub (Zhang & Strüber 2024),” ([Zhang et al., p. 2](zotero://select/library/items/Q3TEL7CL)) ([pdf](zotero://open-pdf/library/items/XUFLKKRA?page=2&annotation=CXYZPJXI)) #a28ae5
	- “the following” ([Zhang et al., p. 2](zotero://select/library/items/Q3TEL7CL)) ([pdf](zotero://open-pdf/library/items/XUFLKKRA?page=2&annotation=K8JFEZ7R)) #ff6666
	- “RQ1: How can we use LLMs to automate the co-evolution of instances with evolving Xtext grammars?” ([Zhang et al., p. 2](zotero://select/library/items/Q3TEL7CL)) ([pdf](zotero://open-pdf/library/items/XUFLKKRA?page=2&annotation=4L9WYU3D)) #e56eee
	- “build two LLM-based co-evolution solutions” ([Zhang et al., p. 2](zotero://select/library/items/Q3TEL7CL)) ([pdf](zotero://open-pdf/library/items/XUFLKKRA?page=2&annotation=XAMFJSC8)) #2ea8e5
	- “We designed a dedicated prompt to guide the LLMs to analyze the two grammars and perform evolution operations on the instance that conforms to the grammar before the evolution.” ([Zhang et al., p. 2](zotero://select/library/items/Q3TEL7CL)) ([pdf](zotero://open-pdf/library/items/XUFLKKRA?page=2&annotation=3BPJDDUH)) #2ea8e5
	- “analyze whether the new instance follows the evolved grammar and whether it retains the auxiliary information (such as comments) from the original instance.” ([Zhang et al., p. 2](zotero://select/library/items/Q3TEL7CL)) ([pdf](zotero://open-pdf/library/items/XUFLKKRA?page=2&annotation=FSVVWXYB)) #2ea8e5
	- “RQ2: How do grammar size, instance size, and grammar change extent affect the capability of LLMs to produce correct solutions when co-evolving textual instances?” ([Zhang et al., p. 2](zotero://select/library/items/Q3TEL7CL)) ([pdf](zotero://open-pdf/library/items/XUFLKKRA?page=2&annotation=F5SW2ZTE)) #e56eee
	- “RQ3: How do these factors affect LLMs’ capability i” ([Zhang et al., p. 2](zotero://select/library/items/Q3TEL7CL)) ([pdf](zotero://open-pdf/library/items/XUFLKKRA?page=2&annotation=SQ24XUQG)) #e56eee
	- “ity in preserving auxiliary information during DSL co-evolution?” ([Zhang et al., p. 2](zotero://select/library/items/Q3TEL7CL)) ([pdf](zotero://open-pdf/library/items/XUFLKKRA?page=2&annotation=3232BZ7C)) #e56eee
	- “exploration of the potential of LLM for the co-evolution of grammar and instances of DSLs” ([Zhang et al., p. 2](zotero://select/library/items/Q3TEL7CL)) ([pdf](zotero://open-pdf/library/items/XUFLKKRA?page=2&annotation=HNYR2GND)) #5fb236
	- “In Xtext language projects, there is a close relationship between grammar and metamodel” ([Zhang et al., p. 2](zotero://select/library/items/Q3TEL7CL)) ([pdf](zotero://open-pdf/library/items/XUFLKKRA?page=2&annotation=NDTR94PT)) #5fb236
	- “ring .” ([Zhang et al., p. 3](zotero://select/library/items/Q3TEL7CL)) ([pdf](zotero://open-pdf/library/items/XUFLKKRA?page=3&annotation=39LLYQCI)) #ff6666
	- “This is illustrated by Listing 2 showing the outcome of applying such an approach to the exam instance, which leads to the loss of all comments and formatting information.” ([Zhang et al., p. 3](zotero://select/library/items/Q3TEL7CL)) ([pdf](zotero://open-pdf/library/items/XUFLKKRA?page=3&annotation=4SSHLIV8)) #5fb236
	- “The evolution of programming languages is primarily driven by the need to improve readability, maintainability, and execution efficiency while supporting increasingly complex software systems.” ([Zhang et al., p. 3](zotero://select/library/items/Q3TEL7CL)) ([pdf](zotero://open-pdf/library/items/XUFLKKRA?page=3&annotation=X8TCPNYG)) #5fb236
	- “DSL evolution is similar to general-purpose programming languages, but due to its domain-specific nature, evolution is typically constrained by domain requirements, necessitating a balance between extensibility and stability.” ([Zhang et al., p. 3](zotero://select/library/items/Q3TEL7CL)) ([pdf](zotero://open-pdf/library/items/XUFLKKRA?page=3&annotation=T66S2DUA)) #5fb236
	- “Recent models like GPT-4o and Claude have been widely explored in code-related applications, with several studies reporting promising results in code completion, generation, and transformation tasks” ([Zhang et al., p. 4](zotero://select/library/items/Q3TEL7CL)) ([pdf](zotero://open-pdf/library/items/XUFLKKRA?page=4&annotation=GG2QR4YY)) #5fb236
	- “framework for evaluating tool support for DSL co-evolution” ([Zhang et al., p. 4](zotero://select/library/items/Q3TEL7CL)) ([pdf](zotero://open-pdf/library/items/XUFLKKRA?page=4&annotation=TPJYFZNY)) #5fb236
	- “In contrast, our work addresses the challenges of textual DSL evolution, particularly the preservation of auxiliary information in source code (such as comments and layout).” ([Zhang et al., p. 4](zotero://select/library/items/Q3TEL7CL)) ([pdf](zotero://open-pdf/library/items/XUFLKKRA?page=4&annotation=48NNMY9D)) #a28ae5
	- “proposed a technical framework to guide LLM adoption in MDE” ([Zhang et al., p. 4](zotero://select/library/items/Q3TEL7CL)) ([pdf](zotero://open-pdf/library/items/XUFLKKRA?page=4&annotation=JYUC9HMX)) #5fb236
	- “provides a valuable reference for our handling of textual instance evolution.” ([Zhang et al., p. 4](zotero://select/library/items/Q3TEL7CL)) ([pdf](zotero://open-pdf/library/items/XUFLKKRA?page=4&annotation=DBZIAPN3)) #a28ae5
	- “They proposed a prompt engineeringbased approach that guides LLMs through structured prompts containing metamodel abstraction gaps, change information, and erroneous code” ([Zhang et al., p. 4](zotero://select/library/items/Q3TEL7CL)) ([pdf](zotero://open-pdf/library/items/XUFLKKRA?page=4&annotation=TYVJPHNE)) #a28ae5
	- “Although the research objectives differ, their work provides valuable insights for our research, and their results confirm the potential of LLMs in handling software artifact evolution problems, which aligns with our findings” ([Zhang et al., p. 4](zotero://select/library/items/Q3TEL7CL)) ([pdf](zotero://open-pdf/library/items/XUFLKKRA?page=4&annotation=L459LIZ4)) #5fb236
	- “we selected a diverse set of seven DSLs as evaluation objects” ([Zhang et al., p. 4](zotero://select/library/items/Q3TEL7CL)) ([pdf](zotero://open-pdf/library/items/XUFLKKRA?page=4&annotation=IG27NN7V)) #2ea8e5
	- “We then designed an LLM-based method to co-evolve grammar and instances and implemented it as two solutions based on two LLMs (i.e., Claude-3.5 and GPT-4o)” ([Zhang et al., p. 4](zotero://select/library/items/Q3TEL7CL)) ([pdf](zotero://open-pdf/library/items/XUFLKKRA?page=4&annotation=WDQGAFXZ)) #2ea8e5
	- “we applied these two solutions to the selected case languages and analyze the results to evaluate the potential of LLM to co-evolve grammar and instances” ([Zhang et al., p. 4](zotero://select/library/items/Q3TEL7CL)) ([pdf](zotero://open-pdf/library/items/XUFLKKRA?page=4&annotation=XXHYNHZ4)) #2ea8e5
	- “from in” ([Zhang et al., p. 4](zotero://select/library/items/Q3TEL7CL)) ([pdf](zotero://open-pdf/library/items/XUFLKKRA?page=4&annotation=VJNUD9ST)) #ff6666
	- “Once the grammar versions before and after the evolution step are determined, for each of them, we look for an instance that complies with them.” ([Zhang et al., p. 5](zotero://select/library/items/Q3TEL7CL)) ([pdf](zotero://open-pdf/library/items/XUFLKKRA?page=5&annotation=587Z2EQC)) #5fb236
	- “the original grammar” ([Zhang et al., p. 5](zotero://select/library/items/Q3TEL7CL)) ([pdf](zotero://open-pdf/library/items/XUFLKKRA?page=5&annotation=FZKNUJEY)) #5fb236
	- “instance conforming to it” ([Zhang et al., p. 5](zotero://select/library/items/Q3TEL7CL)) ([pdf](zotero://open-pdf/library/items/XUFLKKRA?page=5&annotation=EDKB3MLA)) #5fb236
	- “the evolved grammar,” ([Zhang et al., p. 5](zotero://select/library/items/Q3TEL7CL)) ([pdf](zotero://open-pdf/library/items/XUFLKKRA?page=5&annotation=L28BWUFV)) #5fb236
	- “LLMs are expected to analyze the grammar differences and perform instance evolution accordingly” ([Zhang et al., p. 5](zotero://select/library/items/Q3TEL7CL)) ([pdf](zotero://open-pdf/library/items/XUFLKKRA?page=5&annotation=68S8W4N4)) #5fb236
	- “Claude-3.5 requires Anthropic’s API key,” ([Zhang et al., p. 5](zotero://select/library/items/Q3TEL7CL)) ([pdf](zotero://open-pdf/library/items/XUFLKKRA?page=5&annotation=DIB6AE7S)) #5fb236
	- “GPT4o requires OpenAI’s API key” ([Zhang et al., p. 5](zotero://select/library/items/Q3TEL7CL)) ([pdf](zotero://open-pdf/library/items/XUFLKKRA?page=5&annotation=BUYAXEWC)) #5fb236
	- “Although the two scripts call different APIs, they follow the same workflow” ([Zhang et al., p. 5](zotero://select/library/items/Q3TEL7CL)) ([pdf](zotero://open-pdf/library/items/XUFLKKRA?page=5&annotation=JFAN4JNU)) #ffd400  
	  
	  *But the prompts might be different, isn't it?*
	- “the prompting technique we adopt is Zero-shot Chain-of-thoughts prompting” ([Zhang et al., p. 5](zotero://select/library/items/Q3TEL7CL)) ([pdf](zotero://open-pdf/library/items/XUFLKKRA?page=5&annotation=4JLDVRWK)) #ffd400  
	  
	  *Double check this.*
	- “2015-07-18 2546850 15 2015-09-05 f0380e8” ([Zhang et al., p. 6](zotero://select/library/items/Q3TEL7CL)) ([pdf](zotero://open-pdf/library/items/XUFLKKRA?page=6&annotation=J7CVVQ4S)) #ffd400  
	  
	  *This time from the first to the second date is quite long.*
	- “2015-05-03 55911bf 42 2015-07-27 3fa6e6d” ([Zhang et al., p. 6](zotero://select/library/items/Q3TEL7CL)) ([pdf](zotero://open-pdf/library/items/XUFLKKRA?page=6&annotation=X754WTU7)) #ffd400  
	  
	  *See my previous comment*
	- “do” ([Zhang et al., p. 6](zotero://select/library/items/Q3TEL7CL)) ([pdf](zotero://open-pdf/library/items/XUFLKKRA?page=6&annotation=5HAW6RD5)) #ff6666
	- “initial prompt that we iteratively refined” ([Zhang et al., p. 6](zotero://select/library/items/Q3TEL7CL)) ([pdf](zotero://open-pdf/library/items/XUFLKKRA?page=6&annotation=MRSG8UGR)) #ffd400  
	  
	  *Interesting to discuss on the external validity of the obtained prompts, supposed to be the best ones at the end of the iterative process.*
	- “we adjusted and optimized the prompt according to the problem, and entered the next iteration” ([Zhang et al., p. 6](zotero://select/library/items/Q3TEL7CL)) ([pdf](zotero://open-pdf/library/items/XUFLKKRA?page=6&annotation=P547CZKP)) #ffd400  
	  
	  *See my previous comment.*
	- “To this end, we repeated the co-evolution When the instance was correctly co-evolved, we performed nine more co-evolution runs with the same prompt.” ([Zhang et al., p. 6](zotero://select/library/items/Q3TEL7CL)) ([pdf](zotero://open-pdf/library/items/XUFLKKRA?page=6&annotation=EWAKKBRH)) #ff6666
	- “The grammar before evolution is shown in Table 3, which contains five grammar rules.” ([Zhang et al., p. 6](zotero://select/library/items/Q3TEL7CL)) ([pdf](zotero://open-pdf/library/items/XUFLKKRA?page=6&annotation=FBVT9EGY)) #ff6666
	- “default in the grammar rule Feature which means that LLMs need to identify more changes in the grammar.” ([Zhang et al., p. 7](zotero://select/library/items/Q3TEL7CL)) ([pdf](zotero://open-pdf/library/items/XUFLKKRA?page=7&annotation=SI5GTJUI)) #5fb236
	- “we applied two Python scripts containing the final version of prompting text” ([Zhang et al., p. 8](zotero://select/library/items/Q3TEL7CL)) ([pdf](zotero://open-pdf/library/items/XUFLKKRA?page=8&annotation=W7KEWPA3)) #ffd400  
	  
	  *How does the prompt look like? It is interesting to see how different it is with respect the initial prompt previously shown.*
	- “i.e., the instances evolved by the LLMs according to the evolution of the grammar” ([Zhang et al., p. 8](zotero://select/library/items/Q3TEL7CL)) ([pdf](zotero://open-pdf/library/items/XUFLKKRA?page=8&annotation=NPYAS7J8)) #ffd400  
	  
	  *It is necessary to give some more details about the used dataset especially concerning the kinds of language changes that are involved and why not reusing the braking resolvable/unresolvable and unbraking changes classification?*
	- “We verified this version of the prompt text 10 times for both LLM solutions, and in most of these ten times, the instances obtained complied with the evolved grammar.” ([Zhang et al., p. 8](zotero://select/library/items/Q3TEL7CL)) ([pdf](zotero://open-pdf/library/items/XUFLKKRA?page=8&annotation=E4N3DJK5)) #5fb236
	- “an” ([Zhang et al., p. 8](zotero://select/library/items/Q3TEL7CL)) ([pdf](zotero://open-pdf/library/items/XUFLKKRA?page=8&annotation=WCIYWG8Z)) #ff6666
	- “compare” ([Zhang et al., p. 8](zotero://select/library/items/Q3TEL7CL)) ([pdf](zotero://open-pdf/library/items/XUFLKKRA?page=8&annotation=CRM79GAC)) #ff6666  
	  
	  *Decide the present or the past form and keep it consistent throughout the text*
	- “Answer to RQ1: We developed two automated solutions based on Claude-3.5 and GPT-4o, each consisting of a dedicated Python script and an optimized prompting text. These solutions take the grammar before evolution, the evolved grammar, and the instance that conforms to the grammar before the evolution, then use LLMs to analyze grammar differences and generate an evolved instance that conforms to the evolved grammar.” ([Zhang et al., p. 9](zotero://select/library/items/Q3TEL7CL)) ([pdf](zotero://open-pdf/library/items/XUFLKKRA?page=9&annotation=Q7739KI4)) #ffd400  
	  
	  *The answer should reflect the results that you obtained. Essentially the results shown in Tab 4 and Tab 5.  
	  
	  However, I suggest to present the results in terms of typical metrics like precision, recall, F-measure. Such metrics permit to grasp the quality of the approach more easily.*
	- “However, applying an LLM-based solution to large practical cases is currently not advisable.” ([Zhang et al., p. 11](zotero://select/library/items/Q3TEL7CL)) ([pdf](zotero://open-pdf/library/items/XUFLKKRA?page=11&annotation=HM99VZ3G)) #5fb236
	- “Thus, while we can judge the correctness of the evolution done by the LLMs as well as the loss of auxiliary information, we currently cannot use the real updated instances to compare whether a pure co-evolution would be performed similarly by human developers. Future work is required to understand better whether LLMs and human developers choose the same solutions during co-evolution, given multiple technically valid ways of evolving an instance.” ([Zhang et al., p. 11](zotero://select/library/items/Q3TEL7CL)) ([pdf](zotero://open-pdf/library/items/XUFLKKRA?page=11&annotation=84WVF33R)) #5fb236  
	  
	  *So essentially, you claiming the importance of good quality data for supporting the evaluation.*
	- “It is also important to study larger changes, therefore, in future work we plan to study two versions of the language with larger changes, for example, to cover the changes between different UML versions.” ([Zhang et al., p. 11](zotero://select/library/items/Q3TEL7CL)) ([pdf](zotero://open-pdf/library/items/XUFLKKRA?page=11&annotation=2EQ7DWE4)) #5fb236
	- “future work should investigate improved techniques for handling large-scale LLM-based DSL evolution” ([Zhang et al., p. 12](zotero://select/library/items/Q3TEL7CL)) ([pdf](zotero://open-pdf/library/items/XUFLKKRA?page=12&annotation=4CPV3H97)) #2ea8e5
	- “ways to leverage multiple prompts” ([Zhang et al., p. 12](zotero://select/library/items/Q3TEL7CL)) ([pdf](zotero://open-pdf/library/items/XUFLKKRA?page=12&annotation=ZF2PF48A)) #2ea8e5
	- “retrieval-based augmentation” ([Zhang et al., p. 12](zotero://select/library/items/Q3TEL7CL)) ([pdf](zotero://open-pdf/library/items/XUFLKKRA?page=12&annotation=T6SHR25U)) #2ea8e5
	- “having the LLM generate migration code.” ([Zhang et al., p. 12](zotero://select/library/items/Q3TEL7CL)) ([pdf](zotero://open-pdf/library/items/XUFLKKRA?page=12&annotation=2EJBYVAS)) #2ea8e5
- ### [[Comments]]
	- Summary: The paper explores the application of LLMs to address the challenge of adapting texts written in a given DSL when its grammar evolves, potentially rendering existing instances invalid. The authors describe the process they followed, which primarily involves designing a prompt and using it with two LLMs, Claude-3.5 and GPT-4o, across seven evolution scenarios. The results are promising, although further research is needed, particularly to handle co-evolutions in large DSLs effectively.
	- Comments: The paper is interesting and about a relevant topic. Overall I liked the paper even though it requires improvements to address at least the following issues:
	  id:: 67bdeb97-beb1-4754-85e6-5c5649e1a54b
		- When presenting the results, I suggest using standard evaluation metrics such as precision, recall, and F-measure, which are commonly employed in similar contexts.
		- The response to RQ1 should be revised to align with the results obtained, particularly those presented in Tables 4 and 5. Additionally, presenting the results using precision, recall, and F-measure would enhance clarity and facilitate a better understanding of the effectiveness of the approach.
		- Additional details about the dataset are needed, especially regarding the types of language changes considered. It would be useful to clarify why the classification of changes into breaking (resolvable/unresolvable) and non-breaking was not reused.
		- A discussion on whether the two LLMs required different prompt refinements or whether a single prompt worked optimally for both would add valuable insights.
		- The paper contains some grammatical errors. Additionally, please ensure consistency in verb tense (present vs. past) throughout the text, especially in Section 6