type:: [[REVIEWS]]
tags::
year:: 2024
venue:: [[ICSE]]
full-title:: Software Model Evolution with Large Language Models: Experiments on Simulated, Public, and Industrial Datasets
date-start:: [[16-04-2024]] - 14:03
date-submitted:: [[29-09-2024]] 
external-links::
status:: [[DONE]] 
deadline-submission::
file:: [[@icse2025-paper548.pdf]]
parent::
todoist:: https://app.todoist.com/app/task/548-software-model-evolution-with-large-language-models-experiments-on-simulated-7858696723

	- ### [[Highlights]]
	  collapsed:: true
		- # Annotazioni
			- (4/5/2024, 12:00:49)
			- - “Software Model Evolution with Large Language Models: Experiments on Simulated, Public, and Industrial Datasets” ([“icse2025-paper548.pdf”, p. 1](zotero://select/library/items/AWLJ8GQM)) ([pdf](zotero://open-pdf/library/items/CGAJ2DE8?page=1&annotation=WVZI6NN3)) #5fb236
			- *假设*
			- - “model evolution” ([“icse2025-paper548.pdf”, p. 1](zotero://select/library/items/AWLJ8GQM)) ([pdf](zotero://open-pdf/library/items/CGAJ2DE8?page=1&annotation=SBBAN285)) #f0ff00
			- *The term model evolution seems to be used not appropriately.*
			- - “model repositories and software modelhistories interchangeably,” ([“icse2025-paper548.pdf”, p. 2](zotero://select/library/items/AWLJ8GQM)) ([pdf](zotero://open-pdf/library/items/CGAJ2DE8?page=2&annotation=GEL9D6CB)) #f0ff00
			- *These have different meanings.*
			- - “Abstract Syntax Graph” ([“icse2025-paper548.pdf”, p. 3](zotero://select/library/items/AWLJ8GQM)) ([pdf](zotero://open-pdf/library/items/CGAJ2DE8?page=3&annotation=JIAEPEAT)) #a28ae5
			- - “Labeled Directed Graph” ([“icse2025-paper548.pdf”, p. 3](zotero://select/library/items/AWLJ8GQM)) ([pdf](zotero://open-pdf/library/items/CGAJ2DE8?page=3&annotation=3VGVK2BU)) #a28ae5
			- - “Structural Model Difference” ([“icse2025-paper548.pdf”, p. 3](zotero://select/library/items/AWLJ8GQM)) ([pdf](zotero://open-pdf/library/items/CGAJ2DE8?page=3&annotation=69BPZQYU)) #a28ae5
			- - “Simple Change Graph” ([“icse2025-paper548.pdf”, p. 3](zotero://select/library/items/AWLJ8GQM)) ([pdf](zotero://open-pdf/library/items/CGAJ2DE8?page=3&annotation=DCWGQ7IN)) #a28ae5
			- - “adjacent” ([“icse2025-paper548.pdf”, p. 3](zotero://select/library/items/AWLJ8GQM)) ([pdf](zotero://open-pdf/library/items/CGAJ2DE8?page=3&annotation=6WJY57JE)) #f0ff00
			- *I I understand what you mean but meybe you should say better. I guess you are referring to model elements that are related to changes.*
			- - “Endogenous model transformation)” ([“icse2025-paper548.pdf”, p. 3](zotero://select/library/items/AWLJ8GQM)) ([pdf](zotero://open-pdf/library/items/CGAJ2DE8?page=3&annotation=2U4IL5G2)) #f0ff00
			- *It's endogenous because the source and target metanodels are the same, right?*
			- - “edit operation” ([“icse2025-paper548.pdf”, p. 3](zotero://select/library/items/AWLJ8GQM)) ([pdf](zotero://open-pdf/library/items/CGAJ2DE8?page=3&annotation=IIJYU7TW)) #a28ae5
			- - “a set of model transformations that have the same simple change graph” ([“icse2025-paper548.pdf”, p. 3](zotero://select/library/items/AWLJ8GQM)) ([pdf](zotero://open-pdf/library/items/CGAJ2DE8?page=3&annotation=QM2DRTCG)) #e56eee
			- - “Given an edit operation ε and a concrete model m, one can perform the removal of “Remove” nodes and the gluing of “Add” nodes as defined by the simple change graph corresponding to ε, and then set concrete attributes.” ([“icse2025-paper548.pdf”, p. 3](zotero://select/library/items/AWLJ8GQM)) ([pdf](zotero://open-pdf/library/items/CGAJ2DE8?page=3&annotation=4LTEQLH6)) #ffd400
			- *Why are you specifically focusing on the "Remove" then "Add" change? Edit operations can be arbitrary.*
			- - “We are interested in completing software models. That is, for an observed evolution m ε→ n, we want to find a completion γ ∈ E, such that m ε→ n γ→ c is an observable completion.” ([“icse2025-paper548.pdf”, p. 3](zotero://select/library/items/AWLJ8GQM)) ([pdf](zotero://open-pdf/library/items/CGAJ2DE8?page=3&annotation=P5BUSAPN)) #ffd400
			- *So far it is not clear what are the benefits of the given formalization. Let's see what's next.*
			- - “(Software) model completion is the task of further evolving a software model based on a given (partial) model.” ([“icse2025-paper548.pdf”, p. 3](zotero://select/library/items/AWLJ8GQM)) ([pdf](zotero://open-pdf/library/items/CGAJ2DE8?page=3&annotation=BPZWWCD5)) #a28ae5
			- - “Model Completion” ([“icse2025-paper548.pdf”, p. 4](zotero://select/library/items/AWLJ8GQM)) ([pdf](zotero://open-pdf/library/items/CGAJ2DE8?page=4&annotation=TR2YXDJE)) #a28ae5
			- - “Definition III.7 (Model Completion). Given a set of model transformations T , model completion is a computable function C : T → T that, given a model transformation m ε→ n from a source model m to a (partial) target model n, computes a model transformation C(m ε→ n) = n γ→ c. We call the edit operation γ a software model completion.” ([“icse2025-paper548.pdf”, p. 4](zotero://select/library/items/AWLJ8GQM)) ([pdf](zotero://open-pdf/library/items/CGAJ2DE8?page=4&annotation=G4PHQZIB)) #ffd400
			- *The given formalization does not manage the case when you start from a model n without knowing what are the transformations that have previously applied to obtain the model n from some unknown source model m.*
			- - “Language Model)” ([“icse2025-paper548.pdf”, p. 4](zotero://select/library/items/AWLJ8GQM)) ([pdf](zotero://open-pdf/library/items/CGAJ2DE8?page=4&annotation=NQ533X2S)) #a28ae5
			- - “A language model is a conditional probability distribution P(ω|c) for a (sequence of) token(s) ω, given a sequence of context tokens c.” ([“icse2025-paper548.pdf”, p. 4](zotero://select/library/items/AWLJ8GQM)) ([pdf](zotero://open-pdf/library/items/CGAJ2DE8?page=4&annotation=5U7PJWXT)) #a28ae5
			- - “Retrieval-augmented generation includes additional knowledge in the context (or prompt).” ([“icse2025-paper548.pdf”, p. 4](zotero://select/library/items/AWLJ8GQM)) ([pdf](zotero://open-pdf/library/items/CGAJ2DE8?page=4&annotation=5KQTL3LX)) #5fb236
			- - “Fine-tuning adjusts the LLM’s weights based on additional training data.” ([“icse2025-paper548.pdf”, p. 4](zotero://select/library/items/AWLJ8GQM)) ([pdf](zotero://open-pdf/library/items/CGAJ2DE8?page=4&annotation=3A6IXJUC)) #5fb236
			- - “As said in Section I,” ([“icse2025-paper548.pdf”, p. 4](zotero://select/library/items/AWLJ8GQM)) ([pdf](zotero://open-pdf/library/items/CGAJ2DE8?page=4&annotation=MZT3C2LC)) #ff6666
			- - “our approach of how” ([“icse2025-paper548.pdf”, p. 4](zotero://select/library/items/AWLJ8GQM)) ([pdf](zotero://open-pdf/library/items/CGAJ2DE8?page=4&annotation=9V4BRSKE)) #ff6666
			- - “)” ([“icse2025-paper548.pdf”, p. 4](zotero://select/library/items/AWLJ8GQM)) ([pdf](zotero://open-pdf/library/items/CGAJ2DE8?page=4&annotation=34Z9U8W8)) #ff6666
			- - “LLMs” ([“icse2025-paper548.pdf”, p. 4](zotero://select/library/items/AWLJ8GQM)) ([pdf](zotero://open-pdf/library/items/CGAJ2DE8?page=4&annotation=C6TIPQVJ)) #5fb236
			- *[[LLMs]]*
			- - “retrieval-augmented generation” ([“icse2025-paper548.pdf”, p. 4](zotero://select/library/items/AWLJ8GQM)) ([pdf](zotero://open-pdf/library/items/CGAJ2DE8?page=4&annotation=A8SAF4LS)) #5fb236
			- *[[RAG]]*
			- - “straight forwardly” ([“icse2025-paper548.pdf”, p. 4](zotero://select/library/items/AWLJ8GQM)) ([pdf](zotero://open-pdf/library/items/CGAJ2DE8?page=4&annotation=XAWFJNZP)) #ff6666
			- - “First, we do not have to work with the entire software model representation, but we can focus on slices of the models around recently changed elements” ([“icse2025-paper548.pdf”, p. 4](zotero://select/library/items/AWLJ8GQM)) ([pdf](zotero://open-pdf/library/items/CGAJ2DE8?page=4&annotation=J543YLW9)) #ffd400
			- *That's ok, I'm just wondering if considering the context, ie. the nodes that are in the model but not in the simple change graph might improve the performance of the recommender.*
			- - “This is one tactic of dealing with the common problem of the limited context of a LLM (see Section III-B).” ([“icse2025-paper548.pdf”, p. 4](zotero://select/library/items/AWLJ8GQM)) ([pdf](zotero://open-pdf/library/items/CGAJ2DE8?page=4&annotation=DID64S9Z)) #ffd400
			- *This limitation is not discussed in Sec. III-B, thus I would remove the reference to that section.*
			- - “is explained in Section III.” ([“icse2025-paper548.pdf”, p. 4](zotero://select/library/items/AWLJ8GQM)) ([pdf](zotero://open-pdf/library/items/CGAJ2DE8?page=4&annotation=NSF3FQVG)) #ffd400
			- *I would also remove such a back reference.*
			- - “, w” ([“icse2025-paper548.pdf”, p. 4](zotero://select/library/items/AWLJ8GQM)) ([pdf](zotero://open-pdf/library/items/CGAJ2DE8?page=4&annotation=H4NJF9FP)) #ff6666
			- - “s” ([“icse2025-paper548.pdf”, p. 4](zotero://select/library/items/AWLJ8GQM)) ([pdf](zotero://open-pdf/library/items/CGAJ2DE8?page=4&annotation=W9NEY6GF)) #ff6666
			- - “Figure 3: Overview of the RAMC approach.” ([“icse2025-paper548.pdf”, p. 5](zotero://select/library/items/AWLJ8GQM)) ([pdf](zotero://open-pdf/library/items/CGAJ2DE8?page=5&annotation=MP79B49Q)) #ffd400
			- *I would anticipate this section earlier in section IV.*
			- - “Regarding our running example in a⃝ of Figure 1, we also highlighted these model differences by color, that is, “added” model elements are depicted in green.” ([“icse2025-paper548.pdf”, p. 5](zotero://select/library/items/AWLJ8GQM)) ([pdf](zotero://open-pdf/library/items/CGAJ2DE8?page=5&annotation=VR95S5X7)) #ffd400
			- *In general, what is the object of the recommendation? While editing, it might be possible to get as recommendation the name of an attribute to be added to the previously specified class, or even its type. How do you see this should work in practice?*
			- - “weakly connected components” ([“icse2025-paper548.pdf”, p. 5](zotero://select/library/items/AWLJ8GQM)) ([pdf](zotero://open-pdf/library/items/CGAJ2DE8?page=5&annotation=FAQGGDYC)) #ffd400
			- *Weakly connected components are not previously defined. Maybe you can define them in the context of the simple change graph definition.*
			- - “EdgeList” ([“icse2025-paper548.pdf”, p. 5](zotero://select/library/items/AWLJ8GQM)) ([pdf](zotero://open-pdf/library/items/CGAJ2DE8?page=5&annotation=FPW5A75D)) #5fb236
			- - “weakly connected simple change graph components” ([“icse2025-paper548.pdf”, p. 5](zotero://select/library/items/AWLJ8GQM)) ([pdf](zotero://open-pdf/library/items/CGAJ2DE8?page=5&annotation=6QYC7RKH)) #ffd400
			- *"weakly connected ... " they are "simple change graph", isn't it? If this is the case, I would avoid naming differently concepts already introduced. This makes the text confusing.*
			- - “EdgeList” ([“icse2025-paper548.pdf”, p. 5](zotero://select/library/items/AWLJ8GQM)) ([pdf](zotero://open-pdf/library/items/CGAJ2DE8?page=5&annotation=HHUYKVV9)) #5fb236
			- - “Sampling new edges. We can sample multiple model completion candidates from the LLM by using a beam search, or, instructing the model to generate several new edges. The edge sampling algorithms are given in detail on the supplementary Website.” ([“icse2025-paper548.pdf”, p. 5](zotero://select/library/items/AWLJ8GQM)) ([pdf](zotero://open-pdf/library/items/CGAJ2DE8?page=5&annotation=P9A37TGZ)) #ffd400
			- *As previously said in one of my comments, it is necessary to clearly define what are the elements that build recommendations.*
			- - “We evaluate to what extent our approach is able to derive structurally and semantically correct completion operations from the software model history.” ([“icse2025-paper548.pdf”, p. 5](zotero://select/library/items/AWLJ8GQM)) ([pdf](zotero://open-pdf/library/items/CGAJ2DE8?page=5&annotation=WRZ5GJY4)) #a28ae5
			- - “applicability in industrial scenarios.” ([“icse2025-paper548.pdf”, p. 5](zotero://select/library/items/AWLJ8GQM)) ([pdf](zotero://open-pdf/library/items/CGAJ2DE8?page=5&annotation=V4KXG23N)) #ffd400
			- *Check if industrial scenarios have been properly evaluated in the paper.*
			- - “we need to manually check for semantic correctness of serialized simple change graph” ([“icse2025-paper548.pdf”, p. 5](zotero://select/library/items/AWLJ8GQM)) ([pdf](zotero://open-pdf/library/items/CGAJ2DE8?page=5&annotation=9VDJGNH5)) #a28ae5
			- - “The reason is that our data include a significant amount of natural language text (e.g., comments that will rarely match), the order of the serialized edges and node ids is ambiguous (and there are up to n! possibilities), and application- specific identifiers in models (e.g., user-chosen attribute names) can rarely be exactly completed.” ([“icse2025-paper548.pdf”, p. 5](zotero://select/library/items/AWLJ8GQM)) ([pdf](zotero://open-pdf/library/items/CGAJ2DE8?page=5&annotation=SI6XHIFU)) #ffd400
			- *Please revise. It is not clear.*
			- - “On the supplementary Website a table summarizing related work on model completion is given,” ([“icse2025-paper548.pdf”, p. 6](zotero://select/library/items/AWLJ8GQM)) ([pdf](zotero://open-pdf/library/items/CGAJ2DE8?page=6&annotation=ZJI4FQGV)) #ffd400
			- *Where? I've not found it.*
			- - “In this table we list related work together with the proposed method, the evaluation task, evaluation metrics, dataset used, if the approach works on histories or not, and whether any other limitations are present. Among other limitations, this table highlights that most of related work is not applied to histories of models but static snapshots.” ([“icse2025-paper548.pdf”, p. 6](zotero://select/library/items/AWLJ8GQM)) ([pdf](zotero://open-pdf/library/items/CGAJ2DE8?page=6&annotation=84P7J7U2)) #ffd400
			- *In any case, this constitutes a significant portion of the paper and should be included within the main body of the document. While the appendix can offer additional details, the evidence demonstrating the novelty of the paper compared to existing research must be presented within the main text.*
			- - “semantic retrieval” ([“icse2025-paper548.pdf”, p. 6](zotero://select/library/items/AWLJ8GQM)) ([pdf](zotero://open-pdf/library/items/CGAJ2DE8?page=6&annotation=M8IHLFG5)) #ffd400
			- *The semantic retrieval aspect of RQ2 seems to overlap with the "retrieval-augmented generation" of RQ1, isn't it?*
			- - “modification” ([“icse2025-paper548.pdf”, p. 6](zotero://select/library/items/AWLJ8GQM)) ([pdf](zotero://open-pdf/library/items/CGAJ2DE8?page=6&annotation=2B9L7EEC)) #ff6666
			- - “Revisions” ([“icse2025-paper548.pdf”, p. 6](zotero://select/library/items/AWLJ8GQM)) ([pdf](zotero://open-pdf/library/items/CGAJ2DE8?page=6&annotation=86A7DGY9)) #ffd400
			- *What does "Number ore revisions" refer to? Are you covering the case of having different subsequent versions of a given model?*
			- - “Tinnes et al. [63]: We used a metamodel that resembles a simple component model So” ([“icse2025-paper548.pdf”, p. 7](zotero://select/library/items/AWLJ8GQM)) ([pdf](zotero://open-pdf/library/items/CGAJ2DE8?page=7&annotation=HQZ5NT2Z)) #ff6666
			- *Punctuation.*
			- - “samples” ([“icse2025-paper548.pdf”, p. 7](zotero://select/library/items/AWLJ8GQM)) ([pdf](zotero://open-pdf/library/items/CGAJ2DE8?page=7&annotation=UCZP4Y49)) #ffd400
			- *Samples of what? In the datasets you have models, revisions, changes. What have you considered as samples?*
			- - “First, we analyze the correctness of the generated completions:” ([“icse2025-paper548.pdf”, p. 7](zotero://select/library/items/AWLJ8GQM)) ([pdf](zotero://open-pdf/library/items/CGAJ2DE8?page=7&annotation=Y5J23FDX)) #ffd400
			- *The section needs to be revised with the aim of improving the presentation. It is necessary to clearly describe the evaluation approach and supporting tools. Currently, it is not even clear what are the inputs and with respect to what the correctness of the output is assessed.*
			- - “we ensure that every project is included, at least, once, even if it is very small.” ([“icse2025-paper548.pdf”, p. 7](zotero://select/library/items/AWLJ8GQM)) ([pdf](zotero://open-pdf/library/items/CGAJ2DE8?page=7&annotation=CS8AMI2T)) #ffd400
			- *Included where? What does it mean?*
			- - “This leaves us with 210 samples for the SYNTHETIC dataset, 221 samples for the REPAIRVISION dataset, and 200 samples for the INDUSTRY dataset.” ([“icse2025-paper548.pdf”, p. 7](zotero://select/library/items/AWLJ8GQM)) ([pdf](zotero://open-pdf/library/items/CGAJ2DE8?page=7&annotation=VCQBYJ8V)) #ffd400
			- *Such figures need explanation!*
			- - “graph isomorphism test” ([“icse2025-paper548.pdf”, p. 7](zotero://select/library/items/AWLJ8GQM)) ([pdf](zotero://open-pdf/library/items/CGAJ2DE8?page=7&annotation=RPFUXYB3)) #5fb236
			- - “isomorphism check of the completed edge(s)” ([“icse2025-paper548.pdf”, p. 7](zotero://select/library/items/AWLJ8GQM)) ([pdf](zotero://open-pdf/library/items/CGAJ2DE8?page=7&annotation=FAMZL9HZ)) #ffd400
			- *Please clarify what you mean with isomorphism check here.*
			- - “semantic correctness” ([“icse2025-paper548.pdf”, p. 7](zotero://select/library/items/AWLJ8GQM)) ([pdf](zotero://open-pdf/library/items/CGAJ2DE8?page=7&annotation=963Q6N86)) #5fb236
			- - “we sample from the INDUSTRY dataset” ([“icse2025-paper548.pdf”, p. 7](zotero://select/library/items/AWLJ8GQM)) ([pdf](zotero://open-pdf/library/items/CGAJ2DE8?page=7&annotation=G46Y7HBW)) #ffd400
			- *It seems you have operated different sampling steps without discussing what are the corresponding criteria that have been applied to perform them.*
			- - “semantic clusters that we defined.” ([“icse2025-paper548.pdf”, p. 7](zotero://select/library/items/AWLJ8GQM)) ([pdf](zotero://open-pdf/library/items/CGAJ2DE8?page=7&annotation=CLJVKTTF)) #ffd400
			- *The performed clustering step has not been presented.*
			- - “similar change” ([“icse2025-paper548.pdf”, p. 7](zotero://select/library/items/AWLJ8GQM)) ([pdf](zotero://open-pdf/library/items/CGAJ2DE8?page=7&annotation=TPTEZGA2)) #ffd400
			- *How change similarity is done?*
			- - “This leaves 122 samples” ([“icse2025-paper548.pdf”, p. 7](zotero://select/library/items/AWLJ8GQM)) ([pdf](zotero://open-pdf/library/items/CGAJ2DE8?page=7&annotation=S49FKR6M)) #ffd400
			- *How you got such number is not clear. Please explain.*
			- - “We also investigate how this affects correctness, that is, whether the similarity-based retrieval in RAMC affects the correctness of completions. To this end, we compare semantic sampling with few-shot samples that have been randomly retrieved from the training data. We evaluate this for semantic correctness” ([“icse2025-paper548.pdf”, p. 7](zotero://select/library/items/AWLJ8GQM)) ([pdf](zotero://open-pdf/library/items/CGAJ2DE8?page=7&annotation=M7QNHG5Q)) #ffd400
			- *The paper contains several paragraphs/sentences like this one that I would consider too dense since they try to convey too many things without clearly providing the required details to get the message.*
			- - “For this reason, and also to reduce the LLMs usage costs, we perform this analysis only for the INDUSTRY dataset.” ([“icse2025-paper548.pdf”, p. 7](zotero://select/library/items/AWLJ8GQM)) ([pdf](zotero://open-pdf/library/items/CGAJ2DE8?page=7&annotation=D9QK6AVH)) #ffd400
			- *I understand the concern of reducing the costs due to LLMs usage. However, this represents a threat to validity, since you have focused only on one dataset, and this can represent a relevant bias.*
			- - “outlining hypotheses for potential future research directions,” ([“icse2025-paper548.pdf”, p. 9](zotero://select/library/items/AWLJ8GQM)) ([pdf](zotero://open-pdf/library/items/CGAJ2DE8?page=9&annotation=BJBLENDG)) #5fb236
			- - “Not only are more than 90% of completions correct w.r.t. the serialization format, but we also find more than 62% of semantically correct completions for a real-world industrial setting. This indicates that LLMs with retrieval-augmented generation seem to be a promising solution for model completion.” ([“icse2025-paper548.pdf”, p. 9](zotero://select/library/items/AWLJ8GQM)) ([pdf](zotero://open-pdf/library/items/CGAJ2DE8?page=9&annotation=9FGP58AC)) #a28ae5
			- - “For example, we found cases of functional evolution where the language model is missing (domain) knowledge or requirements, or cases of a refactoring without any relevant few-shot sample.” ([“icse2025-paper548.pdf”, p. 10](zotero://select/library/items/AWLJ8GQM)) ([pdf](zotero://open-pdf/library/items/CGAJ2DE8?page=10&annotation=BIYL7WXI)) #ffd400
			- *RAG should overcame this issue. I found this a bit strange.*
			- - “fine-tuning during our evaluation” ([“icse2025-paper548.pdf”, p. 10](zotero://select/library/items/AWLJ8GQM)) ([pdf](zotero://open-pdf/library/items/CGAJ2DE8?page=10&annotation=WMH22VHV)) #ffd400
			- *How the fine-tuning phases compare with the process shown in Fig. 3? This is not clear in the paper.*
	- ### [[Comments]]
		- #.tabular
		  id:: 6636085b-459b-44e1-8fc2-afbfb5561e70
			- ### Paper summary
				- The paper explores the use of Language Model-based Models (LLMs) to facilitate modeler editing operations during modeling sessions. The authors compare the effectiveness of retrieval-augmented generation (implemented in the proposed RAMC approach) with fine-tuning. The results indicate that LLMs hold promise, achieving 62.30% semantic correctness in an industrial setting. The experiments suggest that retrieval-augmented generation (RAG) improves accuracy, while fine-tuning is a viable approach to support the problem of model evolution.
			- ### Strengths
				- + Innovative approach: ML and DL approaches have been recently proposed to support the problem of model assistance. The adoption of LLMs and RAG for software model evolution is novel.
				- + Relevance: The topic of software model evolution is highly relevant to the field of model-driven engineering and software engineering in general, and the paper's focus on LLM-based model completion addresses an important aspect of this domain.
			- ### Weaknesses
				- - Clarity of definitions: some definitions, such as "weakly connected components" and "graph isomorphism test," lack clarity and require further explanation for better understanding.
				- - Transparency of Processes: Certain aspects, such as sampling criteria and the clustering step, lack transparency, making it difficult to fully understand the presented research.
				- - Readability: While the paper's organization is clear, improvements are needed in terms of clarity and readability, especially in providing context and explanations for introduced concepts and processes.  For instance, the evaluation section is the one that requires major revisions.
			- ### Detailed comments for authors
				- Novelty: The paper introduces novel concepts such as model completion using Large Language Models (LLMs) and retrieval-augmented generation (RAG). While the paper demonstrates great potential in this regard, the presentation and some evaluation issues diminish the overall quality of the paper (see detailed comments below).
				- Rigor: Definitions and formalizations are provided for key concepts such as model completion, endogenous model transformation, and edit operations, enhancing the paper's rigor. However, certain definitions, like "weakly connected components" and "graph isomorphism test," lack clarity and require further elucidation. The formalization presented in section 3 could be condensed to allocate space to better explain the evaluation processes (see detailed comments below).
				- Relevance: The paper addresses the relevant and timely topic of software model evolution, which is crucial for software engineering and model-driven development.
				- Verifiability & transparency: The paper provides detailed explanations of the employed methodologies, such as RAG and fine-tuning, enhancing verifiability. Nonetheless, some aspects, including sampling criteria and the clustering step, lack transparency and necessitate additional clarification. A supplementary website collecting the scripts to execute the different experiments is provided.
				- Presentation: The paper's readability could be enhanced by providing more context and explanations for some introduced terms and processes as discussed in the detailed comments.
				- Detailed comments:
					- Below Fig. 1, the authors write, "Given an edit operation ε and a concrete model m, one can perform the removal of "Remove" nodes and the gluing of "Add" nodes as defined by the simple change graph corresponding to ε, and then set concrete attributes.". It is unclear why changes, such as renaming a structural feature or altering the type of an attribute, are not considered.
					- Definition III.7: The given formalization does not manage the case when you start from a model _n_ without knowing what transformations have previously been applied to obtain the model _n_ from some unknown source model _m_
					- Section IV.B: "First, we do not have to work with the entire software model representation, but we can focus on slices of the models around recently changed elements.": Even though it is clear the attempt of the authors to minimize the size of prompts, I'm wondering if also considering the context of changes, i.e., the nodes in the model but not in the simple change graph, might improve the performance of the recommender.
					- Section IV.B: "This is one tactic of dealing with the common problem of the limited context of an LLM (see Section III-B).": This limitation is not discussed in Sec. III-B; thus, I would remove the reference to that section.
					- Figure 3: While reading the descriptive text of Fig. 3, the reader can ask, "What is the object of the recommendation?" While editing, it might be possible to get as a recommendation the name of an attribute to be added to the previously specified class or even its type. How do you see this should work in practice?
					- Page 5, below Fig. 3 - "The simple change graph is then split into its weakly connected components": Weakly connected components are not previously defined. You can define them in the context of the simple change graph definition.
					- The first sentence of section IV.D - "The input to the training phase is a set of serialized weakly connected simple change graph components.": "weakly connected" is a "simple change graph", right? If so, I would avoid naming concepts already introduced differently. This makes the text confusing.
					- By reading the last paragraph of section IV.D, the reader can wonder again what elements build recommendations.
					- In the second paragraph of Section V - "The reason is that our data include a significant amount of natural language text (e.g., comments that will rarely match), the order of the serialized edges and node ids is ambiguous (and there are up to n! possibilities), and application-specific identifiers in models (e.g., user-chosen attribute names) can rarely be exactly completed.": The paper, and especially the evaluation section, contains several sentences like this one that are difficult to understand.
					- When referring to the supplementary Website, the authors mention the availability of a table summarizing related work on model completion. Unfortunately, I could not find such a table on the referred Website. By the way, such a summary table could constitute a significant portion of the paper and should be included within the main body of the document. While the appendix can offer additional details, the evidence demonstrating the novelty of the paper compared to existing research must be presented within the main text.
					- Research question 2, particularly the "semantic retrieval" aspect, seems to overlap with the investigation on RAG discussed in Research question 1.
					- Table 1: The column "number of revisions" should be detailed. Currently, it is not clear how many models contribute to such revisions. Are all revisions equally distributed over the corresponding models? Or are there some models that have seen more changes than others?
					- Section V.C, the authors write, "To answer RQ 1, we preprocess all three datasets from Section V-B and generate a collection with training (75%) and testing samples (25%)": Samples of what? In the datasets, you have models, revisions, and changes. What have you considered as samples?
					- In general, section V needs a significant revision to improve the presentation. It is necessary to describe the evaluation approach and supporting tools clearly. Currently, it is not even clear what the inputs are and how the correctness of the output is assessed.
					- Footnote 12 needs to be clarified. What does it mean that "we ensure that every project is included, at least, once, even if it is very small." Included where? Even the numbers presented immediately after (i.e., the considered number of samples) require an explanation.
					- Beginning of the second column on page 7 - "The automatic validation is done using a graph isomorphism test, which we can reduce to an isomorphism check of the completed edge(s) [...]" Please clarify what you mean by isomorphism check here, and how you perform it.
					- Immediately after - "Since this manual evaluation is very time consuming and requires domain expertise, we sample from the INDUSTRY dataset.": It seems you have operated different sampling steps without discussing the corresponding criteria that have been applied to perform them.
					- The subsequent sentence - "For this purpose, we examine the prompt and completion pairs to classify the changes into semantic clusters that we defined.": The clustering step that was performed has yet to be introduced. It appears for the first time here without proper explanation.
					- Immediately after - "We then only include samples that are unique according to the number of training examples, the class of the change, and whether there is a similar change in the training samples or not. This leaves 122 samples for the model completion task on the INDUSTRY dataset." - How is change similarity assessed? Moreover, the way you got the 122 samples needs to be clarified.
					- Page 7, Research question 2 - "We also investigate how this affects correctness, that is, whether the similarity-based retrieval in RAMC affects the correctness of completions. To this end, we compare semantic sampling with few-shot samples that have been randomly retrieved from the training data. We evaluate this for semantic correctness." The paper contains several paragraphs/sentences like this one that I consider too dense since they convey too many things without clearly providing the required details to get the message.
					- The subsequent sentence - "For this reason, and also to reduce the LLMs usage costs, we perform this analysis only for the INDUSTRY dataset.": I understand the concern about reducing the costs due to LLM usage. However, this represents a threat to validity since you have focused only on one dataset, which can represent a relevant bias.
					- Page 10, after Hypothesis 2 - "For example, we found cases of functional evolution where the language model is missing (domain) knowledge or requirements, or cases of a refactoring without any relevant few-shot sample.": This sentence is surprising since RAG should overcome this issue. More explanation is required.
	- ### [[REVIEWS/Notes]]