24 KiB
24 KiB
type:: REVIEWS tags:: year:: 2024 venue:: ICSE full-title:: SMATCH-M-LLM: Semantic Similarity in Metamodel Matching With Large Language Models date-start:: 29-09-2024 - 14:39 date-submitted:: external-links:: status:: done deadline-submission:: file:: @icse2025-paper1709 parent:: todoist:: https://app.todoist.com/app/task/1709-smatch-m-llm-semantic-similarity-in-metamodel-matching-with-large-language-6WFCX7Rw3gvGj4vc
- ### [[Highlights]]
- # Annotazioni
- (29/9/2024, 14:38:38)
- - “semantic matchers and alternatives to baseline methods for metamodel matching. However, metamodels can be large, which can overwhelm LLMs” ([“icse2025-paper1709”, p. 1](zotero://select/library/items/JE7EMCBD)) ([pdf](zotero://open-pdf/library/items/X7TMLKQV?page=1&annotation=XI3TH66W)) #00b036
- - “initial mapping derived from their elements’ definitions.” ([“icse2025-paper1709”, p. 1](zotero://select/library/items/JE7EMCBD)) ([pdf](zotero://open-pdf/library/items/X7TMLKQV?page=1&annotation=8L9R4LAQ)) #f0ff00
- *How is this obtained?*
- - “Industry standards also drive migration for better integration and tool use, such as legacy proprietary models to AUTOSTAR [5], [6]” ([“icse2025-paper1709”, p. 1](zotero://select/library/items/JE7EMCBD)) ([pdf](zotero://open-pdf/library/items/X7TMLKQV?page=1&annotation=YZNK5RGB)) #00b036
- - “similarities with metamodels” ([“icse2025-paper1709”, p. 1](zotero://select/library/items/JE7EMCBD)) ([pdf](zotero://open-pdf/library/items/X7TMLKQV?page=1&annotation=3UHI37HC)) #00b036
- - “KM3 and DOT” ([“icse2025-paper1709”, p. 2](zotero://select/library/items/JE7EMCBD)) ([pdf](zotero://open-pdf/library/items/X7TMLKQV?page=2&annotation=AECWWVDE)) #f0ff00
- *I'm not sure this is the right motivating example.....*
- - “When we prompt GPT-4 using the providing only specific parts of the metamodels, Node is correctly identified as a correspondence for DataType.” ([“icse2025-paper1709”, p. 2](zotero://select/library/items/JE7EMCBD)) ([pdf](zotero://open-pdf/library/items/X7TMLKQV?page=2&annotation=ZT88JBLG)) #f0ff00
- *That table does not show the matched elements. For instance oncernig the third row, Instead of showing only Node, I would also shows the Data Type of the Dot metamodel.*
- - “large” ([“icse2025-paper1709”, p. 2](zotero://select/library/items/JE7EMCBD)) ([pdf](zotero://open-pdf/library/items/X7TMLKQV?page=2&annotation=EVSNWPQ5)) #f0ff00
- *Large? They are very small.*
- - “III. THE SMATH-M-LLM APPROACH” ([“icse2025-paper1709”, p. 2](zotero://select/library/items/JE7EMCBD)) ([pdf](zotero://open-pdf/library/items/X7TMLKQV?page=2&annotation=K4IXEZHP)) #ffd400
- *In general this section needs reworking because it is not clear. A running and explanatory example is needed to give an overview of the approach before giving the details of the single components shown in Fig. 3.*
- - “prompting them with relevant fragments of the source and target metamodels.” ([“icse2025-paper1709”, p. 2](zotero://select/library/items/JE7EMCBD)) ([pdf](zotero://open-pdf/library/items/X7TMLKQV?page=2&annotation=37WEJ6I3)) #5fb236
- - “Fig. 3. SMATH-M-LLM” ([“icse2025-paper1709”, p. 3](zotero://select/library/items/JE7EMCBD)) ([pdf](zotero://open-pdf/library/items/X7TMLKQV?page=3&annotation=BJBSQFX7)) #ffd400
- *It's not clear why the process includes the calculation of initial correspondences and then finalize these after creating source and target fragments.*
- - “components SMATH-M” ([“icse2025-paper1709”, p. 3](zotero://select/library/items/JE7EMCBD)) ([pdf](zotero://open-pdf/library/items/X7TMLKQV?page=3&annotation=HUZZXE6A)) #ff4400
- *Of*
- - “definitions’ similarity.” ([“icse2025-paper1709”, p. 3](zotero://select/library/items/JE7EMCBD)) ([pdf](zotero://open-pdf/library/items/X7TMLKQV?page=3&annotation=89H6XEEL)) #f0ff00
- *What tool is exploited in his phase?*
- - “Fragment Constructor partitions the source metamodel into fragments using a community detection algorithm, it creates an initial target fragment for each source fragment by identifying the target concepts that correspond to the source concepts within the relevant fragment.” ([“icse2025-paper1709”, p. 3](zotero://select/library/items/JE7EMCBD)) ([pdf](zotero://open-pdf/library/items/X7TMLKQV?page=3&annotation=URL5KP6G)) #f0ff00
- *The needs for this component and how it works should be elaborated.*
- - “In our work, the stopping criteria are satisfied if the correspondences are non-abstract concepts and all their features are presented in the target fragment. In the following sections, we describe each component in detail and demonstrate its application to the motivational example.” ([“icse2025-paper1709”, p. 3](zotero://select/library/items/JE7EMCBD)) ([pdf](zotero://open-pdf/library/items/X7TMLKQV?page=3&annotation=NSXSZWHG)) #f0ff00
- *The process is not clear. Maybe an explanatory example is needed.*
- - “Definitions,” ([“icse2025-paper1709”, p. 3](zotero://select/library/items/JE7EMCBD)) ([pdf](zotero://open-pdf/library/items/X7TMLKQV?page=3&annotation=XSW2S9DB)) #f0ff00
- *Which one?*
- - “include more general roles, namely super-type and container concepts.” ([“icse2025-paper1709”, p. 3](zotero://select/library/items/JE7EMCBD)) ([pdf](zotero://open-pdf/library/items/X7TMLKQV?page=3&annotation=L5GUCTY2)) #f0ff00
- *Explain.*
- - “represented by their direct super-type.” ([“icse2025-paper1709”, p. 3](zotero://select/library/items/JE7EMCBD)) ([pdf](zotero://open-pdf/library/items/X7TMLKQV?page=3&annotation=KNFPDMVK)) #00b036
- - “This is represented by lines 1-2 in Algorithm 1.” ([“icse2025-paper1709”, p. 3](zotero://select/library/items/JE7EMCBD)) ([pdf](zotero://open-pdf/library/items/X7TMLKQV?page=3&annotation=9SLRJMIQ)) #f0ff00
- *We are still missing the description of listing 1 and you already talk about algorithm 1 which has not been yet introduced?*
- - “Applying this step to the metamodels KM3 and DOT (see Section II) results in selecting five representative concepts out of 16 concepts for KM3 (LocatedElement, ModelElement, Classifier, TypedElement, StructuralFeature) and eight out of 26 concepts for DOT (Label, Compartment, GraphElement, Arc, Shape, Nodelike, NodeShape, ComplexNodeShape). Notably, DataType and Node are excluded from the representative sets because they do not meet the specified criteria of being a super-type or a container. Additionally, although Package and Class are container concepts in KM3, they are not included in the representative set because they are already represented by Classifier and ModelElement, respectively. Furthermore, the contained element StructuralFeature is part of the representative set, which negates the need to include Class separately.” ([“icse2025-paper1709”, p. 3](zotero://select/library/items/JE7EMCBD)) ([pdf](zotero://open-pdf/library/items/X7TMLKQV?page=3&annotation=B5P8YBGX)) #f0ff00
- *By looking at these concepts I don't see sets of concepts that should be matched. Dot is related to graphical representations, km3 about metanodels why this should be matched without specifying the need of doing so? In that specific case, dot can be used to specify the graphical representation of metanodels (ie km3). Such a goal of the match is missing and I do not see how this is inferred (at least so far).*
- - “generate definitions for the selected concepts to enrich the matching process.” ([“icse2025-paper1709”, p. 3](zotero://select/library/items/JE7EMCBD)) ([pdf](zotero://open-pdf/library/items/X7TMLKQV?page=3&annotation=KP32HX6Y)) #f0ff00
- *I see where you want to go, and it can make sense, let's see*
- - “These definitions offer richer semantic information about the concepts.” ([“icse2025-paper1709”, p. 3](zotero://select/library/items/JE7EMCBD)) ([pdf](zotero://open-pdf/library/items/X7TMLKQV?page=3&annotation=MYNS33PC)) #ffd400
- *This can be very domain specific and potentially not part of the training data (I'm thinking for instance to custom domain specific languages that are not part of the training data.*
- - “Algorithm 1.” ([“icse2025-paper1709”, p. 3](zotero://select/library/items/JE7EMCBD)) ([pdf](zotero://open-pdf/library/items/X7TMLKQV?page=3&annotation=R6ZRFSKH)) #f0ff00
- *I suggest to introduce this properly.*
- - “TABLE II CONCEPTS AND GENERATED DEFINITIONS” ([“icse2025-paper1709”, p. 4](zotero://select/library/items/JE7EMCBD)) ([pdf](zotero://open-pdf/library/items/X7TMLKQV?page=4&annotation=9UFHVZG9)) #f0ff00
- *This means that the concepts should have appeared in the training data. What about new dsl?*
- - “Table II shows examples of definitions generated by GPT-4 for some concepts in KM3 and DOT metamodels.” ([“icse2025-paper1709”, p. 4](zotero://select/library/items/JE7EMCBD)) ([pdf](zotero://open-pdf/library/items/X7TMLKQV?page=4&annotation=L5DRBNBJ)) #ffd400
- *From the shown definition it is probable that the considered metamodels are part of the training data. See my previous comment.*
- - “we represent them as continuous dense vectors,” ([“icse2025-paper1709”, p. 4](zotero://select/library/items/JE7EMCBD)) ([pdf](zotero://open-pdf/library/items/X7TMLKQV?page=4&annotation=9FLZARHL)) #f0ff00
- *Can you show an example?*
- - “we select the most similar representative target concept based on the highest cosine similarity score (lines 8-10). This curated list of concept correspondences (concept correspondences in Algorithm 1) constitutes the final output of this component.” ([“icse2025-paper1709”, p. 4](zotero://select/library/items/JE7EMCBD)) ([pdf](zotero://open-pdf/library/items/X7TMLKQV?page=4&annotation=XBW4X5DR)) #f0ff00
- *How many? What's the acceptance threshold?*
- - “The task of refining the initial correspondences to achieve exact matches is done by the subsequent components.” ([“icse2025-paper1709”, p. 4](zotero://select/library/items/JE7EMCBD)) ([pdf](zotero://open-pdf/library/items/X7TMLKQV?page=4&annotation=W2FPCQ9Z)) #f0ff00
- *No humans in the loop?*
- - “measure cosine similarity” ([“icse2025-paper1709”, p. 4](zotero://select/library/items/JE7EMCBD)) ([pdf](zotero://open-pdf/library/items/X7TMLKQV?page=4&annotation=TP7LUFJ9)) #ffd400
- *Between what? It is not clear how couples of concepts are selected for doing such a measurement.*
- - “The Fragment Constructor component is responsible for preparing pairs of source and target fragments, with the pairing based on the list of correspondences identified by the Concept Matcher.” ([“icse2025-paper1709”, p. 4](zotero://select/library/items/JE7EMCBD)) ([pdf](zotero://open-pdf/library/items/X7TMLKQV?page=4&annotation=VAFULMFU)) #ffd400
- *Also in this cases it is needed to present an illustrative example to explain the steps of the process.*
- - “Source Fragmentation” ([“icse2025-paper1709”, p. 4](zotero://select/library/items/JE7EMCBD)) ([pdf](zotero://open-pdf/library/items/X7TMLKQV?page=4&annotation=DIGXRZEH)) #ffd400
- *You mix activities (e.g., pre-processing, source fragmentation) with artifacts (e.g. target fragment)*
- - “after constructing the fragments using Louvain, we ensure that relevant representative concepts are included in each fragment.” ([“icse2025-paper1709”, p. 4](zotero://select/library/items/JE7EMCBD)) ([pdf](zotero://open-pdf/library/items/X7TMLKQV?page=4&annotation=FZI3LDGM)) #5fb236
- - “Before providing the fragments to the LLM, we convert them into PlantUML format [26]. This format is chosen because it is concise, and it is widely recognized, thereby increasing the likelihood that LLMs will interpret it accurately” ([“icse2025-paper1709”, p. 6](zotero://select/library/items/JE7EMCBD)) ([pdf](zotero://open-pdf/library/items/X7TMLKQV?page=6&annotation=8DRJRFZQ)) #ffd400
- *Even this part of representing fragments to LLM after having converted them into PlantUML is not clear. User might want to match any potential couple of DSLs that might not have nothing todo with UML. Moreover, in which part of Listing 2, PlantUML specifications are given? It is not clear from Listing 2. n*
- - “baseline matching techniques” ([“icse2025-paper1709”, p. 6](zotero://select/library/items/JE7EMCBD)) ([pdf](zotero://open-pdf/library/items/X7TMLKQV?page=6&annotation=JFQCUEMX)) #a28ae5
- - “seven metamodels” ([“icse2025-paper1709”, p. 6](zotero://select/library/items/JE7EMCBD)) ([pdf](zotero://open-pdf/library/items/X7TMLKQV?page=6&annotation=WUR767IP)) #a28ae5
- - “three established baseline methods” ([“icse2025-paper1709”, p. 7](zotero://select/library/items/JE7EMCBD)) ([pdf](zotero://open-pdf/library/items/X7TMLKQV?page=7&annotation=HGJVPFBI)) #a28ae5
- - “Ontology Matching Methods” ([“icse2025-paper1709”, p. 7](zotero://select/library/items/JE7EMCBD)) ([pdf](zotero://open-pdf/library/items/X7TMLKQV?page=7&annotation=CEU6IJMY)) #5fb236
- - “Lexical Database Methods.” ([“icse2025-paper1709”, p. 7](zotero://select/library/items/JE7EMCBD)) ([pdf](zotero://open-pdf/library/items/X7TMLKQV?page=7&annotation=7VFQRPV6)) #5fb236
- - “LLM-based Methods.” ([“icse2025-paper1709”, p. 7](zotero://select/library/items/JE7EMCBD)) ([pdf](zotero://open-pdf/library/items/X7TMLKQV?page=7&annotation=QTJYU4PG)) #5fb236
- - “semantically equivalent from its respective transformation rules provided” ([“icse2025-paper1709”, p. 7](zotero://select/library/items/JE7EMCBD)) ([pdf](zotero://open-pdf/library/items/X7TMLKQV?page=7&annotation=QQBMWPEV)) #ffd400
- *This does not mean that concepts are semantically equivalent!!!!*
- - “The proposed LLM-based approach significantly outperforms baseline semantic matchers using ontology and lexical databases, indicating its potential to replace these methods with superior accuracy in identifying correspondences, addressing the known challenges” ([“icse2025-paper1709”, p. 8](zotero://select/library/items/JE7EMCBD)) ([pdf](zotero://open-pdf/library/items/X7TMLKQV?page=8&annotation=HWV4E53Z)) #5fb236
- - “Impact of context” ([“icse2025-paper1709”, p. 8](zotero://select/library/items/JE7EMCBD)) ([pdf](zotero://open-pdf/library/items/X7TMLKQV?page=8&annotation=G977A7K3)) #ffd400
- *The notion of context is not clear. It is not evident how the concepts to be matched get automatically distinguished from the rest of the concepts belonging to the metamodels of interest.*
- - “use of fragments” ([“icse2025-paper1709”, p. 8](zotero://select/library/items/JE7EMCBD)) ([pdf](zotero://open-pdf/library/items/X7TMLKQV?page=8&annotation=C4WEUVBF)) #ffd400
- - “Prompting for each concept individually improves accuracy significantly but increases cost.” ([“icse2025-paper1709”, p. 8](zotero://select/library/items/JE7EMCBD)) ([pdf](zotero://open-pdf/library/items/X7TMLKQV?page=8&annotation=H3EFMZXL)) #ffd400
- *The paper is not effective in explaining how such a prompting approach is done.*
- - “Concrete super-classes. We assume that concrete superclasses are not permitted. We designed our approach accordingly and refined the metamodels that have super-classes (e.g., UML2.0) by abstracting concrete super-classes. In fact, our method mandates mapping to a non-superclass by removing the super-class from the fragment if it is selected by the LLM.” ([“icse2025-paper1709”, p. 10](zotero://select/library/items/JE7EMCBD)) ([pdf](zotero://open-pdf/library/items/X7TMLKQV?page=10&annotation=HKN86PQQ)) #ffd400
- *Even though the authors recognize this as a potential threat to validity, they should discuss how frequent this occurs at least for the cases they have analyzed for the performed experiments.*
- ### [[Comments]]
- #.tabular
- ### Paper summary
- This paper presents the application of LLMs to deal with the problem of metamodel matching in model-driven engineering (MDE). Existing approaches rely on ontology and lexical databases, which face limitations in identifying semantic equivalences between metamodels with syntactic heterogeneity. The proposed approach, SMATCH-M-LLM, properly encodes source and target metamodels in prompts and aims to improve matching accuracy. According to the performed experiments, considering three baseline methods and ten cases, the proposed LLM-based method outperforms baseline approaches, showing superior F-measure scores.
- ### Strengths
- + The paper tackles a challenging problem of metamodel matching, an essential task in model management, with an innovative application of LLMs.
- + Experimental results demonstrate clear performance improvements over traditional methods, validating the effectiveness of the proposed approach.
- + The paper contributes to the field by introducing LLMs as a new method for metamodel matching, with promising results that open new research directions.
- ### Weaknesses
- - The process for constructing fragments and refining matches using LLMs lacks detailed illustrative examples, making it difficult to fully understand the flow of the approach.
- - Discussions on scalability, handling new domain-specific languages (DSLs), and generalizing the approach are underdeveloped, and the limitations of relying on GPT-4’s training data for DSLs are not addressed.
- - Several sections, particularly the explanation of algorithms and tables, need reworking for clarity and improved presentation.
- - The description of the motivational example (KM3 and DOT metamodels) is unclear and lacks sufficient depth. The choice of example itself is questionable as it does not seem to illustrate the problem effectively (see the detailed comments below).
- ### Detailed comments for authors
- Novelty: The novelty lies in the application of LLMs to the problem of metamodel matching. While this idea is promising and novel in its context, the novelty is somewhat undermined by the lack of clarity in the execution of the idea.
- Rigor: The process of deriving initial correspondences from element definitions is not fully transparent, and more information on the role of embeddings and cosine similarity would enhance rigor (see detailed comments below).
- Relevance: The topic is highly relevant to the field of MDE, where semantic matching and automation are key concerns. However, the paper could benefit from a deeper discussion on how it applies beyond the ten evaluated cases and how it can be generalized or adapted for other DSLs, particularly those that may not be part of LLM training data (see detailed comments below).
- Verifiability & transparency: Verifiability is a concern, as several core components of the approach (e.g., generation of element definitions and fragment construction) lack sufficient details for replication. Algorithm 1 and the generated concept definitions, for instance, should be introduced earlier and explained in greater depth. Providing running examples of actual prompts and embeddings created while following the process in Fig. 3 would increase transparency.
- Presentation: The presentation needs significant improvement. The paper's flow is interrupted by unclear sections and insufficient examples (see below).
- Detailed comments:
- **Motivational example**: The choice of KM3 and DOT as the example is unclear. As the authors know, KM3 is a DSL for specifying metamodels, whereas DOT is a metamodel for specifying graphical representations. Indeed, a KM32DOT model transformation exists to generate DOT diagrams from KM3 models automatically. Such a transformation was manually developed, and the encoded transformation rules cannot be considered semantic equivalences. A similar comment can be made for most cases, as shown in Table IV. The authors are confusing the problem of semantic metamodel equivalences with the automated generation of model transformations among heterogeneous metamodels. These are two different problems and the paper is supposed to address the first one. Addressing the latter requires reworking the paper and also considering different related work on the adoption of AI and NLP techniques to support the development of model transformations.
- **Section 3**: In general this section needs reworking because it is not clear. A running and explanatory example is needed to give an overview of the approach before giving the details of the single components shown in Fig. 3. For instance, it is not clear why the process includes the calculation of initial correspondences and then finalizes these after creating source and target fragments.
- **Fragment construction**: The paper briefly describes how fragments are created, but the need for the Louvain algorithm is not justified. Moreover, it is unclear how the resulting fragments aid the LLM process. An illustrative example of fragment input and output would helps.
- **Element matcher (sec 3.C)**: Even this part of representing fragments to LLM after having converted them into PlantUML is not clear. Users might want to match any potential couple of DSLs that might not have anything to do with UML. Moreover, in which part of Listing 2, PlantUML specifications are given? It is not clear from Listing 2.
- **Algorithm descriptions**: The paper references Listing 1 and Algorithm 1 without introducing them properly. This disrupts the flow and makes it difficult to understand how the LLM generates definitions or how cosine similarity is applied to match concepts. For instance, it is not clear how couple of concepts are selected for the measurement of their cosine similarity.
- **RQ2: Impact of context:** The notion of context is not clear. It is not evident how the concepts to be matched get automatically distinguished from the rest of the concepts belonging to the metamodels of interest.
- **Answer to RQ2:** *"Prompting for each concept individually improves accuracy significantly but increases cost."* The paper is not effective in explaining how such a prompting approach is done.
- **Scalability**: The notion of scalability is not clear. The authors mention using fragments to avoid overwhelming the LLM with large metamodels, but no clear indication is provided regarding the size the method can handle. The claim that large metamodels are involved is questionable, especially by looking at the cases considered for the experiments.
- **Handling new DSLs**: The paper does not address how the approach deals with metamodels from custom DSLs that may not appear in GPT-4’s training data. For instance, by looking at the generated definitions shown In Table II, it is clear that the considered metamodels are part of the training data. This represents a significant limitation that warrants discussion.
- **Human involvement**: It is unclear whether humans are involved at any stage in refining or verifying correspondences. Given the complexity of metamodel matching, some human oversight may be necessary, which is not discussed in the current draft.
- **Threats to validity**: The authors briefly mention the potential bias in LLMs, but this section should be expanded. How frequent are issues with concrete super-classes? What is the potential impact of iterating with an LLM that may hallucinate correspondences?
- Artifacts
- I have found no accompanying documentation or detailed guidelines to explain how to use the package that the authors made available and how to replicate the experiments shown in the paper.
- ### [[REVIEWS/Notes]]
- ### YELLOW CONCERNS
background-color:: yellow
- {{query (and [[ffd400]] [[icse2025-paper1709]] )}}
collapsed:: true
- ### ❓️Questions
- {{query (and [[question]] [[icse2025-paper1709]] )[[question]]}}
query-table:: true
query-properties:: [:block]