logseq/pages/@icse2025-paper1709.md

tags:: #zotero title:: @icse2025-paper1709 item-type:: document original-title:: icse2025-paper1709 links:: Local library, Web library

• Attachments

  • PDF {{zotero-imported-file X7TMLKQV, "icse2025-paper1709.pdf"}}

• Notes

  • Annotazioni

    (29/9/2024, 14:38:38)

    • “semantic matchers and alternatives to baseline methods for metamodel matching. However, metamodels can be large, which can overwhelm LLMs” (“icse2025-paper1709”, p. 1) #00b036

    • “initial mapping derived from their elements’ definitions.” (“icse2025-paper1709”, p. 1) #f0ff00 How is this obtained?

    • “Industry standards also drive migration for better integration and tool use, such as legacy proprietary models to AUTOSTAR [5], [6]” (“icse2025-paper1709”, p. 1) #00b036

    • “similarities with metamodels” (“icse2025-paper1709”, p. 1) #00b036

    • “KM3 and DOT” (“icse2025-paper1709”, p. 2) #f0ff00 I'm not sure this is the right motivating example.....

    • “When we prompt GPT-4 using the providing only specific parts of the metamodels, Node is correctly identified as a correspondence for DataType.” (“icse2025-paper1709”, p. 2) #f0ff00 That table does not show the matched elements. For instance oncernig the third row, Instead of showing only Node, I would also shows the Data Type of the Dot metamodel.

    • “large” (“icse2025-paper1709”, p. 2) #f0ff00 Large? They are very small.

    • “III. THE SMATH-M-LLM APPROACH” (“icse2025-paper1709”, p. 2) #ffd400 In general this section needs reworking because it is not clear. A running and explanatory example is needed to give an overview of the approach before giving the details of the single components shown in Fig. 3.

    • “prompting them with relevant fragments of the source and target metamodels.” (“icse2025-paper1709”, p. 2) #5fb236

    • “Fig. 3. SMATH-M-LLM” (“icse2025-paper1709”, p. 3) #ffd400 It's not clear why the process includes the calculation of initial correspondences and then finalize these after creating source and target fragments.

    • “components SMATH-M” (“icse2025-paper1709”, p. 3) #ff4400 Of

    • “definitions’ similarity.” (“icse2025-paper1709”, p. 3) #f0ff00 What tool is exploited in his phase?

    • “Fragment Constructor partitions the source metamodel into fragments using a community detection algorithm, it creates an initial target fragment for each source fragment by identifying the target concepts that correspond to the source concepts within the relevant fragment.” (“icse2025-paper1709”, p. 3) #f0ff00 The needs for this component and how it works should be elaborated.

    • “In our work, the stopping criteria are satisfied if the correspondences are non-abstract concepts and all their features are presented in the target fragment. In the following sections, we describe each component in detail and demonstrate its application to the motivational example.” (“icse2025-paper1709”, p. 3) #f0ff00 The process is not clear. Maybe an explanatory example is needed.

    • “Definitions,” (“icse2025-paper1709”, p. 3) #f0ff00 Which one?

    • “include more general roles, namely super-type and container concepts.” (“icse2025-paper1709”, p. 3) #f0ff00 Explain.

    • “represented by their direct super-type.” (“icse2025-paper1709”, p. 3) #00b036

    • “This is represented by lines 1-2 in Algorithm 1.” (“icse2025-paper1709”, p. 3) #f0ff00 We are still missing the description of listing 1 and you already talk about algorithm 1 which has not been yet introduced?

    • “Applying this step to the metamodels KM3 and DOT (see Section II) results in selecting five representative concepts out of 16 concepts for KM3 (LocatedElement, ModelElement, Classifier, TypedElement, StructuralFeature) and eight out of 26 concepts for DOT (Label, Compartment, GraphElement, Arc, Shape, Nodelike, NodeShape, ComplexNodeShape). Notably, DataType and Node are excluded from the representative sets because they do not meet the specified criteria of being a super-type or a container. Additionally, although Package and Class are container concepts in KM3, they are not included in the representative set because they are already represented by Classifier and ModelElement, respectively. Furthermore, the contained element StructuralFeature is part of the representative set, which negates the need to include Class separately.” (“icse2025-paper1709”, p. 3) #f0ff00 By looking at these concepts I don't see sets of concepts that should be matched. Dot is related to graphical representations, km3 about metanodels why this should be matched without specifying the need of doing so? In that specific case, dot can be used to specify the graphical representation of metanodels (ie km3). Such a goal of the match is missing and I do not see how this is inferred (at least so far).

    • “generate definitions for the selected concepts to enrich the matching process.” (“icse2025-paper1709”, p. 3) #f0ff00 I see where you want to go, and it can make sense, let's see

    • “These definitions offer richer semantic information about the concepts.” (“icse2025-paper1709”, p. 3) #ffd400 This can be very domain specific and potentially not part of the training data (I'm thinking for instance to custom domain specific languages that are not part of the training data.

    • “Algorithm 1.” (“icse2025-paper1709”, p. 3) #f0ff00 I suggest to introduce this properly.

    • “TABLE II CONCEPTS AND GENERATED DEFINITIONS” (“icse2025-paper1709”, p. 4) #f0ff00 This means that the concepts should have appeared in the training data. What about new dsl?

    • “Table II shows examples of definitions generated by GPT-4 for some concepts in KM3 and DOT metamodels.” (“icse2025-paper1709”, p. 4) #ffd400 From the shown definition it is probable that the considered metamodels are part of the training data. See my previous comment.

    • “we represent them as continuous dense vectors,” (“icse2025-paper1709”, p. 4) #f0ff00 Can you show an example?

    • “we select the most similar representative target concept based on the highest cosine similarity score (lines 8-10). This curated list of concept correspondences (concept correspondences in Algorithm 1) constitutes the final output of this component.” (“icse2025-paper1709”, p. 4) #f0ff00 How many? What's the acceptance threshold?

    • “The task of refining the initial correspondences to achieve exact matches is done by the subsequent components.” (“icse2025-paper1709”, p. 4) #f0ff00 No humans in the loop?

    • “measure cosine similarity” (“icse2025-paper1709”, p. 4) #ffd400 Between what? It is not clear how couples of concepts are selected for doing such a measurement.

    • “The Fragment Constructor component is responsible for preparing pairs of source and target fragments, with the pairing based on the list of correspondences identified by the Concept Matcher.” (“icse2025-paper1709”, p. 4) #ffd400 Also in this cases it is needed to present an illustrative example to explain the steps of the process.

    • “Source Fragmentation” (“icse2025-paper1709”, p. 4) #ffd400 You mix activities (e.g., pre-processing, source fragmentation) with artifacts (e.g. target fragment)

    • “after constructing the fragments using Louvain, we ensure that relevant representative concepts are included in each fragment.” (“icse2025-paper1709”, p. 4) #5fb236

    • “Before providing the fragments to the LLM, we convert them into PlantUML format [26]. This format is chosen because it is concise, and it is widely recognized, thereby increasing the likelihood that LLMs will interpret it accurately” (“icse2025-paper1709”, p. 6) #ffd400 Even this part of representing fragments to LLM after having converted them into PlantUML is not clear. User might want to match any potential couple of DSLs that might not have nothing todo with UML. Moreover, in which part of Listing 2, PlantUML specifications are given? It is not clear from Listing 2. n

    • “baseline matching techniques” (“icse2025-paper1709”, p. 6) #a28ae5

    • “seven metamodels” (“icse2025-paper1709”, p. 6) #a28ae5

    • “three established baseline methods” (“icse2025-paper1709”, p. 7) #a28ae5

    • “Ontology Matching Methods” (“icse2025-paper1709”, p. 7) #5fb236

    • “Lexical Database Methods.” (“icse2025-paper1709”, p. 7) #5fb236

    • “LLM-based Methods.” (“icse2025-paper1709”, p. 7) #5fb236

    • “semantically equivalent from its respective transformation rules provided” (“icse2025-paper1709”, p. 7) #ffd400 This does not mean that concepts are semantically equivalent!!!!

    • “The proposed LLM-based approach significantly outperforms baseline semantic matchers using ontology and lexical databases, indicating its potential to replace these methods with superior accuracy in identifying correspondences, addressing the known challenges” (“icse2025-paper1709”, p. 8) #5fb236

    • “Impact of context” (“icse2025-paper1709”, p. 8) #ffd400 The notion of context is not clear. It is not evident how the concepts to be matched get automatically distinguished from the rest of the concepts belonging to the metamodels of interest.

    • “use of fragments” (“icse2025-paper1709”, p. 8) #ffd400

    • “Prompting for each concept individually improves accuracy significantly but increases cost.” (“icse2025-paper1709”, p. 8) #ffd400 The paper is not effective in explaining how such a prompting approach is done.

    • “Concrete super-classes. We assume that concrete superclasses are not permitted. We designed our approach accordingly and refined the metamodels that have super-classes (e.g., UML2.0) by abstracting concrete super-classes. In fact, our method mandates mapping to a non-superclass by removing the super-class from the fragment if it is selected by the LLM.” (“icse2025-paper1709”, p. 10) #ffd400 Even though the authors recognize this as a potential threat to validity, they should discuss how frequent this occurs at least for the cases they have analyzed for the performed experiments.