3.2 KiB
3.2 KiB
type:: REVIEWS tags:: year:: 2026 venue:: full-title:: A Model-Driven Approach To Support The Understanding Of Machine Learning Pipelines date-start:: 04-02-2026 - 10:16 date-submitted:: external-links:: status:: DOING deadline-submission:: file:: /zotero parent:: todoist:: https://app.todoist.com/app/task/14-nicolas-lacroix-mireille-blay-fornarino-philippe-collet-frederic-precioso-and-6frhCMhFhcFm3hPc
- ### [[Comments]]
- SUMMARY: This paper is about the critical challenge of keeping the consistency between a metamodel and its corresponding grammar during language evolution in model-driven engineering settings. The authors propose a method based on Large Language Models to automate grammar adaptation by learning from historical version changes, specifically by comparing an original generated grammar with its manually adapted counterpart to propagate those changes to a newly evolved version. The study evaluates three different models (i.e., Claude Sonnet 4.5, ChatGPT 5.1, and Gemini 3) against a rule-based baseline using six real-world domain-specific languages and a longitudinal case study of the QVTo language across four official versions. Results show that for small-to-medium scale grammars, the LLM-based approach achieves perfect rule-level adaptation consistency and output similarity, outperforming rule-based methods in complex scenarios such as those involving syntactic predicates or order-insensitive attribute combinations. However, the research also identifies that LLMs struggle with large-scale grammars like EAST-ADL, exhibiting systematic omissions despite having a sufficient context window.
- COMMENTS: The paper is about a critical problem, which has been investigated for years by the modeling and programming communities. I enjoyed reading the paper, which is well-written and structured. I liked the empirical demonstration of how LLMs compare to traditional rule-based tools. While rule-based approaches rely on predefined transformation rules that often fail when encountering nested optionality or complex backtracking structures, the LLM-based method successfully captures the underlying design intent by observing historical examples.
I have only a few suggestions for improvements:
-
- While the authors specify using the web interface of Claude Sonnet 4.5 for initial refinements, providing more detail on the automation (if any) of the subsequent cross-model evaluations would enhance the reproducibility of the experiments
- Additionally, the paper would benefit from explicitly summarizing the identified boundaries into actionable guidelines, clearly delineating the specific grammar constructs where LLMs are highly recommended versus the scale thresholds where they begin to fail. Such a recap would bridge the gap between theoretical findings and practical software engineering applications.
- The following sentence in the introduction: "Results show that on the 72 test set DSLs, all three LLMs achieved 100% Rule-level Adap- 73 tation Consistency (RAC) and 100% output similarity, while 74 the rule-based approach achieved 84.21% RAC ..." can be confusing because the metric name ("Rule-level...") sounds similar to the "rule-based" approach. Finding an alternative way to solve the confusion would help.