19 KiB
19 KiB
type:: REVIEWS
tags:: 💺EditoringChairing
year:: 2025
venue:: IEEE SOFTWARE
full-title::
date-start:: 23-12-2025 - 15:34
date-submitted::
external-links::
status:: DONE
deadline-submission::
file:: 
- ### [[Comments]]
- **Final decision**
- Reject
- Reviewers agree that the paper is about a relevant and interesting problem. However, the paper cannot be accepted because of several issues that can be summarized as follows:
- The main message of the paper is not effective because it fails to connect with the existing work on model and DSL evolution, and relies heavily on a specific DSL family. It is unclear whether the specificities of the considered languages can be a source of bias or affect the generalizability of the discussion.
- Following the previous point, the lack of novelty makes it challenging to understand the general and broader implications beyond the specific case discussed. The lessons learned section tries to generalize the findings, but it is not convincing.
- The evaluation is informal and lacks details. I understand the page constraint plays a role here; however, reviewers identified overstatements without sufficient evidence.
- Technical and conceptual issues are identified, including inaccuracies in the MDE tool discussion and insufficient justification for using LLMs over traditional methods in specific tasks.
-
- **Preliminary comments from those of the reviewers**
- Across the three reviews, several converging criticalities emerge. First, the paper’s **positioning and core message are unclear and potentially misleading**: the link to *DSL evolution* is weak, the contribution is tightly bound to a single DSL family (USchema/Athena–Orion), and the paper risks suggesting that probabilistic LLMs can replace well-established, deterministic DSL tooling without adequately discussing limitations, risks, or boundaries. Second, the **novelty and contextualization are insufficient**, with limited engagement with the substantial literature on DSL/model co-evolution and related MDE approaches, making it hard to assess what generalizes beyond this specific experience. Third, the **evaluation is informal and under-specified**: claims are often too strong given the lack of methodological detail, limited scale, absence of analysis of non-determinism, and no clear quantification of effort reduction. Finally, there are **technical and conceptual issues**, including inaccuracies in the discussion of MDE tooling, unclear justification for using LLMs over traditional approaches in some tasks, and missing discussion of key aspects such as data migration, reliability, and broader societal or ecological implications.
- > [[gpt3]]
The text highlights several key criticisms from three reviews. Firstly, the paper's main message is unclear and may be misleading, as it fails to effectively connect to DSL evolution and relies heavily on a specific DSL family. Secondly, the lack of novelty and context in the paper makes it challenging to understand its broader implications beyond the specific case discussed. Thirdly, the evaluation is deemed informal and lacks specific details, leading to overstatements without sufficient evidence. Lastly, technical and conceptual issues are identified, such as inaccuracies in MDE tool discussion and insufficient justification for using LLMs over traditional methods in certain tasks.
- **REVIEWER 1** / REJECT
- The paper is generally well-written and sound, and it addresses an interesting and timely topic.
It has however a few important problems that hamper my acceptance.
- The paper claims to evaluates GPT-4o in DSL evolution (in the title and throughout the paper). I had some difficulties understanding that it actually addresses two (very different) classes of tasks: tasks for DSL users (mainly script generation) and DSL developers (mainly script transformation). The link with DSL evolution is not clear, and feels forced.
- While the paper is quite convincing on the tasks for DSL users, I think it is deeply misleading on the tasks for DSL developers. It seems to promote the idea that LLMs should be used to replace traditional DSL tooling like compilers or translators. It completely omits discussing all the drawbacks of such idea. Replacing deterministic translators with probabilistic tools like LLMs would require a much more extensive experimentation. Ecological and societal issues are never mentioned.
- As a less critical issue, the paper should better highlight the peculiarities of the two considered DSLs. While on the surface these DSLs seem very similar to other languages (that the LLM "understands" out-of-the-box) I suppose that they have some specific features that need to be taught the LLM. They should be detailed.
- Finally, reference [1], that contributes to motivating the paper, is just a very short blog post, and does not contain much evidence of the claim.
- In summary, while the topic is interesting, the paper's core message may be misleading to practitioners, and this hampers in my opinion the paper's publication.
- **REVIEWER 2** / MAJOR
- ----------------------------------------
Summary of the submission.
----------------------------------------
The paper presents an experience report on using GPT-4o to support the evolution and use of two domain-specific languages from the USchema family: Athena (for database schema specification) and Orion (for schema evolution). The article proposes a four-step prompting strategy to teach the LLM how to generate DSL scripts and translate them to and from various database technologies. Through two validation experiments, the paper reports that GPT-4o produced syntactically correct and semantically valid outputs in nearly all cases, with only minor issues. As such, the paper argues that this approach can reduce the manual effort traditionally required to implement and maintain complex model-to-text transformations. The paper concludes with lessons learned about the strengths and limitations of combining LLMs with DSL engineering.
----------------------------------------
Detailed comments
----------------------------------------
The paper describes an experience report on the use of large language models to support the evolution of domain-specific languages, focusing on two DSLs within the USchema family. Overall, I found the article quite interesting and divulgative enough to justify a submission to IEEE Software. I appreciated the idea of reporting an experience-driven study to discuss the broader theme of how LLMs can be integrated into DSL engineering workflows. The lessons learned are significant for the community and, more generally, the paper presents insights that are timely and relevant for researchers and practitioners exploring hybrid AI-assisted software development processes. The article effectively reports on the opportunities and challenges of applying LLMs to non-trivial engineering tasks; its form is clear and accessible to a broad audience. So, in this sense, I believe the article is aligned with the objectives of the journal and has the potential to represent a valuable contribution.
At the same time, some aspects should be made more explicit or clarified. More specifically, let me report my major concerns in the following - please note that these issues are listed in no particular order of severity.
- (1) In the 'Evaluation and Results' section, the article describes the main findings coming from the two validation experiments discussed. While reading this section, I found the claims to be a bit too bold with respect to what is actually reported. In particular, the paper frequently uses expressions such as "complete correctness in all schema transformations" or "full correctness" across multiple target languages. However, the section does not provide enough methodological detail to support such strong claims.
- For instance, the paper does not specify (1) how correctness was operationalized (e.g., syntactic vs. semantic correctness criteria, classification of errors, acceptance thresholds); (2) how many test cases were used per transformation direction, nor whether these cases cover the full combinatorial variability of the DSL features; (3) how independence between training examples and evaluation cases was ensured, since few-shot examples may inadvertently encode the exact mapping rules being tested.
- I fully understand that these details may have been omitted because of the strict page limitation constraints imposed by the journal. Nonetheless, I would still suggest to briefly summarize the evaluation procedures to better contextualize the reported results. To save some space, the article may consider combining the last two sections (Lessons Learned and Conclusion) together.
- (2) As far as I can tell, the DSLs evaluated (Athena and Orion) belong to the same USchema family, which may make generalization easier than for unrelated DSLs. This is basically due to the very nature of the experience, which targets cases addressed in previous research. Yet, a more cautious formulation of the claims would make the contribution stronger. In this sense, I would just ask to more explicitly acknowledge that the experience is limited to cases belonging to the same USchema family and that, for this reason, the considerations reported may not fully generalize to other DSLs.
- (3) I appreciated that, in Figure 2, the article reports the prompting and validation strategy. However, some aspects of the prompt strategy are not explicitly discussed. In particular, the article does not address the key concern of LLM non-determinism and how this affects the reliability of the proposed workflow: LLMs can produce different outputs across runs even when using the same prompt, depending on temperature, sampling strategy, and internal stochasticity. As such, the paper should clearly (and briefly) discuss (1) whether the experiments have been conducted in a controlled environment (e.g., by ixing temperature, top-p, or other decoding parameters); (2) whether the reported successful transformations reflect single-run outputs or consistent results across multiple executions; (3) how often incorrect or inconsistent outputs occurred during experimentation, and whether prompt refinements were needed to stabilize behavior. I honestly believe that these details would be essential to assess the overall soundness of the conclusions drawn in the experience report.
- (4) Similar considerations may be reported when considering the effort reduction discussion. While the paper convincingly argues that the LLM-based workflow avoids the need to manually implement and maintain M2T transformations, the evidence provided is largely qualitative and does not fully report the magnitude of effort saved. For example, the article reports the lines of code of traditional transformations and the complexity of certain SCO mappings, but it does not quantify the actual prompting effort, the number of iterations required to achieve stable outputs, or the time spent in validation and correction. These aspects should be somehow accounted in the effort analysis, as they are preliminary activities to reach a level where the LLM may actually be exploited.
- So, in conclusion, I think the paper represents a nice contribution with effective and timely insights for the research community. The points above are relatively minor, assuming they simply reflect clarifications that were not included in the original manuscript due to space limitations. My recommendation is that the paper undergoes a revision to incorporate these clarifications.
- **REVIEWER 3** / Reject
- This article reports on an experience using large language models (LLMs) to support the evolution of two domain-specific languages (DSLs), Athena and Orion, by means of a structured prompting strategy. The paper argues that LLMs and DSLs are complementary: LLMs reduce engineering effort, while DSLs provide the formal backbone needed to ensure correctness and maintainability. The topic is timely and relevant for the IEEE Software readership, and the paper is generally well written and easy to follow.
However, while the paper presents an interesting experience report, several conceptual, methodological, and positioning issues significantly limit the clarity and strength of its contribution. In its current form, the paper lacks adequate contextualization within the literature on model and DSL co-evolution, provides an unclear justification for the use of LLMs over traditional MDE techniques, and presents an evaluation that is informal and insufficiently grounded.
Strengths
The paper addresses a timely topic at the intersection of DSL engineering and LLM-based automation.
The experience-based nature of the article aligns well with the IEEE Software audience.
The paper is generally readable and written in an accessible style.
Major Concerns
A major limitation of the paper is the lack of contextualization with respect to the extensive body of work on model and DSL co-evolution. The paper frames the contribution mainly as an application of LLMs to DSL evolution. However, it does not sufficiently relate this to existing research on co-evolution between models, metamodels, transformations, and instances.
Several recent and closely related works are not discussed, including:
Kebaili et al. (2024) [1], which empirically studies the use of LLMs for metamodel and code co-evolution.
Zhang et al. (2025) [2], which explicitly investigates LLM support for co-evolution between DSL definitions and instances.
Moreover a recent sutdy [3] investigated the usage of LLMs in various MDE tasks.
Without positioning the proposed approach with respect to this literature, it remains unclear what is novel beyond applying LLMs to a specific DSL family.
The paper's contribution is difficult to assess. The study is tightly coupled to a specific DSL family (Athena/Orion) and a specific data modeling context, yet the paper does not clearly articulate:
which insights generalize beyond this particular notation,
which lessons are specific to the USchema-based DSLs,
and how the approach compares to existing DSLs and modeling languages that already support schema evolution and persistence-layer abstractions.
There is a substantial body of work on modeling languages and frameworks for persistence, schema evolution, and query adaptation (e.g., Fink et al., 2020) [4] that is not discussed, nor are the differences between those approaches and Athena/Orion clarified.
The sidebar explaining how DSL generators are built in MDE contains several technical inaccuracies and inconsistencies:
Acceleo is presented as an example of a model-to-model (M2M) transformation language, whereas it is primarily a model-to-text (M2T) template-based language.
The statement that generators “often consist solely of an M2T transformation when the mapping is not complicated” is misleading; M2T transformations can be highly complex and involve sophisticated mappings.
The overall explanation risks confusing non-expert readers and oversimplifying well-known MDE practices.
I recommend either revising this sidebar substantially for technical accuracy or removing it altogether.
Box 1 lists several automated tasks (e.g., generating SQL, CQL, or MongoDB schemas from Athena scripts) that correspond to classic M2T transformations. It is not clearly justified why LLMs are preferable to established template-based approaches in these cases. As presented, the use of LLMs sometimes feels like “using a sledgehammer to crack a nut.”
The paper would benefit from a clearer discussion of:
which tasks genuinely benefit from LLM-based automation,
which tasks are already well supported by traditional MDE techniques,
and where the boundary lies between the two.
While the paper introduces a four-step prompting strategy, it does not explain how these steps were identified, nor how each step contributes to the final outcome. The strategy is presented as a design choice rather than as the result of a systematic process.
In particular:
- There is no analysis of alternative prompting strategies (e.g., few-shot vs. grammar-only, with or without documentation).
- The impact of each step on correctness or coverage is not evaluated.
- An ablation study would be highly beneficial to understand which components of the strategy are essential.
Moreover, it is unclear why the authors used ChatGPT instead of other LLMs (even freely available). This selection should be discussed throughout the paper.
Without this, the prompting strategy remains anecdotal rather than principled.
Although a rigorous evaluation is not mandatory for an IEEE Software article, the evaluation section, as presented, is informal and difficult to interpret:
The goals of the two experiments are not clearly defined.
The contribution of each experiment to validating the approach is unclear.
The reported examples are relatively simple (often fewer than ten entities), making it hard to assess how the approach scales to realistic evolution scenarios.
As a result, the evaluation does not convincingly demonstrate the challenges of real-world model and DSL evolution, nor the advantages of the proposed approach in such settings.
It is unclear whether the approach supports data migration in addition to schema evolution. While Orion generates DDL and DML commands, the paper does not explicitly discuss:
whether data consistency is preserved,
how data migration is validated,
or what guarantees are provided in practice.
Given that data migration is a central concern in schema evolution, this omission is significant.
Minor Comments:
The GitHub repository is referenced late in the paper and should be introduced earlier.
The repository contents mix English and Spanish examples, which reduces the performance of the training part.
Some claims in the text (e.g., the number of entities in examples) appear inconsistent with the repository.
Minor editorial issues (dates, phrasing, and references) should be addressed.
Overall Recommendation:
The paper addresses a relevant topic and reports an interesting practical experience. However, the lack of proper contextualization, unclear contribution, technical inaccuracies, and weak evaluation substantially limit its current impact.
[1] Z. K. Kebaili, D. E. Khelladi, M. Acher, and O. Barais, “An empirical study on leveraging LLMs for metamodels and code co-evolution,” Journal of Object Technology, vol. 23, no. 3, pp. 1–14, 2024.
[2] W. Zhang, R. Hebig, and D. Strüber, “Leveraging LLMs to support co-evolution between definitions and instances of textual DSLs,” 2025.
[3] Di Rocco, J., Di Ruscio, D., Di Sipio, C. et al. On the use of large language models in model-driven engineering. Softw Syst Model 24, 923–948 (2025). https://doi-org.univaq.idm.oclc.org/10.1007/s10270-025-01263-8
[4] J. Fink, M. Gobert, and A. Cleve, “Adapting Queries to Database Schema Changes in Hybrid Polystores,” in Proc. IEEE 20th Int. Working Conf. on Source Code Analysis and Manipulation (SCAM), 2020, pp. 127–131.
-
