5.9 KiB
5.9 KiB
type:: REVIEWS tags:: year:: 2025 venue:: ICSE full-title:: MQ-LLM4DSL: Module and Quality Driven Domain-Specific Language Design with Large Language Models date-start:: 16-05-2025 - 19:31 date-submitted:: external-links:: status:: DONE deadline-submission:: file:: @MQ-LLM4DSL: module and quality driven domain-specific language design with large language models parent:: todoist:: https://app.todoist.com/app/task/737-mq-llm-4-dsl-module-and-quality-driven-domain-specific-language-design-with-6XXr6Px8QhrJ8mFc
- ### [[Highlights]]
- ### [[Comments]]
- #.tabular
- - ### Paper summary
- The paper presents MQ-LLM4DSL, a methodology for designing domain-specific languages (DSLs) by leveraging large language models (LLMs). The approach includes a four-step quality-driven process consisting of the following phases: pattern extraction from domain corpora, restructuring of functional modules, DSL design, and quality evaluation and improvement. The approach has been evaluated by comparing it with an existing baseline. According to the performed experiments, the produced DSLs are of higher quality than those produced with the considered baseline.
- - ### Strengths
- + The usage of LLMs for supporting the design of domain-specific languges is a relevant topic
- + The focus on quality aspects of DSL is also a strenght of the paper
- - ### Weaknesses
- - The overall presentation is vague and lacks concrete examples or case studies demonstrating the practicality and effectiveness of the approach.
- - The claims regarding automation and quality are not backed by sufficient empirical evidence or implementation detail.
- - Multiple minor but distracting presentation issues (e.g., repeated references, inconsistent spacing, unclear phrasing).
- - ### Detailed comments for authors
- Novelty: The paper is supposed to be novel with respect to the integration of quality-driven evaluation with LLM-based DSL generation, but it remains unclear how this differs from prior work such as Corpus2DSL, Model2DSL, or Ontology2DSL. In particular, there is no explicit comparison of the proposed approach with respect to those that fall under these three categories.
Rigor: This is a critical point of the work. The approach description remains at a conceptual or proposal level rather than offering a concrete, reproducible methodology. Assertions like "we have selected quality indicators" are not justified with methodological rigor or references. It is unclear how the indicators given in Table 2 were chosen and validated. Moreover, the connection between quality evaluations and actual design improvements is not well grounded. Moreover, it is not clear what kinds of DSL the approach can support (i.e., textual vs graphical).
Relevance: The problem addressed is relevant to the DSL and MDE communities. However, the lack of concrete examples and real DSL case studies makes it difficult to assess the practical relevance of the proposed methodology.
Verifiability & transparency: Details on how the proposed approach was implemented are missing. It is not clear how much of the approach has been realized in software. The experimental setup section is too generic. Readers are required to download the artifact for clarification. Thus, the paper needs to be improved to make it self-contained and explanatory.
Presentation: The paper suffers from repeated reference numbers, missing spaces before parentheses. Moreover, a motivating example is missing, especially in the presentation of the main methodology.
Additional issues:
- p. 2: "Corpus2Model2DSL" - Mentioned in text but not shown in Figure 1.
- p. 2: "Based on this... present results." - The paragraph is vague; a concrete example would help clarify what is being improved.
- p. 2: "Figure 2" - In my opinion, a feedback loop between the evaluation and design phases is missing from the approach.
- p. 3: "requirements, which is Expressiveness." - Singular focus on Expressiveness contradicts earlier mentions of other quality attributes.
- p. 3: "... select quality indicators that can provide information for DSL improvement..." - Needs rewording for clarity.
- p. 3: "3 Approach" - The section is vague and more like a project proposal. Examples and DSL cases are needed.
- p. 4: "Figure 4" - DSL development should allow refinement of requirements; currently presented as one-way process, even though I think that some unforeseen cases might even require the refinements of the initial requirements (thus a bidirectional process is required).
- p. 6: "Quality Evaluation and Improvement" - Quality checks should be included throughout the process, not only at the end. In particular, the approach is supposed to be quality-driven. Thus, I would expect to have quality checks in different stages of the process, instead of having quality evaluations only at the very end of the process. It is difficult to understand how improvements are guided with the aim of enhancing the quality of the DSL under development.
- Questions
- How does your approach concretely differ from existing methods such as Corpus2DSL, Model2DSL, and Ontology2DSL? Why have you not provided a comparative analysis or mapping that clearly highlights the added value of MQ-LLM4DSL over these prior approaches?
- Given that the proposed methodology is positioned as quality-driven, how are quality evaluations integrated into the DSL development process beyond the final step, and what concrete mechanisms (e.g., metrics, feedback loops, or tool support) guide iterative improvement of DSLs based on these evaluations?
- ### [[REVIEWS/Notes]]
- ### YELLOW CONCERNS
background-color:: yellow
- {{query (and [[ffd400]] [[icse2026-paper737]] )}}
collapsed:: true
- ### ❓️Questions
- {{query (and [[question]] [[icse2026-paper737]] )[[question]]}}
query-table:: true
query-properties:: [:block]