logseq/pages/@MQ-LLM4DSL%3A module and quality driven domain-specific language design with large language models.md at e3954f8a00a80e10957f4c8e632aff5baa6ec49b

Files

T

ddiruscio d9ea3d552d Auto saved by Logseq

2025-06-02 17:15:13 +02:00

16 KiB

Raw Blame History

tags:: #zotero date:: 2018 title:: @MQ-LLM4DSL: module and quality driven domain-specific language design with large language models item-type:: journalArticle original-title:: MQ-LLM4DSL: module and quality driven domain-specific language design with large language models language:: en library-catalog:: Zotero links:: Local library, Web library

Abstract
- Domain-Specific Languages (DSLs) are essential in software engineering, offering precise expression and excellent usability for specific domains, significantly improving software development efficiency. However, DSL design is challenging, requiring deep domain knowledge and language expertise to ensure quality. Mainstream DSL design methods use intermediate models like ontologies and metamodels to guide construction through manually defined rules, demanding substantial expert effort. Although some approaches transform ontologies into DSLs using explicit rules, ensuring full requirement retention remains a challenge. These methods primarily focus on the expressiveness of DSLs, while considerations for other quality aspects are insufficient. Some methods emphasize the evaluation of DSL during the design process and the improvement based on the evaluation results, incorporating this process as a part of the DSL design method. However, these methods do not provide clear improvement approaches based on specific evaluations. We propose the MQ-LLM4DSL method. This method utilizes the functional modules of domain corpus as an intermediate model for DSL design. It also provides specific improvement methods based on the core quality factors of the DSL and iteratively optimizes the quality of each step in the DSL design. Experimental results show that DSLs designed with MQ-LLM4DSL outperform those created using the baseline method, achieving an improvement: 22% in Expressiveness, 16% in Flexibility and 9% in EaseofLearn.
Attachments
- MQ-LLM4DSL_cleaned {{zotero-imported-file 5A6TMKFV, "MQ-LLM4DSL_cleaned.pdf"}}
- PDF {{zotero-imported-file DKM2YEJW, "2018 - MQ-LLM4DSL module and quality driven domain-specific language design with large language models.pdf"}}
Notes
- I'm reviewing a research paper and I took the following notes:
  
  Annotazioni
  
  (16/5/2025, 19:27:20)
  - “Although some approaches transform ontologies into DSLs using explicit rules, ensuring full requirement retention remains a challenge.” (“MQ-LLM4DSL: module and quality driven domain-specific language design with large language models”, 2018, p. 1) #5fb236
  - “Some methods emphasize the evaluation of DSL during the design process and the improvement based on the evaluation results, incorporating this process as a part of the DSL design method. However, these methods do not provide clear improvement approaches based on specific evaluations.” (“MQ-LLM4DSL: module and quality driven domain-specific language design with large language models”, 2018, p. 1) #5fb236
  - “Automatically generating high-quality DSLs from domain-specific corpora has long been an open research challenge, attracting significant interest from both academia and industry.” (“MQ-LLM4DSL: module and quality driven domain-specific language design with large language models”, 2018, p. 1) #5fb236
  - “domain corpora as the functional requirements for DSL design” (“MQ-LLM4DSL: module and quality driven domain-specific language design with large language models”, 2018, p. 1) #5fb236
  - “Other quality requirements consist of usability, flexibility, etc” (“MQ-LLM4DSL: module and quality driven domain-specific language design with large language models”, 2018, p. 1) #5fb236
  - “Corpus2DSL” (“MQ-LLM4DSL: module and quality driven domain-specific language design with large language models”, 2018, p. 1) #2ea8e5
  - “Model2DSL” (“MQ-LLM4DSL: module and quality driven domain-specific language design with large language models”, 2018, p. 1) #2ea8e5
  - “The presence of conceptual models allows designers to build precise results based on their understanding of the requirements, which also achieves traceability of the design and reduces complexity by breaking down tasks” (“MQ-LLM4DSL: module and quality driven domain-specific language design with large language models”, 2018, p. 1) #5fb236
  - “Unfortunately, these methods focus on discussing the rationality of conceptual models and lack specific steps for building models and transforming them into DSLs.” (“MQ-LLM4DSL: module and quality driven domain-specific language design with large language models”, 2018, p. 1) #5fb236
  - “Ontology2DSL” (“MQ-LLM4DSL: module and quality driven domain-specific language design with large language models”, 2018, p. 1) #2ea8e5
  - “[17, 33, 33]” (“MQ-LLM4DSL: module and quality driven domain-specific language design with large language models”, 2018, p. 1) #ff6666 Reference [33] occurs twice.
  - “Existing DSL design methods still require a significant amount of work from experts at different design stages.” (“MQ-LLM4DSL: module and quality driven domain-specific language design with large language models”, 2018, p. 1) #5fb236
  - “Corpus2Model2DSL” (“MQ-LLM4DSL: module and quality driven domain-specific language design with large language models”, 2018, p. 1) #ffd400 It's not shown in Fig. 1, isn't it?
  - “The above DSL design methods lack focus on improving DSL quality.” (“MQ-LLM4DSL: module and quality driven domain-specific language design with large language models”, 2018, p. 2) #5fb236
  - “lack approaches to improve DSL quality through quality evaluation.” (“MQ-LLM4DSL: module and quality driven domain-specific language design with large language models”, 2018, p. 2) #5fb236
  - “However, these methods do not provide clear improvement approaches based on specific evaluations” (“MQ-LLM4DSL: module and quality driven domain-specific language design with large language models”, 2018, p. 2) #5fb236
  - “two limitations of existing methods” (“MQ-LLM4DSL: module and quality driven domain-specific language design with large language models”, 2018, p. 2) #ffd400 Which one exactly?
  - “domain corpora to DSL design, and to ensure the expressiveness and other qualities of DSLs.” (“MQ-LLM4DSL: module and quality driven domain-specific language design with large language models”, 2018, p. 2) #5fb236
  - “MQ-LLM4DSL(Figure 2)” (“MQ-LLM4DSL: module and quality driven domain-specific language design with large language models”, 2018, p. 2) #ff6666 A space is missing before the open parenthesis. There are many occurrences of the same problem.
  - “Based on this, clear executable steps for designing DSLs are provided so that LLMs can easily understand and execute the steps and present results. By summarizing and analyzing existing DSL evaluation methods, we have selected quality indicators and evaluation methods that can guide DSL improvement. Then, combining the DSL design steps with the evaluation methods, we provide methods for evaluation-feedback to further improve DSL quality.” (“MQ-LLM4DSL: module and quality driven domain-specific language design with large language models”, 2018, p. 2) #ffd400 This description is vague and requires a concrete example to show what you plan to improve. A motivating and explanatory example would help understand the challenges the work aims to address.
  - “The method consists of the following four parts:” (“MQ-LLM4DSL: module and quality driven domain-specific language design with large language models”, 2018, p. 2) #5fb236
  - “1. Pattern Extraction: Use LLMs to extract entities, relationships, and other abstract concepts (referred to as elements) from domain corpora;” (“MQ-LLM4DSL: module and quality driven domain-specific language design with large language models”, 2018, p. 2) #2ea8e5
  - “2. Restructuring and Refining of the Functional Module Architecture: Use patterns as the initial functional modules, restructure the architecture based on the common parts of the patterns and the similar relationships among functional modules;” (“MQ-LLM4DSL: module and quality driven domain-specific language design with large language models”, 2018, p. 2) #2ea8e5
  - “3. Module DSL Design and Integration:” (“MQ-LLM4DSL: module and quality driven domain-specific language design with large language models”, 2018, p. 2) #2ea8e5
  - “M DSL” (“MQ-LLM4DSL: module and quality driven domain-specific language design with large language models”, 2018, p. 2) #ffd400 Is this a typo?
  - “Evaluation and Improvement:” (“MQ-LLM4DSL: module and quality driven domain-specific language design with large language models”, 2018, p. 2) #2ea8e5
  - “Figure 2:” (“MQ-LLM4DSL: module and quality driven domain-specific language design with large language models”, 2018, p. 2) #ffd400 According to the figure, it seems that the DSL quality evaluation process gives some input to the DSL development process. However, I would expect that there is also a connection on the other way round.
  - “quality during use and quality during maintenance.” (“MQ-LLM4DSL: module and quality driven domain-specific language design with large language models”, 2018, p. 2) #5fb236
  - “Therefore, we cannot improve the DSL based on the evaluation results of each quality indicator individually.” (“MQ-LLM4DSL: module and quality driven domain-specific language design with large language models”, 2018, p. 3) #5fb236
  - “we need to determine the relationship between various quality indicators and identify which quality assessments can guide DSL improvements” (“MQ-LLM4DSL: module and quality driven domain-specific language design with large language models”, 2018, p. 3) #5fb236
  - “requirements, which is Expressiveness.” (“MQ-LLM4DSL: module and quality driven domain-specific language design with large language models”, 2018, p. 3) #ffd400 Only expressiveness? It seems you wanted to focus on more than one requirement.
  - “Expressiveness” (“MQ-LLM4DSL: module and quality driven domain-specific language design with large language models”, 2018, p. 3) #5fb236
  - “Therefore, improvements to the DSL should prioritize Expressiveness. Then, consider quality indicators that have little to no impact on the existing Expressiveness during improvement. Finally, consider improvements that potentially affect Expressiveness. Based on this, select quality indicators that can provide information for DSL improvement. Following the aforementioned approach to filter existing DSL evaluation methods, we obtain the results(Table 2).” (“MQ-LLM4DSL: module and quality driven domain-specific language design with large language models”, 2018, p. 3) #ffd400 this is not clear, it needs rewording.
  - “MQ-LLM4DSL ,” (“MQ-LLM4DSL: module and quality driven domain-specific language design with large language models”, 2018, p. 3) #ff6666 extra space before ","
  - “3 Approach” (“MQ-LLM4DSL: module and quality driven domain-specific language design with large language models”, 2018, p. 3) #ffd400 This section is high level and it looks like more a project proposal than a developed approach presentation. I suggest revise the whole section (and also the other ones) with the aim of making the paper more concrete and robust. It is necessary to present explanatory examples and real DSL cases that can be used to make the paper more concrete. Currently the paper is about DSL design and development, and there is no "trace" of DSL in the paper.
  - “Table 2: DSL evaluation methods for quality improvement” (“MQ-LLM4DSL: module and quality driven domain-specific language design with large language models”, 2018, p. 4) #ffd400 The content of Table 2 is not properly supported by proper evidence. How have authors defined such quality indicators? I'm not convinced about them. For instance, many aspects are related to "Understanding" including the supporting tool of the considered DSL. You might have different implementations of the same DSL, and quality aspects may change. For instance, how would you cover graphical vs textual DSLs?
  - “Figure 4: MQ-LLM4DSL overview” (“MQ-LLM4DSL: module and quality driven domain-specific language design with large language models”, 2018, p. 4) #ffd400 In my opinion, the process cannot be only unidirectional. Also back-propagation should be supported. During the development of the DSL it can be required to refine requirements in order to add some unforeseen ones or refine existing ones.
  - “3.1.1 Pattern Extraction.” (“MQ-LLM4DSL: module and quality driven domain-specific language design with large language models”, 2018, p. 4) #2ea8e5
  - “3.1.2 Functional Module Architecture Restructuring and Refinement.” (“MQ-LLM4DSL: module and quality driven domain-specific language design with large language models”, 2018, p. 4) #2ea8e5
  - “Refinement is then performed on the existing functional module architecture, further subdividing and designing modules that represent large, complex, or insufficiently clear functions, based on the existing architecture.” (“MQ-LLM4DSL: module and quality driven domain-specific language design with large language models”, 2018, p. 4) #5fb236
  - “3.1.3 Module DSLs Design and Integration.” (“MQ-LLM4DSL: module and quality driven domain-specific language design with large language models”, 2018, p. 4) #2ea8e5
  - “3.1.4 Quality Evaluation and Improvement.” (“MQ-LLM4DSL: module and quality driven domain-specific language design with large language models”, 2018, p. 5) #2ea8e5
  - “Quality Evaluation and Improvement” (“MQ-LLM4DSL: module and quality driven domain-specific language design with large language models”, 2018, p. 6) #ffd400 The approach is supposed to be quality driven, thus I would expect to have quality checks in different stages of the process, instead of having quality evaluations only the very end of the approach. It is difficult to understand how improvement are guided with the aim of improving the quality of the DSL under development. Moreover, it is not clear if the process shown in Fig. 4 has been actually implemented and tool supported.
  - “4.5 Detailed Experimental Setup” (“MQ-LLM4DSL: module and quality driven domain-specific language design with large language models”, 2018, p. 8) #ffd400 *Also this section needs details. It is not clear how the experiments have been actually executed. Currently, readers are supposed to download the package from zenodo and figure out on their own how the approach has been actually developed, what are the software components related to the what has been presented in the paper etc. It is necessary to make the paper self-explanatory and then readers taht want know more can explore the code. Not the vice versa.

  COnsider that those that are tagget with #5fb236 are just highlights, those that are tagged with #e56eee and #a28ae5 are imporant sentences. Please pay attention instead to the notes that are tagged with #ffd400. Those that are tagged with #ff6666 are typos or errors. Could you please draft a review by organizing it as follows: 
  
  SUMMARY: Just a few sentence to summarize the work 
  
  STRENGHTS:
  
  WEAKNESSES:
  
  COMMENTS: Organize the notes with respect to the following criteria:
  
  - 
  `Novelty`
  
  - 
  `Rigor`
  
  - 
  `Relevance (of the contribution)`
  
  - 
  `Verifiability and Transparency`
  
  - 
  `Presentation`
  
  And then add a Detailed Comments section to report the notets that contain issues or typos.

16 KiB Raw Blame History

Attachments

Notes

Annotazioni

16 KiB

Raw Blame History