logseq/pages/@MODELS_2025_paper_100.md

tags:: [[#zotero]]
title:: @MODELS_2025_paper_100
item-type:: [[document]]
original-title:: MODELS_2025_paper_100
language:: en
links:: [Local library](zotero://select/library/items/MCTU2L36), [Web library](https://www.zotero.org/users/1039502/items/MCTU2L36)

- ### Attachments
	- [PDF](zotero://select/library/items/3AQWFSFZ) {{zotero-imported-file 3AQWFSFZ, "MODELS_2025_paper_100.pdf"}}
- ### Notes
	- I'm reviewing a research paper and I took the following notes:
	  
	  # Annotazioni 
	   (22/5/2025, 11:21:13)
	  
	  - “Integrating AI Development Process” (“MODELS_2025_paper_100”, p. 1) #5fb236
	  
	  - “current practices often separate the development of ML and non-ML components, resulting in fragmented process development.” (“MODELS_2025_paper_100”, p. 1) #5fb236
	  
	  - “establishment of effective feedback loops” (“MODELS_2025_paper_100”, p. 1) #5fb236
	  
	  - “The framework aims to improve the management of AI system development by providing a unified and formalized representation of its constituent processes.” (“MODELS_2025_paper_100”, p. 1) #a28ae5
	  
	  - “We demonstrate the feasibility of our framework through a case study, illustrating how it facilitates more systematic, transparent, and traceable process management in AI system development.” (“MODELS_2025_paper_100”, p. 1) #a28ae5
	  
	  - “The integration of ML into software products introduces new software engineering (SE) challenges and intensifies existing ones” (“MODELS_2025_paper_100”, p. 1) #e56eee
	  
	  - “The SE challenges introduced by ML integration largely stem from the need to coordinate multiple engineering pipelines, including system and software engineering, data engineering, and ML training” (“MODELS_2025_paper_100”, p. 1) #e56eee
	  
	  - “However, integrating such heterogeneous artifacts into a cohesive development process remains inherently challenging” (“MODELS_2025_paper_100”, p. 1) #5fb236
	  
	  - “These integration challenges are further exacerbated by current AI development practices, which frequently treat the development of ML and non-ML components in isolation [6].” (“MODELS_2025_paper_100”, p. 1) #5fb236
	  
	  - “In order to address the challenges stemming from the lack of co-development between ML and non-ML components, we introduce AIEngOrchestrator, a model-based framework that orchestrates AI system development through an artifactcentric lens.” (“MODELS_2025_paper_100”, p. 1) #ffd400
	  *SO the focus of the orchestrator is the AI system development? Not the system being executed?*
	  
	  - “Central to the framework is a domain-specific modeling language (DSL), GSM4SE4AI, an artifact-centric process modeling language that enables precise modeling of both the structural and behavioral aspects of AI development artifacts.” (“MODELS_2025_paper_100”, p. 1) #a28ae5
	  
	  - “it establishes a unified structure that makes relationships between artifacts clear and manageable.” (“MODELS_2025_paper_100”, p. 1) #ffd400
	  *manageable for what?*
	  
	  - “This integration offers end-to-end traceability, concurrent change propagation within and across development pipelines, and effective change impact analysis. A case study is used to demonstrate the applicability of the proposed approach.” (“MODELS_2025_paper_100”, p. 1) #ffd400
	  *What is the final goal of the approach?*
	  
	  - “key principles from Software Engineering for AI” (“MODELS_2025_paper_100”, p. 1) #5fb236
	  
	  - “core concepts in process modeling” (“MODELS_2025_paper_100”, p. 1) #5fb236
	  
	  - “These systems are typically characterized by their reliance on machine learning (ML) models or other AI techniques to achieve their functionality.” (“MODELS_2025_paper_100”, p. 1) #5fb236
	  
	  - “ML Module: Thi” (“MODELS_2025_paper_100”, p. 2) #2ea8e5
	  
	  - “classification, prediction, recommendation, or anomaly detection” (“MODELS_2025_paper_100”, p. 2) #5fb236
	  
	  - “Non-ML Module:” (“MODELS_2025_paper_100”, p. 2) #2ea8e5
	  
	  - “Software Engineering Pipeline is responsible for the specification, design, implementation, and testing of the non-ML module” (“MODELS_2025_paper_100”, p. 2) #5fb236
	  
	  - “Data Engineering Pipeline focuses on the collection, curation, and preprocessing of data required for training machine learning models.” (“MODELS_2025_paper_100”, p. 2) #5fb236
	  
	  - “ML Training Pipeline handles model selection, training, and validation.” (“MODELS_2025_paper_100”, p. 2) #5fb236
	  
	  - “System Engineering Pipeline oversees the integration of all system components—both ML and non-ML—as well as their overall verification and validation” (“MODELS_2025_paper_100”, p. 2) #5fb236
	  
	  - “Despite this interdependence, these pipelines are often developed in isolation with limited coordination” (“MODELS_2025_paper_100”, p. 2) #a28ae5
	  
	  - “This fragmentation disrupts the overall development lifecycle, primarily by hindering traceability, which in turn leads to cascading errors, costly rework, and increased integration risks. These challenges highlight the urgent need for a unified development approach that ensures traceability across pipelines and promotes coordinated engineering efforts.” (“MODELS_2025_paper_100”, p. 2) #ffd400
	  *Can you make some examples of problems due to the mentioned limited coordination? What are the currently available technologies that are exploited to mitigate such issues?*
	  
	  - “Process modeling refers to the construction of a structured representation of a process, designed to enhance its comprehension, analysis, and optimization [9]. This representation typically identifies the actions to be executed (activities), the roles responsible for executing them, and the corresponding inputs and outputs (artifacts).” (“MODELS_2025_paper_100”, p. 2) #5fb236
	  
	  - “Rather than prescribing a strict sequence of tasks, activities are triggered dynamically in response to the presence or changes in data values” (“MODELS_2025_paper_100”, p. 2) #a28ae5
	  
	  - “We selected GSM as the foundational modeling approach for our framework due to its strong alignment with the dynamic, data-centric nature of AI system development. Building on these foundations, our framework adopts GSM to model the AI development process declaratively, using data-driven conditions to govern execution, as detailed in the following sections.” (“MODELS_2025_paper_100”, p. 2) #a28ae5
	  
	  - “This section establishes the modeling foundation of the AIEngOrchestrator framework by introducing a domain-specific modeling language, GSM4SE4AI (Guard-Stage-Milestone for Software Engineering for AI).” (“MODELS_2025_paper_100”, p. 2) #5fb236
	  
	  - “GSM4SE4AI is to provide a structured and coordinated process modeling approach specifically tailored to the integrated development of AI systems” (“MODELS_2025_paper_100”, p. 2) #5fb236
	  
	  - “this language facilitates comprehensive end-toend traceability across the entire AI development lifecycle.” (“MODELS_2025_paper_100”, p. 2) #5fb236
	  
	  - “detailed exploration of GSM4SE4AI: (A) the abstract syntax; (B) the concrete syntax of the macro-level model, illustrating the high-level representation and relations between artifacts; and (C) the concrete syntax of the micro-level model, detailing the lifecycle and information modeling within individual artifacts.” (“MODELS_2025_paper_100”, p. 2) #5fb236
	  
	  - “A. GSM4SE4AI Abstract Syntax” (“MODELS_2025_paper_100”, p. 2) #2ea8e5
	  
	  - “AI Development Process” (“MODELS_2025_paper_100”, p. 3) #2ea8e5
	  
	  - “(1) Guard:” (“MODELS_2025_paper_100”, p. 3) #5fb236
	  
	  - “(2) Stage: a unit of work that is activated when its guard condition is satisfied.” (“MODELS_2025_paper_100”, p. 3) #5fb236
	  
	  - “(3) Milestone: a set of business rules also following the ECA pattern, which determines when a stage should be completed.” (“MODELS_2025_paper_100”, p. 3) #5fb236
	  
	  - “its concrete syntax, defined across two distinct levels of granularity” (“MODELS_2025_paper_100”, p. 3) #ffd400
	  *Is this actually implemented? Is it tool supported?*
	  
	  - “the concrete syntax captures: (i) the structural relationships among multiple artifact types—referred to as the macro-level representation, and (ii) the internal behavior of individual artifacts—referred to as the micro-level representation.” (“MODELS_2025_paper_100”, p. 3) #5fb236
	  
	  - “high-level traceability and impact analysis across development pipelines” (“MODELS_2025_paper_100”, p. 3) #a28ae5
	  
	  - “tracking of specific data attributes within artifacts (e.g., learning rate used in a hyperparameter tuning artifact) enabling precise verification and runtime introspection” (“MODELS_2025_paper_100”, p. 3) #a28ae5
	  
	  - “Concrete Syntax” (“MODELS_2025_paper_100”, p. 3) #ffd400
	  *This is supposed to include actual modeling constructs that modelers can use to specify models conforming to the proposed metamodel. Actually by reading the descriptive text of the whole "Macro-Level Concrete Syntax" section, the abstraction level is still high and I don't see any modeling language or some constructs of it that can be used. The whole subsection is lenghty and not easy to grasp. At the end of the sub-section is not clear what's can be used out of the different presented concepts.*
	  
	  - “This subsection introduces the concrete syntax used to represent the macro-level model in GSM4SE4AI, with a focus on the definition of AI artifact and their relationships.” (“MODELS_2025_paper_100”, p. 3) #5fb236
	  
	  - “GSM4SE4AI, the core artifacts—along with their relationships and the rationale behind their inclusion—are modeled to reflect the objective of constructing a Traceability Information Model (TIM) tailored specifically for AI system” (“MODELS_2025_paper_100”, p. 3) #5fb236
	  
	  - “A core component of any software development process is the definition and application of a TIM. Such models provide guidance on which development artifacts should be created and maintained, as well as the relationships that need to be established among them.” (“MODELS_2025_paper_100”, p. 3) #5fb236
	  
	  - “TIM is designed to ultimately support essential project analyses, including change impact assessment, consistency checking, and requirements validation” (“MODELS_2025_paper_100”, p. 4) #5fb236
	  
	  - “To realize this TIM within the context of SE4AI, GSM4SE4AI introduces a set of traceable AI artifacts and their corresponding links (traceable elements), organized around four complementary traceability approaches. Each approach targets a distinct aspect of traceability, and the associated artifacts are modeled to reflect that objective.” (“MODELS_2025_paper_100”, p. 4) #5fb236
	  
	  - “Approach I: Full-Scope Coverage of AI Development” (“MODELS_2025_paper_100”, p. 4) #2ea8e5
	  
	  - “(MLA1) ML Requirement Specification (MLRS)” (“MODELS_2025_paper_100”, p. 4) #2ea8e5
	  
	  - “(MLA2) Data Splitting Artifact” (“MODELS_2025_paper_100”, p. 4) #2ea8e5
	  
	  - “(MLA3) Training Algorithm Artifact” (“MODELS_2025_paper_100”, p. 4) #2ea8e5
	  
	  - “(MLA4) Hyperparameter Tuning Artifact” (“MODELS_2025_paper_100”, p. 4) #2ea8e5
	  
	  - “(MLA5) ML Module Artifact” (“MODELS_2025_paper_100”, p. 5) #2ea8e5
	  
	  - “(MLA6) ML Validation Artifact” (“MODELS_2025_paper_100”, p. 5) #2ea8e5
	  
	  - “System Requirement Specification (SRS),” (“MODELS_2025_paper_100”, p. 5) #5fb236
	  
	  - “This subsection introduces the concrete syntax used to represent the micro-level model, with a focus on the definition of AI artifact behaviors.” (“MODELS_2025_paper_100”, p. 6) #a28ae5
	  
	  - “As the artifact progresses, subsequent Stages are triggered based on Guard conditions that evaluate the values of associated Data Attributes.” (“MODELS_2025_paper_100”, p. 6) #5fb236
	  
	  - “is automatically generated to collect the necessary input from the user.” (“MODELS_2025_paper_100”, p. 6) #ffd400
	  *What is the execution environmnet? When? at which stage of which process? We are missing an overview presentation about the provided framework supporting I guess the different pipelines presented earlier in the paper. The different artifacts of modeling constructs are presented without a proper context and enviroment presentation.*
	  
	  - “Data attributes are explicitly selected and modeled to enhance traceability across different dimensions of AI system development.” (“MODELS_2025_paper_100”, p. 6) #ffd400
	  *See my previous comment.*
	  
	  - “These attributes enable developers to trace how specific model configurations were derived and to revisit the reasoning behind them for future audits or refinements.” (“MODELS_2025_paper_100”, p. 7) #5fb236
	  
	  - “This section explains how the framework supports runtime execution that is driven by changes in artifact data attributes, rather than by fixed activity flows.” (“MODELS_2025_paper_100”, p. 7) #ffd400
	  *Unfortunately the framework (intended as a software that can be used) has not been presented.*
	  
	  - “Fig. 7. SRS-ML artifact behavior defined through data-driven stages. Transitions are shown for illustration, but control flow is implicit” (“MODELS_2025_paper_100”, p. 7) #ffd400
	  *How this process, that it is supposed to be an ideal one, can be actually put in practice also in relation with the existing technology? As a general comment we are missing concrete links from the presented ides with existing technologies and tools that cannot be completely substituted. It is necessary to ensure a link with them.*
	  
	  - “Execution coordination is governed by the artifact relationship layer (see Section III). For instance, once a higherlevel artifact—such as the SRS-ML—has completed all its mandatory stages, related lower-level artifacts (e.g., MLRS, DRS, or SWRS) become eligible for instantiation.” (“MODELS_2025_paper_100”, p. 8) #ffd400
	  *What executes such coordination?*
	  
	  - “enabling consistent propagation of modifications across artifact boundaries.” (“MODELS_2025_paper_100”, p. 8) #ffd400
	  *How such propagation is done? How changes are represented and executed/analyzed?*
	  
	  - “IntentClassifierModule” (“MODELS_2025_paper_100”, p. 8) #5fb236
	  
	  - “TextbasedChat” (“MODELS_2025_paper_100”, p. 8) #5fb236
	  
	  - “DialogManagerModule” (“MODELS_2025_paper_100”, p. 8) #5fb236
	  
	  - “A. Scenario 1. Change Impact Analysis across AI Pipelines” (“MODELS_2025_paper_100”, p. 8) #2ea8e5
	  
	  - “This scenario exemplifies how the framework facilitates the concurrent propagation of changes across pipelines, supporting the systematic determination of how a change in one artifact may affect related artifacts across the AI development pipeline.” (“MODELS_2025_paper_100”, p. 8) #ffd400
	  *The representation and management of such changes is not clear.*
	  
	  - “A subsequent update introduced a new requirement mandating the logging of all user interactions during runtime.” (“MODELS_2025_paper_100”, p. 8) #5fb236
	  
	  - “Thanks to the explicit traceability links between artifacts across AI development pipelines supported by GSM4SE4AI, structured change impact analysis across pipelines becomes feasible and systematic. As illustrated in Fig. 9, the change is initiated within the SWRS of the TextbasedChat module. Through inter-transverse change propagation, this softwarelevel change triggers an update in the DRS, evolving it from Version 1.0 to Version 2.0.” (“MODELS_2025_paper_100”, p. 8) #ffd400
	  *This mechanism is very abstract and it's not evident how it happens in practice. I don't think software migrations due to change propagation happen automatically.*
	  
	  - “s(” (“MODELS_2025_paper_100”, p. 9) #ff6666
	  *Missing space*
	  
	  - “VI. PROOF-OF-CONCEPT PROTOTYPE” (“MODELS_2025_paper_100”, p. 9) #ffd400
	  *what's the envisioned architecture of the orchestrator? Section 6 shows some proof-of-concept dialog boxes without presenting a possible architecture actually involving all the different pipelines and change propagation requirements. Also this section is too high level.*
	  
	  - “proof-of-concep” (“MODELS_2025_paper_100”, p. 9) #ff6666
	  *proof-of-concept*
	  
	  - “A case study and prototype show improved traceability, change propagation, and consistency.” (“MODELS_2025_paper_100”, p. 10) #ffd400
	  *Improved with respect to what? It is necessary to convince about the mentioned improved by considering existing technologies.*
	  
	  Consider that those that are tagged with #5fb236 are just highlights, those that are tagged with #e56eee and #a28ae5 are important sentences. Please pay attention instead to the notes that are tagged with #ffd400. Those that are tagged with #ff6666 are typos or errors. Could you please draft a review by organizing it as follows: SUMMARY: Just a few sentences to summarize the work. COMMENTS: Organize the notes, especially those that contain issues or typos. Moreover, list the strengths and weaknesses of the work (no more than 3 items each). At the end, list 3 questions for the authors that might be involved in a rebuttal phase.