43 lines
4.4 KiB
Markdown
43 lines
4.4 KiB
Markdown
type:: [[REVIEWS]]
|
|
tags::
|
|
year:: 2025
|
|
venue:: [[MODELS]]
|
|
full-title:: AI Engineering Orchestrator: A Model-Based Framework for AI Development Process Management
|
|
date-start:: [[22-05-2025]] - 11:26
|
|
date-submitted::
|
|
external-links::
|
|
status:: [[DONE]]
|
|
deadline-submission::
|
|
file:: [[@MODELS_2025_paper_100]]
|
|
parent::
|
|
todoist:: https://app.todoist.com/app/task/ai-engineering-orchestrator-a-model-based-framework-for-ai-development-process-m-6Xh6hh3mh383RQgg
|
|
|
|
- ### [[Highlights]]
|
|
- ### [[Comments]]
|
|
- Summary of the paper: The paper presents a model-based framework to support the orchestration of different pipelines involving the development and management of AI-based software systems. The main building block of the framework, named AIEngOrchestrator, is the GSM4SE4AI domain specific language. The DSL integrates concepts from Software Engineering for AI and from process modeling. The framework aims at imprving the traceability, and change propagation among the artifacts that are produced and managed in different pipelines of AI-based software systems. A case study is presented to discuss the different components of the framework and of the provided DSL.
|
|
id:: 682eee3e-64ca-487c-a471-8bac07f47aad
|
|
- Strengths:
|
|
- The paper is about an interesting problem involving ML and SE workflows.
|
|
- The discussed framework touches on a broad range of AI development activities (data, ML, SE pipelines).
|
|
- Weaknesses:
|
|
- No clear description of execution support or system architecture is given.
|
|
- Concrete syntax remains abstract, making it hard to understand how to apply the framework.
|
|
id:: 682ef0c7-628b-4c97-ad3e-c62a57747422
|
|
- Claims about improvement are not supported by comparative evaluation or evidence.
|
|
- Detailed comments:
|
|
- The orchestration of heterogeneous pipelines in AI development (software engineering, ML training, data engineering, etc.) is a relevant and pressing challenge nowadays. However, the paper is affected by several issues that compromise the overall clarity and practical relevance of the proposed solution. Below, I elaborate on major concerns:
|
|
- Early in the paper it is necessary to clarify whether AIEngOrchestrator targets the development process or also runtime execution of the development AI systems.
|
|
- When stating that the framework "makes relationships between artifacts clear and manageable", the purpose of such manageability should be clarified (e.g., for traceability, validation, evolution?)
|
|
- The paper discusses fragmented pipelines but lacks concrete examples of problems caused by poor coordination. Examples and references to existing technologies that fail in supporting such coordinations would improve credibility of the proposed framework.
|
|
- The "macro-level concrete syntax" section introduces high-level concepts but fails to deliver actual modeling constructs. It is difficult to understand what parts of the language are implemented and how modelers should use them.
|
|
- The execution environment of the proposed framework is unclear. Moreover, it is not evident at which stage of development the framework is supposed to operate. It is necessary to introduce contextual overviews to link the different artifacts and pipelines introduced in section 2.
|
|
- By reading section 3, it remains unclear whether a tool exists to support model execution, artifact coordination, and change propagation. What is the execution engine, and how does it interact with the modeled artifacts?
|
|
- Several claims are made about change propagation across pipelines, even though no concrete mechanisms are described. How are changes represented, managed, and analyzed?
|
|
- The prototype section is too high-level. Dialog boxes are shown, but no software architecture is described that ties together the proposed concepts.
|
|
- The claim that the case study and prototype show “improved traceability, change propagation, and consistency” lacks comparison with any baseline or existing technology.
|
|
- Typos
|
|
- Page 9: Root Cause Analysis(RCA) -> Root Cause Analysis (RCA)
|
|
- Page 9: “proof-of-concep” -> should be “proof-of-concept”.
|
|
- Questions for the authors
|
|
- Q1: Is there any implemented tool or runtime engine supporting the execution of models specified in GSM4SE4AI? If so, how is coordination among pipelines handled concretely?
|
|
- Q2: How does the proposed framework integrate with existing SE/ML tools? Are there plans or capabilities for such integration? |