4.3 KiB
4.3 KiB
type:: REVIEWS tags:: year:: 2024 venue:: ICSE full-title:: date-start:: 02-10-2024 - 21:32 date-submitted:: external-links:: status:: DONE deadline-submission:: file:: @icse2025-paper1545 parent:: todoist:: https://app.todoist.com/app/task/1545-a-large-scale-study-of-model-integration-in-ml-enabled-software-systems-6WFCX7RWrh4RGjv6
- ### [[Highlights]]
- ### [[Comments]]
- #.tabular
- - ### Paper summary
- The paper presents an empirical study on the integration of machine learning models into ML-enabled software systems. It analyzes over 2,900 GitHub repositories to extract insights into how ML models are reused, embedded, and integrated with traditional software components. The study identifies several challenges in the integration process, such as the reuse of pre-trained models, dependency management issues, and the need for architectural patterns for ML integration. The authors also discuss the reuse of ML models and their interaction with non-ML code.
- - ### Strengths
- +
- Interesting paper on a relevant problem
- Analysis of over 2,900 repositories providinging insights into real-world practices for integrating ML models into software systems.
- - ### Weaknesses
- -
- Some references (e.g., failure rates of ML projects) are from 2019, and the field of ML evolves quickly. More recent data would improve the relevance of these points.
- The section discussing "ML Integration Architectures" covers diverse characteristics of ML models rather than focusing on integration architectures. The title does not fully reflect the content, leading to confusion.
- - ### Detailed comments for authors
-
- Novelty: The paper presents an interesting empirical study on the integration of ML models into software systems. The work builds upon existing research, but its focus on large datasets and real-world systems.
- Rigor: The study is well conducted by combining automated analysis of codebases with manual classification. However, the manual classification is limited to a small sample, which could introduce some biases.
- Relevance: The paper addresses highly relevant topics, such as the challenges of integrating ML models with traditional software systems, which are critical for both academic and industrial communities.
- Verifiability & Transparency: The dataset made available by the authors increases the paper's transparency and verifiability.
- Presentation: The paper is generally well-organized, but certain sections lack clarity. Specifically, the title of the "ML Integration Architectures" section is misleading, as it covers a broad range of ML-related characteristics rather than focusing on integration architectures. Additionally, the qualitative analysis of questions such as "How do multiple models interact?" could be presented in a more structured manner.
- #### Detailed Comments:
- Some references are outdated. In particular, the references regarding the failure rates of ML projects are from 2019. Given the rapid development in ML and AI engineering, more recent data should be used to strengthen the argument.
- The paper rightly mentions several challenges related to data quality and ML integration but could provide more concrete examples from the analyzed systems to support these claims.
- The decision to manually classify 160 out of 2,928 repositories is understandable, but the potential biases and limitations of this approach should be more clearly addressed.
- The different paragraphs of Section 5 (e.g., "Number and Type of ML Models", "Interaction between ML models", etc.) need to be motivated and introduced to simplify the reading of the whole section. Moreover, the title of the section "ML Integration Architectures" does not reflect the content of the section, which instead discusses different characteristics of ML models, like their origin, the types of input and output data, required pre and post-processing steps, etc.). In this respect, it is not about architecture but instead about ML characteristics and usages.
- ### [[REVIEWS/Notes]]
- ### YELLOW CONCERNS
background-color:: yellow
- {{query (and [[ffd400]] [[icse2025-paper1545]] )}}
collapsed:: true
- ### ❓️Questions
- {{query (and [[question]] [[icse2025-paper1545]] )[[question]]}}
query-table:: true
query-properties:: [:block]