109 lines
9.2 KiB
Markdown
109 lines
9.2 KiB
Markdown
tags:: [[#zotero]]
|
||
title:: @icse2025-paper1545
|
||
item-type:: [[document]]
|
||
original-title:: icse2025-paper1545
|
||
links:: [Local library](zotero://select/library/items/K2MYJQ7Q), [Web library](https://www.zotero.org/users/1039502/items/K2MYJQ7Q)
|
||
|
||
- ### Attachments
|
||
- [PDF](zotero://select/library/items/YT3PSTGK) {{zotero-imported-file YT3PSTGK, "icse2025-paper1545.pdf"}}
|
||
- ### Notes
|
||
- # Annotazioni
|
||
(2/10/2024, 16:56:44)
|
||
|
||
“Traditionally, software engineering focuses on manually created artifacts such as source code and the process of creating them, as well as best practices for integrating them, i.e., software architectures.” (“icse2025-paper1545”, p. 1)
|
||
|
||
“ML models must be embedded in traditional software, often forming complex topologies” (“icse2025-paper1545”, p. 1)
|
||
|
||
“little is known about the characteristics of real-world ML-enabled systems.” (“icse2025-paper1545”, p. 1) What do you mean?
|
||
|
||
“Our findings provide practitioners and researchers with insight into practices for embedding and integrating ML models, bringing data science and software engineering closer together.” (“icse2025-paper1545”, p. 1)
|
||
|
||
“In fact, industrial surveys report that 78 % of ML projects stall [6],” (“icse2025-paper1545”, p. 1) Such reference is of 2019... things might have changed.
|
||
|
||
“as many as 85 % fail [7].” (“icse2025-paper1545”, p. 1) Same comment like before.
|
||
|
||
“Many factors can cause such failures [8], [9] [10], including data quality, waterfall-like development processes, limited experience and education, but also lack of proper tools [11], [8], [12], [13].” (“icse2025-paper1545”, p. 1)
|
||
|
||
“probabilistic models” (“icse2025-paper1545”, p. 1)
|
||
|
||
“data quality, integration of ML models with traditional software components, tool support for managing MLrelated artifacts, and quality assurance, particularly concerning dependability and safety” (“icse2025-paper1545”, p. 1)
|
||
|
||
“Current research in software engineering seeks to address these deficiencies by proposing novel workflow patterns and providing improved tool support supporting ML experiment” (“icse2025-paper1545”, p. 1)
|
||
|
||
“we still know very little about how ML models are actually embedded and integrated into ML-enabled systems.” (“icse2025-paper1545”, p. 1)
|
||
|
||
“We are especially interested in the reuse of ML models across software systems, as well as the integration of ML models.” (“icse2025-paper1545”, p. 1)
|
||
|
||
“model reuse and sharing ML models across systems is essential in practice.” (“icse2025-paper1545”, p. 1)
|
||
|
||
“identifying patterns of how (multiple) ML-models are integrated, as well as the relationship between ML models and source code” (“icse2025-paper1545”, p. 2)
|
||
|
||
“predictions, preprocessing data or, since recently, creating content via generative AI” (“icse2025-paper1545”, p. 2)
|
||
|
||
“These arise from the fundamentally different nature of ML models compared to traditional software” (“icse2025-paper1545”, p. 2)
|
||
|
||
“different toolchains for ML models and software.” (“icse2025-paper1545”, p. 2)
|
||
|
||
“There also is a lack of tools for managing (e.g., versioning) the many different artifacts (called ML assets in the remainder) for ML” (“icse2025-paper1545”, p. 2)
|
||
|
||
“Traditional software engineering relies on wellestablished version-control and collaborative development systems, such as Git.” (“icse2025-paper1545”, p. 2)
|
||
|
||
“TensorFlow and PyTorch are frameworks allowing developers to implement their ML models modularly.” (“icse2025-paper1545”, p. 2)
|
||
|
||
““Module,” which represents either complete models or their building blocks (e.g., layers).” (“icse2025-paper1545”, p. 2)
|
||
|
||
“(i) in transfer learning [38] the model is trained on a generic dataset and is then fine-tuned to solve a more specific problem;” (“icse2025-paper1545”, p. 2)
|
||
|
||
“(ii) pretrained models (PTMs) are fully trained and usually reused without any fine-tuning.” (“icse2025-paper1545”, p. 2)
|
||
|
||
“there is still a lack of hard empirical data on reuse in real ML-enabled systems.” (“icse2025-paper1545”, p. 2)
|
||
|
||
“Optimizing non-functional properties via retraining can then easily lead to a combinatorial explosion due to the different training parameters for models or the interactions of multiple models.” (“icse2025-paper1545”, p. 3)
|
||
|
||
“A study suggests a microservice architecture for ML models [42], where ML models are modularized and interact with the rest of the system via REST.” (“icse2025-paper1545”, p. 3)
|
||
|
||
“high quality set of ML-enabled software systems of a certain size and maturity. For instance, we excluded tutorials and small toy projects.” (“icse2025-paper1545”, p. 3)
|
||
|
||
“we excluded projects that were forked directly from GitHub, which can be accessed through GitHub’s REST API.” (“icse2025-paper1545”, p. 3)
|
||
|
||
“2,928 repositories, containing systems with medium to high maturity.” (“icse2025-paper1545”, p. 3)
|
||
|
||
“classification of the relation of ML to the software system.” (“icse2025-paper1545”, p. 4) What do you mean with "relation of ML to the software system"?
|
||
|
||
“We searched the code for class definitions that extend these two classes or any of their subclasses provided in the libraries.” (“icse2025-paper1545”, p. 4)
|
||
|
||
“Stored model files: Typically, trained models can be saved to a file for later reuse. TensorFlow and PyTorch each provide their own file types for saving trained ML models as binary files.” (“icse2025-paper1545”, p. 4)
|
||
|
||
“Metrics. Based on the identified ML-related assets, we calculate metric characterizing the ML-enabled systems in respect to their ML-related parts.” (“icse2025-paper1545”, p. 4)
|
||
|
||
“(i) Use of pre-trained models involves obtaining an already trained ML model from an external source;” (“icse2025-paper1545”, p. 4)
|
||
|
||
“(ii) reuse of only the architecture of an ML model, by copying the related implementation in source code.” (“icse2025-paper1545”, p. 4)
|
||
|
||
“a sample of our subject systems” (“icse2025-paper1545”, p. 4) Can you say something more about this sample?
|
||
|
||
“ML Functionalities and Tasks: What are the ML models used for? What functionality is implemented using ML? Which ML task realizes the functionality?” (“icse2025-paper1545”, p. 5) How to answer such qualitative questions?
|
||
|
||
“Integration of Models: How do multiple models interact? (Parallel, alternative, sequential, fusion/forking)” (“icse2025-paper1545”, p. 5) How have you assessed this?
|
||
|
||
“There are slightly more applications than frameworks,” (“icse2025-paper1545”, p. 6)
|
||
|
||
“ML-related and non-MLrelated code.” (“icse2025-paper1545”, p. 6)
|
||
|
||
“The large majority of ML-enabled systems on GitHub are libraries and frameworks. Interestingly, end-user-oriented applications are still a minority. These systems use ML in different ways, either developing new ML technology or applying ML on concrete problems. A smaller, but still substantial, number of tools that support ML engineering exists. Our systems vary in size and contain many different ML assets, but also large amounts of non-ML code. Quality assurance, unfortunately, is largely neglected.” (“icse2025-paper1545”, p. 7)
|
||
|
||
“Reuse of ML models is prevalent among the systems. 1,209 systems load pre-trained models and 1,690 systems copy ML implementations directly. The origins of the copied ML implementations are a few repositories that contain ML technology provided by well-known companies.” (“icse2025-paper1545”, p. 8)
|
||
|
||
“Number and Type of ML Models.” (“icse2025-paper1545”, p. 8) The different paragraphs of this section (e.g., "Number and Type of ML Models", "Interaction between ML models", etc.) need to be motivated and introduced to simplify the reading of the whole section. Moreover, the title of the section "ML Integration Architectures" does not reflect the content of the section, which instead discuss different characteristsics of ML models, like their origin, the types of the input and output data, required pre and post processing stepts, etc.). In this respect, it is not about architecture, but insted about ML characteristics and usages?
|
||
|
||
“models is the most common pattern (15 systems), where multiple models are offered for the same task, either automatically by the program or based on user input.” (“icse2025-paper1545”, p. 9)
|
||
|
||
“While more than half of our subject systems were libraries that provide ML functionalities, only 18 % of systems are applications that use ML to provide end-user-oriented functionality” (“icse2025-paper1545”, p. 10)
|
||
|
||
“ML model implementations are often copied between repositories” (“icse2025-paper1545”, p. 10)
|
||
|
||
“many pre-trained models from model hubs are used” (“icse2025-paper1545”, p. 10)
|
||
|
||
“poor dependency management” (“icse2025-paper1545”, p. 10)
|
||
|
||
“how ML models are integrated often depends on the domain of the system. However, despite the lack of well-researched integration patterns [15], we observed different kinds of interations of ML models.” (“icse2025-paper1545”, p. 10) |