12 KiB
12 KiB
type:: REVIEWS tags:: year:: 2024 venue:: TOSEM full-title:: Foundation Model Engineering: Engineering Foundation Models Just as Engineering Software date-start:: 12-10-2024 - 16:28 date-submitted:: 12-10-2024 external-links:: status:: DONE deadline-submission:: file:: parent:: todoist:: https://app.todoist.com/app/task/tosem-2024-0271-r1-revised-version-6W4xfQcvxh55W4Rg
- ### [[Highlights]]
collapsed:: true
- # Annotazioni
- (12/10/2024, 16:29:38)
- - “FM requirement analysis” ([“TOSEM-2024-0271.R1_Proof_hi”, p. 1](zotero://select/library/items/B589JIUJ)) ([pdf](zotero://open-pdf/library/items/RCM5YIFG?page=2&annotation=SVWW67ZP)) #5fb236
- - “application scenarios for Foundation Models (FMs)” ([“TOSEM-2024-0271.R1_Proof_hi”, p. 1](zotero://select/library/items/B589JIUJ)) ([pdf](zotero://open-pdf/library/items/RCM5YIFG?page=2&annotation=IBMH7PUA)) #5fb236
- - “research opportunities for the requirement engineering of FM” ([“TOSEM-2024-0271.R1_Proof_hi”, p. 1](zotero://select/library/items/B589JIUJ)) ([pdf](zotero://open-pdf/library/items/RCM5YIFG?page=2&annotation=ZUGZKCMV)) #5fb236
- - “requirement analysis” ([“TOSEM-2024-0271.R1_Proof_hi”, p. 1](zotero://select/library/items/B589JIUJ)) ([pdf](zotero://open-pdf/library/items/RCM5YIFG?page=2&annotation=KGIB6T88)) #5fb236
- - “I agree with the author on the overall framework of foundation model development” ([“TOSEM-2024-0271.R1_Proof_hi”, p. 2](zotero://select/library/items/B589JIUJ)) ([pdf](zotero://open-pdf/library/items/RCM5YIFG?page=3&annotation=LU3YWA52)) #5fb236
- - “For example, model training and fine-tuning require substantial GPU computational resources, which are often not affordable for a single person or a small team. It would be beneficial to discuss the impact of these infrastructure-level differences on development.” ([“TOSEM-2024-0271.R1_Proof_hi”, p. 2](zotero://select/library/items/B589JIUJ)) ([pdf](zotero://open-pdf/library/items/RCM5YIFG?page=3&annotation=JP7W8JQA)) #a28ae5
- - “subsection discussing the application scenarios for FMs In Section 3 (State of the Practice)” ([“TOSEM-2024-0271.R1_Proof_hi”, p. 2](zotero://select/library/items/B589JIUJ)) ([pdf](zotero://open-pdf/library/items/RCM5YIFG?page=3&annotation=8BVVN4CG)) #2ea8e5
- - “two sections In Section 4 (Research Opportunities)” ([“TOSEM-2024-0271.R1_Proof_hi”, p. 2](zotero://select/library/items/B589JIUJ)) ([pdf](zotero://open-pdf/library/items/RCM5YIFG?page=3&annotation=EFWUSKZ4)) #2ea8e5
- - “data management such as data cleaning and labeling” ([“TOSEM-2024-0271.R1_Proof_hi”, p. 3](zotero://select/library/items/B589JIUJ)) ([pdf](zotero://open-pdf/library/items/RCM5YIFG?page=4&annotation=RJWZMYRB)) #a28ae5
- - “developers often manually curate and process data, which can be labor-intensive and prone to errors.” ([“TOSEM-2024-0271.R1_Proof_hi”, p. 3](zotero://select/library/items/B589JIUJ)) ([pdf](zotero://open-pdf/library/items/RCM5YIFG?page=4&annotation=AI6WF56E)) #a28ae5
- - “Difference between FME and traditional deep-learning model development” ([“TOSEM-2024-0271.R1_Proof_hi”, p. 3](zotero://select/library/items/B589JIUJ)) ([pdf](zotero://open-pdf/library/items/RCM5YIFG?page=4&annotation=AS4C3WQU)) #5fb236
- - “FME requires continuously evolving a model for different purposes of downstream tasks” ([“TOSEM-2024-0271.R1_Proof_hi”, p. 3](zotero://select/library/items/B589JIUJ)) ([pdf](zotero://open-pdf/library/items/RCM5YIFG?page=4&annotation=5IXXTN36)) #5fb236
- - “We have added two dedicated paragraphs in Section 4 (Research Opportunities) to discuss the research opportunities for secured data and model engineering.” ([“TOSEM-2024-0271.R1_Proof_hi”, p. 3](zotero://select/library/items/B589JIUJ)) ([pdf](zotero://open-pdf/library/items/RCM5YIFG?page=4&annotation=8VB5SUHA)) #a28ae5
- - “Through the establishment of FM engineering, we aim to provide a robust, automated, and extensible framework that addresses the imminent challenges, and discovering new research opportunities for the software engineering field” ([“TOSEM-2024-0271.R1_Proof_hi”, p. 4](zotero://select/library/items/B589JIUJ)) ([pdf](zotero://open-pdf/library/items/RCM5YIFG?page=5&annotation=CJVPIIFC)) #a28ae5
- - “Foundation Model Engineering: Engineering Foundation Models Just as Engineering Software” ([“TOSEM-2024-0271.R1_Proof_hi”, p. 4](zotero://select/library/items/B589JIUJ)) ([pdf](zotero://open-pdf/library/items/RCM5YIFG?page=5&annotation=64KCFE2M)) #a28ae5
- *This paper is very much related to [[PROJECTS/MOSAICO]]*
- - “254,871 models with Transformer architecture” ([“TOSEM-2024-0271.R1_Proof_hi”, p. 5](zotero://select/library/items/B589JIUJ)) ([pdf](zotero://open-pdf/library/items/RCM5YIFG?page=6&annotation=LLSH5Q3S)) #a28ae5
- - “127,462 datasets” ([“TOSEM-2024-0271.R1_Proof_hi”, p. 5](zotero://select/library/items/B589JIUJ)) ([pdf](zotero://open-pdf/library/items/RCM5YIFG?page=6&annotation=EXXQN8XY)) #a28ae5
- - “170,000 AI app demos,” ([“TOSEM-2024-0271.R1_Proof_hi”, p. 5](zotero://select/library/items/B589JIUJ)) ([pdf](zotero://open-pdf/library/items/RCM5YIFG?page=6&annotation=99RFZLR7)) #a28ae5
- - “Innovative techniques are needed for minimizing the barriers posed by interdisciplinary work and effective collaboration.” ([“TOSEM-2024-0271.R1_Proof_hi”, p. 6](zotero://select/library/items/B589JIUJ)) ([pdf](zotero://open-pdf/library/items/RCM5YIFG?page=7&annotation=7CTSJA59)) #a28ae5
- - “When the base FM updates, particularly security-related fixes, it necessitates a corresponding update to the finetuned FMs. It is challenging to efficiently and cost-effectively manage the evolution of these finetuned FMs, catching up with the base model updates while maintaining their domain-specific enhancements.” ([“TOSEM-2024-0271.R1_Proof_hi”, p. 6](zotero://select/library/items/B589JIUJ)) ([pdf](zotero://open-pdf/library/items/RCM5YIFG?page=7&annotation=IQ8XDHPX)) #a28ae5
- - “This diverse ecosystem of models, datasets, and applications is a clear indication that the future of LLMs lies in their ability to be customized.” ([“TOSEM-2024-0271.R1_Proof_hi”, p. 6](zotero://select/library/items/B589JIUJ)) ([pdf](zotero://open-pdf/library/items/RCM5YIFG?page=7&annotation=85C7VYSV)) #a28ae5
- - “ata management and model management” ([“TOSEM-2024-0271.R1_Proof_hi”, p. 6](zotero://select/library/items/B589JIUJ)) ([pdf](zotero://open-pdf/library/items/RCM5YIFG?page=7&annotation=BXTTQZZA)) #a28ae5
- - “Data management with weak supervision.” ([“TOSEM-2024-0271.R1_Proof_hi”, p. 6](zotero://select/library/items/B589JIUJ)) ([pdf](zotero://open-pdf/library/items/RCM5YIFG?page=7&annotation=HLQZC6TT)) #2ea8e5
- - “combining limited amounts of labeled data with large quantities of unlabeled data, using algorithms to infer labels and improve data quality. This method significantly reduces the time and resources required for data preparation.” ([“TOSEM-2024-0271.R1_Proof_hi”, p. 6](zotero://select/library/items/B589JIUJ)) ([pdf](zotero://open-pdf/library/items/RCM5YIFG?page=7&annotation=FUD8G86V)) #5fb236
- - “Model management with workflows and continuous integration” ([“TOSEM-2024-0271.R1_Proof_hi”, p. 6](zotero://select/library/items/B589JIUJ)) ([pdf](zotero://open-pdf/library/items/RCM5YIFG?page=7&annotation=BWBCNR52)) #2ea8e5
- - “We also envision a distributed version control system like Git [41] to track the update of FMs, manage model branches, and resolve conflicts of model updates.” ([“TOSEM-2024-0271.R1_Proof_hi”, p. 6](zotero://select/library/items/B589JIUJ)) ([pdf](zotero://open-pdf/library/items/RCM5YIFG?page=7&annotation=ASR7IUU2)) #a28ae5
- - “Programmatic FM development with declarative specifications” ([“TOSEM-2024-0271.R1_Proof_hi”, p. 7](zotero://select/library/items/B589JIUJ)) ([pdf](zotero://open-pdf/library/items/RCM5YIFG?page=8&annotation=PQKFPI68)) #2ea8e5
- - “unified and declarative APIs that abstract away the underlying complexities of model and data management.” ([“TOSEM-2024-0271.R1_Proof_hi”, p. 7](zotero://select/library/items/B589JIUJ)) ([pdf](zotero://open-pdf/library/items/RCM5YIFG?page=8&annotation=KTFMDGGK)) #5fb236
- - “This not only makes the process more accessible to a wider range of users, including those with less technical expertise but also accelerates the development cycle by enabling quicker iterations and refinements of models.” ([“TOSEM-2024-0271.R1_Proof_hi”, p. 7](zotero://select/library/items/B589JIUJ)) ([pdf](zotero://open-pdf/library/items/RCM5YIFG?page=8&annotation=GE2FP7QD)) #5fb236
- *Kind of [[Lowcode]] for engineering and use of [[Foundation Models]]*
- - “discover research opportunities for the software engineering community in the FM er” ([“TOSEM-2024-0271.R1_Proof_hi”, p. 7](zotero://select/library/items/B589JIUJ)) ([pdf](zotero://open-pdf/library/items/RCM5YIFG?page=8&annotation=LKVC7DZJ)) #5fb236
- - ““FM crisis"” ([“TOSEM-2024-0271.R1_Proof_hi”, p. 7](zotero://select/library/items/B589JIUJ)) ([pdf](zotero://open-pdf/library/items/RCM5YIFG?page=8&annotation=JMHLZJ4C)) #5fb236
- - “This modular approach allows for targeted fine-tuning of each expert model, ensuring that the collective output is both accurate and efficient in handling the task at hand.” ([“TOSEM-2024-0271.R1_Proof_hi”, p. 12](zotero://select/library/items/B589JIUJ)) ([pdf](zotero://open-pdf/library/items/RCM5YIFG?page=13&annotation=5H7ZMWYE)) #5fb236
- - “integrating new data,” ([“TOSEM-2024-0271.R1_Proof_hi”, p. 12](zotero://select/library/items/B589JIUJ)) ([pdf](zotero://open-pdf/library/items/RCM5YIFG?page=13&annotation=FWD3XQGV)) #5fb236
- - “enabling users to tailor Foundation Models for diverse needs with ease” ([“TOSEM-2024-0271.R1_Proof_hi”, p. 12](zotero://select/library/items/B589JIUJ)) ([pdf](zotero://open-pdf/library/items/RCM5YIFG?page=13&annotation=DSLHDV4J)) #5fb236
- - “Each new user or task traditionally necessitates a new pod and GPU, leading to potentially excessive resource consumption” ([“TOSEM-2024-0271.R1_Proof_hi”, p. 12](zotero://select/library/items/B589JIUJ)) ([pdf](zotero://open-pdf/library/items/RCM5YIFG?page=13&annotation=RCUXHWDY)) #5fb236
- - “This realization opens up possibilities for sharing the base model across multiple adapters, thereby optimizing resource use.” ([“TOSEM-2024-0271.R1_Proof_hi”, p. 12](zotero://select/library/items/B589JIUJ)) ([pdf](zotero://open-pdf/library/items/RCM5YIFG?page=13&annotation=ZPZD29JS)) #5fb236
- ### [[Comments]]
- The paper addresses an important issue that is gaining prominence in the software engineering community. Managing foundational models has become a critical aspect in today's context, and the authors identify and elaborate on the crucial actions underlying the engineering of these models. I found the paper to be enjoyable, and I particularly appreciated the overviews presented in Fig. 3 and Fig. 4. The authors have included new paragraphs to address comments received during the initial review, which were not critical. As a result, I am very positive about the paper and recommend its acceptance.
- ### [[REVIEWS/Notes]]
- ### YELLOW CONCERNS
background-color:: yellow
- {{query (and [[ffd400]] [[TOSEM-2024-0271.R1]] )}}
collapsed:: true
- ### ❓️Questions
- {{query (and [[question]] [[TOSEM-2024-0271.R1]] )[[question]]}}
query-table:: true
query-properties:: [:block]