7.7 KiB
7.7 KiB
type:: REVIEWS tags:: year:: 2025 venue:: ICSE full-title:: AutoEmbed: towards automated software development for generic embedded IoT systems via LLMs date-start:: 06-05-2025 - 15:25 date-submitted:: external-links:: status:: DONE deadline-submission:: file:: @AutoEmbed: towards automated software development for generic embedded IoT systems via LLMs parent:: todoist:: https://app.todoist.com/app/task/50-auto-embed-towards-automated-software-development-for-generic-embedded-io-t-s-6XXr6PwcRP2xm4Mc
- ### [[Highlights]]
- ### [[Comments]]
collapsed:: true
- Paper summary
- The paper presents AutoEmbed, an LLM-based approach to support the automated development of embedded IoT systems. The platform aims to facilitate different phases of the development process, including identifying the required modules and libraries needed to implement the desired system. The paper presents an evaluation consisting of 4 different development platforms, 71 modules, and over 350 IoT tasks.
- Strengths
- + Timely and highly relevant problem
- + Fully automated development workflow
- + Large-scale evaluation
- Weaknesses
- - Presentation issues affecting readability and understanding
- - Reproducibility is limited as the dataset is not available during review
- - Evaluation metrics are not always tied to system capabilities
- Detailed comments for authors
- **Novelty**: The proposed approach presents a novel and interesting integration of LLMs into developing embedded IoT systems.
- Rigor: The system is presented by distinguishing the different architectural components of AutoEmbed. I have some concerns related to the fact that the shown equations, being generic, contribute little to the reader's understanding. Moreover, there is an unclear connection between evaluation metrics and system components. For instance, it is not clear how the library solving problem has been investigated by using the "coding accuracy" and "completion rate" metrics that seem to work at a different level of abstraction (more at the code level). Isn't it?
- Relevance: The contribution is highly relevant to the intersection of AI-based software engineering and embedded systems. However, the paper does not clarify the main intended users (e.g., embedded developers, domain experts with no programming background), and the assumptions about hardware availability and developer expertise.
- Verifiability & transparency: Unfortunately, the EmbedTask dataset is unavailable during review, which hampers reproducibility.
- Presentation: The paper suffers from several presentation issues as listed below:
- Some figures are either redundant or unclear, e.g., Fig. 3b and Fig. 4 represent the same thing, and consequently, I suggest merging them. Fig. 3b should show an explicitly compile–flash–debug loop. Currently, it seems that only two iterations of the process are performed.
- Text/figure mismatches, e.g., steps in Fig. 8 are not reflected in corresponding paragraphs.
- Terminologies like *functionality* and *tasks* are used loosely and need better definition.
- High-level formalizations often lack accompanying concrete examples.
- Detailed comments:
- Page 2: "In general-purpose programming, the workflow typically involves coding, debugging, and deployment. In contrast, embedded system programming introduces additional steps such as compiling and flashing, which require specialized configurations and are particularly error-prone": this sentence should be revised because compilation can also occur in general-purpose programming and not only in embedded system programming.
- Page 3 on "Solving dependencies": Why not put in the formalization, $L_s$, directly in $E_s$? $L_s$ should be found depending on the modules building up the system being developed.
- Page 4 on "Version count": Why is it between 0 and 1? Many versions of the same library might be available, aren't they?
- Page 4 on "Library knowledge extraction": What do you mean by functionality here? The given formalizations (equations) do not add more. I would replace them with some concrete examples to complement the graphical representation in Fig. 7. Without concrete examples, reviewers might understand terms like function or functionality differently.
- Page 5 on equation 5: See my previous comment about the opportunity to put in equations that do not add clarifications. They stay at a too high level of detail, so they do not contribute too much to the presentation. I would replace them with some concrete and explanatory examples.
- Page 6 on the example shown in Fig. 9: What's the process's input? How are the requirements of the wanted system given?
- Page 6 on Step 1 of the process: What if the code compiles correctly but the system does not implement all the wanted functionalities, or none? Related to the previous comment, it is unclear how the set of validations that get executed is obtained from the initial input. In other words, it is unclear how the set of validations is derived from the input.
- Page 6 on the EmbedTask dataset: Are these tasks the requirements of different applications? What's the granularity of the task description?
- Page 6 on the three-level tasks: It's unclear how this can work. How have these three levels been decided? It's always three levels? It isn't easy to distinguish the steps that are example-specific from those that are supposed to be generic.
- Page 6 on the Evaluation setup: The experimental settings involving all the different platforms and devices depicted in Fig. 10 should be better described, especially concerning the level of human involvement in the evaluation process.
- Typos:
- Page 6: "the modifications. (step 4 )" -> "the modifications (step 4 )."
- Page 6: "Prompt design: [https://autoembed.github.io/#Prompt."](https://autoembed.github.io/#Prompt.%22) You can drop "Prompt design" and give the URL directly.
- QUESTIONS
- How is the correctness of the generated system verified beyond compilation—particularly when logical errors might be present?*
- Could you clarify how metrics such as "coding accuracy" and "completion rate" reflect the effectiveness of individual components of AutoEmbed (e.g., library resolution, knowledge augmentation)? Are there any component-specific metrics available?
- Regarding the EmbedTask dataset: could you elaborate on the granularity of the tasks and how the classification into three difficulty levels was defined?
- [[Metareview]]
- The paper proposes AutoEmbed, an LLM-based framework for automating embedded IoT development. While the idea is interesting and the evaluation broad, according to the reviewers the submission suffers from significant limitations. In particular, valid concerns are raised regarding the clarity and rigor of the methodology, the definition and validity of evaluation metrics, and the limited discussion of system correctness and robustness. Reviewers are not satisfied by the rebuttal letter, which does not address these key issues. Given these unresolved concerns, particularly around logical correctness , the paper is not ready for acceptance in its current form. Nevertheless, the reviewers have provided specific suggestions that we believe will be valuable in improving the paper for potential publication in the future. We hope you find these suggestions helpful as you work towards enhancing the quality of the research.
- ### [[REVIEWS/Notes]]
- ### YELLOW CONCERNS
background-color:: yellow
collapsed:: true
- {{query (and [[ffd400]] [[icse2026-paper50]] )}}
collapsed:: true
- ### ❓️Questions
- {{query (and [[question]] [[icse2026-paper50]] )[[question]]}}
query-table:: true
query-properties:: [:block]