logseq/pages/@AutoEmbed%3A towards automated software development for generic embedded IoT systems via LLMs.md

tags:: #zotero title:: @AutoEmbed: towards automated software development for generic embedded IoT systems via LLMs item-type:: journalArticle original-title:: AutoEmbed: towards automated software development for generic embedded IoT systems via LLMs language:: en library-catalog:: Zotero links:: Local library, Web library

• Abstract
  • Embedded Internet of Things (IoT) system development is crucial for enabling seamless connectivity and functionality across a wide range of applications. However, such a complex process requires cross-domain knowledge of hardware and software and hence often necessitates direct developer involvement, making it labor-intensive, time-consuming, and error-prone. To address this challenge, this paper introduces AutoEmbed, the first fully automated software development platform for general-purpose embedded IoT systems. The key idea is to leverage the reasoning ability of Large Language Models (LLMs) and embedded system expertise to automate the hardware-in-the-loop development process. The main methods include a component-aware library resolution method for addressing hardware dependencies, a library knowledge generation method that injects utility domain knowledge into LLMs, and an auto-programming method that ensures successful deployment. We evaluate AutoEmbed’s performance across 71 modules and four mainstream embedded development platforms with over 350 IoT tasks. Experimental results show that AutoEmbed can generate codes with an accuracy of 95.7% and complete tasks with a success rate of 86.5%, surpassing human-in-the-loop baselines by 15.6%–37.7% and 25.5%–53.4%, respectively. We also show AutoEmbed ’s potential through case studies in environmental monitoring and remote control systems development.

• Attachments

  • PDF {{zotero-imported-file 3MGT3I8L, "AutoEmbed Towards Automated Software Development for Generic Embedded IoT Systems via LLMs.pdf"}}

• Notes

  • I'm reviewing a research paper and I took the following notes:

    Annotazioni

    (6/5/2025, 15:15:36)

    • “AutoEmbed: Towards Automated Software Development for Generic Embedded IoT Systems via LLMs” (“AutoEmbed: towards automated software development for generic embedded IoT systems via LLMs”, p. 1) #ffd400 Overall it is an interesting paper about a relevant topic. In my opinion there are some presentation issues, which affect the quality of the paper, which requires improvements to better present the approach and to be more convincing from the evaluation point of view.

    • “this paper introduces AutoEmbed, the first fully automated software development platform for general-purpose embedded IoT systems.” (“AutoEmbed: towards automated software development for generic embedded IoT systems via LLMs”, p. 1) #a28ae5

    • “automate the hardware-in-the-loop development process” (“AutoEmbed: towards automated software development for generic embedded IoT systems via LLMs”, p. 1) #a28ae5

    • “We evaluate AutoEmbed’s performance across 71 modules and four mainstream embedded development platforms with over 350 IoT tasks.” (“AutoEmbed: towards automated software development for generic embedded IoT systems via LLMs”, p. 1) #a28ae5

    • “accuracy of 95.7%” (“AutoEmbed: towards automated software development for generic embedded IoT systems via LLMs”, p. 1) #e56eee

    • “We also show AutoEmbed ’s potential through case studies in environmental monitoring and remote control systems development.” (“AutoEmbed: towards automated software development for generic embedded IoT systems via LLMs”, p. 1) #a28ae5

    • “For example, in a smart city [34, 42], embedded systems control street lighting and traffic signals based on sensor data to optimize energy use and traffic flow.” (“AutoEmbed: towards automated software development for generic embedded IoT systems via LLMs”, p. 1) #5fb236

    • “understanding” (“AutoEmbed: towards automated software development for generic embedded IoT systems via LLMs”, p. 1) #5fb236

    • “how devices interact with the physical world” (“AutoEmbed: towards automated software development for generic embedded IoT systems via LLMs”, p. 1) #5fb236

    • “Initially, developers must manually address dependencies by installing and configuring the essential libraries for hardware modules.” (“AutoEmbed: towards automated software development for generic embedded IoT systems via LLMs”, p. 1) #5fb236

    • “Keil μVision, and IAR” (“AutoEmbed: towards automated software development for generic embedded IoT systems via LLMs”, p. 1) #ffd400 what are these? To check

    • “A fully automated platform for developing embedded IoT systems” (“AutoEmbed: towards automated software development for generic embedded IoT systems via LLMs”, p. 1) #ffd400 Let's see. but I expect a strong evaluation consisting of IoT systems developed with AutoEmbed as promised/described in the paper.

    • “diversity” (“AutoEmbed: towards automated software development for generic embedded IoT systems via LLMs”, p. 1) #5fb236

    • “hardware modules” (“AutoEmbed: towards automated software development for generic embedded IoT systems via LLMs”, p. 1) #5fb236

    • “ncreases the complexity” (“AutoEmbed: towards automated software development for generic embedded IoT systems via LLMs”, p. 1) #5fb236

    • “the traditional embedded IoT system development process is labor-intensive, time-consuming, and error-prone” (“AutoEmbed: towards automated software development for generic embedded IoT systems via LLMs”, p. 1) #5fb236

    • “Large Language Models (LLMs) to streamline embedded IoT system development.” (“AutoEmbed: towards automated software development for generic embedded IoT systems via LLMs”, p. 1) #a28ae5

    • “harness their capabilities to simplify the development process by automating processes such as dependency-solving, coding, and deployment.” (“AutoEmbed: towards automated software development for generic embedded IoT systems via LLMs”, p. 1) #e56eee

    • “they fall short in driving specific hardware devices (e.g., microcontrollers and sensors) due to a lack of hardware-specific knowledge in embedded systems, such as peripheral interface configurations and library dependencies.” (“AutoEmbed: towards automated software development for generic embedded IoT systems via LLMs”, p. 1) #e56eee

    • “LLM-powered automation system that streamlines the dependency-solving, programming, and deployment processes in embedded system development, resulting in fully-developed embedded systems for various IoT applications as illustrated in Fig. 1.” (“AutoEmbed: towards automated software development for generic embedded IoT systems via LLMs”, p. 1) #e56eee

    • “Challenge 1: Diversity in Hardware Dependency” (“AutoEmbed: towards automated software development for generic embedded IoT systems via LLMs”, p. 1) #2ea8e5

    • “Each component relies on specific library dependencies to function effectively, leading to unique challenges in dependency resolution.” (“AutoEmbed: towards automated software development for generic embedded IoT systems via LLMs”, p. 1) #a28ae5

    • “accurately identifying and selecting the essential libraries for different hardware components is a significant challenge.” (“AutoEmbed: towards automated software development for generic embedded IoT systems via LLMs”, p. 2) #e56eee

    • “ibrary selection, revealing that different libraries exhibit distinct compatibility with specific models and varying support for development board architectures” (“AutoEmbed: towards automated software development for generic embedded IoT systems via LLMs”, p. 2) #a28ae5

    • “we propose an automated dependency solving method that can efficiently identify the most suitable libraries for specific hardware components.” (“AutoEmbed: towards automated software development for generic embedded IoT systems via LLMs”, p. 2) #e56eee

    • “Challenge 2: Lack of Library Knowledge” (“AutoEmbed: towards automated software development for generic embedded IoT systems via LLMs”, p. 2) #2ea8e5

    • “knowledge generation method that extracts and injects library API and utility knowledge into the LLM’s memory, enabling syntactically and contextually appropriate solutions.” (“AutoEmbed: towards automated software development for generic embedded IoT systems via LLMs”, p. 2) #a28ae5

    • “Challenge 3: Complexity of Embedded System Programming.” (“AutoEmbed: towards automated software development for generic embedded IoT systems via LLMs”, p. 2) #2ea8e5

    • “In general-purpose programming, the workflow typically involves coding, debugging, and deployment. In contrast, embedded system programming introduces additional steps such as compiling and flashing, which require specialized configurations and are particularly error-prone” (“AutoEmbed: towards automated software development for generic embedded IoT systems via LLMs”, p. 2) #ffd400 Compilation can occur also in general-purpose programming and not only in embedded system programming!

    • “compile loop” (“AutoEmbed: towards automated software development for generic embedded IoT systems via LLMs”, p. 2) #2ea8e5

    • “Figure 3: Programming pipeline comparison.” (“AutoEmbed: towards automated software development for generic embedded IoT systems via LLMs”, p. 2) #ffd400 I would improve Fig. 3.b with a loop involving Compiling->Flashing->Debugging->Compiling .... because otherwise it seems that the typical pipeline involve only two flashing stepts, whcih of course it is not what the authors meant.

    • “flash loop” (“AutoEmbed: towards automated software development for generic embedded IoT systems via LLMs”, p. 2) #2ea8e5

    • “we design and implement AutoEmbed, a comprehensive framework that fully automates the dependency-solving, programming, and deployment processes for embedded system development.” (“AutoEmbed: towards automated software development for generic embedded IoT systems via LLMs”, p. 2) #e56eee

    • “Our extensive evaluation, involving over 70 hardware modules, four development platforms, and over 350 IoT tasks, demonstrates that AutoEmbed achieves an average coding accuracy of 95.7% and an average completion rate of 86.5%, highlighting the effectiveness of AutoEmbed.” (“AutoEmbed: towards automated software development for generic embedded IoT systems via LLMs”, p. 2) #e56eee Interesting to see how these configurations have been actually checked (see the evaluation section later in the paper).

    • “library resolution method for hardware dependency solving” (“AutoEmbed: towards automated software development for generic embedded IoT systems via LLMs”, p. 2) #2ea8e5

    • “knowledge generation method that enhances LLMs with specialized library knowledge” (“AutoEmbed: towards automated software development for generic embedded IoT systems via LLMs”, p. 2) #2ea8e5

    • “auto-programming method to ensure successful deployment” (“AutoEmbed: towards automated software development for generic embedded IoT systems via LLMs”, p. 2) #2ea8e5

    • “EmbedTask dataset” (“AutoEmbed: towards automated software development for generic embedded IoT systems via LLMs”, p. 2) #2ea8e5

    • “which will be made publicly available upon paper acceptance.” (“AutoEmbed: towards automated software development for generic embedded IoT systems via LLMs”, p. 2) #ffd400 This poses some issues concerning the reproducibility of the approach during the review phase.

    • “95.7% across various IoT tasks” (“AutoEmbed: towards automated software development for generic embedded IoT systems via LLMs”, p. 2) #5fb236

    • “Es = g(D, M)” (“AutoEmbed: towards automated software development for generic embedded IoT systems via LLMs”, p. 2) #5fb236

    • “development platform” (“AutoEmbed: towards automated software development for generic embedded IoT systems via LLMs”, p. 2) #2ea8e5

    • “connected modules” (“AutoEmbed: towards automated software development for generic embedded IoT systems via LLMs”, p. 2) #2ea8e5

    • “Figure 4:” (“AutoEmbed: towards automated software development for generic embedded IoT systems via LLMs”, p. 3) #ffd400 This looks like Fig. 3.b. I suggest to remove one of them.

    • “Solving Dependencies.” (“AutoEmbed: towards automated software development for generic embedded IoT systems via LLMs”, p. 3) #ffd400 Why not putting in the formalization, Ls directly in Es? Ls should be found depending on the modules building up the system being developed.

    • “probability is computed using a softmax function” (“AutoEmbed: towards automated software development for generic embedded IoT systems via LLMs”, p. 3) #5fb236

    • “memory-augmented LLMs to generate task prompts and automate system programming” (“AutoEmbed: towards automated software development for generic embedded IoT systems via LLMs”, p. 3) #5fb236

    • “3.1.1 Hardware Configuration” (“AutoEmbed: towards automated software development for generic embedded IoT systems via LLMs”, p. 3) #2ea8e5

    • “3.1.2 Solving Library Dependencies” (“AutoEmbed: towards automated software development for generic embedded IoT systems via LLMs”, p. 3) #2ea8e5

    • “The score is based on the number of available library versions, normalized between 0 and 1” (“AutoEmbed: towards automated software development for generic embedded IoT systems via LLMs”, p. 4) #ffd400 Why between 0 and 1? This is not clear. Many versions of the same library might be available, isn't it?

    • “3.2.1 Library Knowledge Extraction” (“AutoEmbed: towards automated software development for generic embedded IoT systems via LLMs”, p. 4) #2ea8e5

    • “This involves two main steps: API extraction and obtaining API usage information” (“AutoEmbed: towards automated software development for generic embedded IoT systems via LLMs”, p. 4) #5fb236

    • “searches for example (.ino) files within the same library and extracts knowledge on how to use the extracted APIs (e.g., orders, parameters, and return values).” (“AutoEmbed: towards automated software development for generic embedded IoT systems via LLMs”, p. 4) #5fb236

    • “3.2.2 Functionality Knowledge Understanding” (“AutoEmbed: towards automated software development for generic embedded IoT systems via LLMs”, p. 4) #2ea8e5

    • “functionality” (“AutoEmbed: towards automated software development for generic embedded IoT systems via LLMs”, p. 4) #ffd400 What do you mean with functionality here? The given formalizations (equations) do not add more. I would replace them with some concrete examples, to complement the graphical representation given in Fig. 7. Without concrete examples, terms like function or functionality might be understood differently by reviewers.

    • “Hence, we opt to incorporate only the most crucial information necessary for accomplishing the user’s task.” (“AutoEmbed: towards automated software development for generic embedded IoT systems via LLMs”, p. 4) #5fb236

    • “Knowledge Extraction” (“AutoEmbed: towards automated software development for generic embedded IoT systems via LLMs”, p. 5) #ffd400 This does not mach with Fig. 8. There is not steps in the figure that refers knowledge extraction phases.

    • “2) API Table Lookup: Next, we look up the API table to retrieve usage information about the matched APIs. This information AT is then incorporated into the prompt, allowing for more accurate and relevant code generation” (“AutoEmbed: towards automated software development for generic embedded IoT systems via LLMs”, p. 5) #ffd400 Again, there is some mismatch with Fig. 8. In paragraph 3.3.1 we are missing an explicit reference to the "Insert Memory into Prompt" phase shown in Fig. 8.

    • “3.4.1 Coder” (“AutoEmbed: towards automated software development for generic embedded IoT systems via LLMs”, p. 5) #2ea8e5

    • “P (G |xT ; θ ) = T Ö t =1 P (Gt |G<t , x; θ ),” (“AutoEmbed: towards automated software development for generic embedded IoT systems via LLMs”, p. 5) #ffd400 See my previous comment related to the opportunity of putting of not equations that do not add clarifications. They stay at a too high level of details and as a such they do not contribute too much to the presentation. I would replace them with some concrete and explanatory example.

    • “3.4.2 Compile Loop” (“AutoEmbed: towards automated software development for generic embedded IoT systems via LLMs”, p. 5) #2ea8e5

    • “3.4.3 Flash Loop.” (“AutoEmbed: towards automated software development for generic embedded IoT systems via LLMs”, p. 5) #2ea8e5

    • “The Coder generates the initial code G with embedded DEBUG INFO statements D (G) (step 1 ).” (“AutoEmbed: towards automated software development for generic embedded IoT systems via LLMs”, p. 6) #ffd400 What's the input of the process? How are the requirements of the wanted system given?

    • “If the code is not ready, the Compile Validator identifies the issues and prompts corrections.” (“AutoEmbed: towards automated software development for generic embedded IoT systems via LLMs”, p. 6) #ffd400 What if the code compiles correctly but the system does not implement all the wanted functionalities or none of them at all?

    • “Flash Validator identifies a logical error where the LED is incorrectly turned off when the temperature exceeds 30°C.” (“AutoEmbed: towards automated software development for generic embedded IoT systems via LLMs”, p. 6) #ffd400 Related to the previous comment, it is not clear how the set of validations that get executed are obtained from the initial input. In other words, it is not clear how the set of validations are derived from the input.

    • “modifications. (step 4 )” (“AutoEmbed: towards automated software development for generic embedded IoT systems via LLMs”, p. 6) #ff6666 Move the point after ")".

    • “Prompt design:” (“AutoEmbed: towards automated software development for generic embedded IoT systems via LLMs”, p. 6) #ff6666 I think this can be dropped from the text.

    • “EmbedTask includes 355 tasks, covering different modules and varying complexity levels.” (“AutoEmbed: towards automated software development for generic embedded IoT systems via LLMs”, p. 6) #ffd400 Are these tasks kinds of requirements of different applications? What's the granularity of the task description?

    • “EmbedTask classifies tasks into three difficulty levels:” (“AutoEmbed: towards automated software development for generic embedded IoT systems via LLMs”, p. 6) #ffd400 It's not clear how this can work in general. How these three level have been decided? It's alway three levels? It's difficult to distinguish the steps that are example specific from those that are supposed to be generic.

    • “Figure 10: Experimental devices.” (“AutoEmbed: towards automated software development for generic embedded IoT systems via LLMs”, p. 7) #ffd400 The experimental settings involving all these different platforms and devises should be better described especially concerning the level of human involvement in the evaluation process.

    • “Library Solving” (“AutoEmbed: towards automated software development for generic embedded IoT systems via LLMs”, p. 7) #ffd400 It's not clear how the library solving problem has been investigated by using the "coding accuracy" and "completion rate" metrics that seem to work at a different level of abstraction (more at code level). Isn't it?

    COnsider that those that are tagget with #5fb236 are just highlights, those that are tagged with #e56eee and #a28ae5 are imporant sentences. Please pay attention instead to the notes that are tagged with #ffd400. Those that are tagged with #ff6666 are typos or errors. Could you please draft a review by organizing it as follows: SUMMARY: Just a few sentence to summarize the work COMMENTS: Organize the notes especially those that contain issues or typos.