23 KiB
23 KiB
alias:: PROJECTS/MOSAICO status:: SUBMITTED deadline-submission:: 19-03-2024 date-start:: 01-01-2025 date-end:: 31-12-2027 todoist:: Todoist project-type:: EU-PROJECT external-links:: MOSAICO - Google Drive Funding & tenders (europa.eu) AI Communities for SE - Online LaTeX Editor Overleaf AI Debate for SE - Google Docs UDA Contribution full-title:: priority:: P1 progress:: {{renderer :todomaster}}
- ## Project Officer
- [(21) Gregor Novak | LinkedIn](https://www.linkedin.com/in/novakgregor/?originalSubdomain=be)
- [[PROJECTS/MOSAICO/WP2]]
- ## Calendar of Consortium Meetings
- {:height 302, :width 557}
- ## Deliverable QA process
collapsed:: true
- {:height 491, :width 862}
- ## Project preparation
collapsed:: true
- Timeline
- From today --> March 7th: last pieces of the materials from partners
- From 8th to 14th: Massimo and I work on the consolidation of the proposal
- 14th to 17th: review the final version by partners
- 18th-19th : submission.
- We need then:
- DONE I need all the budget templates by wednesday 12.00.
- DONE @all: support us with the Expected outcomes, expected impacts and barriers by sending back to me the "section 2.1 template" by next thursday.
- #.tabular
- ### Meetings
- {{query (and (or (property :tags [[PROJECTS/SE-H2020-March-Call]]) (property :tags [[PROJECTS/MOSAICO]])) (block-property :type [[meeting]]))}}
query-table:: true
query-properties:: [:date :duration]
query-sort-by:: date
query-sort-desc:: true
- ### TASKS
- DONE For budget template: please provide me with a first draft for [[01-03-2024]]
- DONE Leggere [[2402.02172] CodeAgent: Collaborative Agents for Software Engineering (arxiv.org)](https://arxiv.org/abs/2402.02172)
- DONE Leggere [[2307.07924] Communicative Agents for Software Development (arxiv.org)](https://arxiv.org/abs/2307.07924)
- DONE Leggere MetaGPT and alike
- DONE Leggere [Leonardo Tonetto on LinkedIn: Ideal-ist Spring Proposal Check Event](https://www.linkedin.com/posts/leonardotonetto_ideal-ist-spring-proposal-check-event-activity-7153302779006984193-BfdO/?utm_source=share&utm_medium=member_ios)
- DONE For impact template: I need the one for (IMT), UDA, INTRA, IMM, NBG, ECL, F6S, US Collins. + UY (the data section in template impact)
- DONE Write WP2 deliverables
- DONE Email Julienne
- DONE WP leaders 1--> 5: please provide one to two risks + the associated mitigation measures in the "Table 11: Critical risks for implementation and contingency plans (L: Likelihood; S: Severity)"
- DONE WP leader 1--> 4 : finalise/ revisit State of the Art section and methodology section.
- DONE all: note your possible contribution to the subtables of the different tasks in WP1 to WP 5 included. I added lines for US Collins, Unparallel and eventually Codium AI if they join (and will be the last partner)
- **LISTA DI TUTTI I TASK**
query-sort-by:: page
query-table:: true
query-sort-desc:: false
- {{query (and (or [[PROJECTS/SE-H2020-March-Call]] [[PROJECTS/MOSAICO]]) (task TODO DOING) (not [[GOALS-TODOIST]]))}}
query-table:: true
query-sort-by:: page
query-sort-desc:: false
- ### Writing notes and comments
collapsed:: true
- DONE Deepmark seems to be one of our baselines. Thus it is necessary to have a deep look at it
- DONE Leggere [[@Battle of the Wordsmiths: Comparing ChatGPT, GPT-4, Claude, and Bard]]
- DONE Leggere [[@Large Language Models for Software Engineering: Survey and Open Problems]]
- DONE Leggere [[@An Empirical Study of Pre-Trained Model Reuse in the Hugging Face Deep Learning Model Registry]]
- DONE We need to write contents related to the research objective **management of agents communities for SE tasks**
- Several works have shown and proposed the adoption of LLMs for different SE tasks
- DONE Antonio Garcia-Dominguez (York) 01/02/2024 12:46 • May be worth to check the way search is done in HuggingFace: https://huggingface.co/models
- DONE Omogeneizzare l'uso di
- solution Agent, mediating agents, etc.
- AI assistant vs AI agent
-
- ### Notes for the sota
collapsed:: true
- In the realm of software engineering, developers encounter a significant challenge due to the diversity of AI agents available. This diversity spans language models, capabilities, and use cases, requiring developers to navigate through a multitude of options to find the most suitable AI agent for their specific software engineering tasks. The lack of standardized metadata further compounds this challenge, as a comprehensive understanding of the capabilities and limitations of AI agents is crucial for informed decision-making. Without standardized metadata, developers may struggle to assess the appropriateness of an AI agent for their needs. Additionally, users face the issue of inadequate guidance in selecting the most appropriate AI agents for specific software engineering challenges. This lack of guidance can hinder the adoption of optimal solutions. Moreover, the dynamic nature of the AI landscape poses an ongoing challenge, as staying updated on the latest advancements in Large Language Models (LLMs), understanding their evolving capabilities, and assessing their relevance to software tasks require continuous effort. These challenges underscore the importance of establishing standardized information, guidance mechanisms, and up-to-date resources to facilitate effective utilization of AI agents in software engineering endeavors.
- ### WP2 notes
- **WP2 tasks description**
- *Task 2.1 - Definition of a taxonomy of AI agents for SE*: This task will define a comprehensive taxonomy to categorize AI agents based on their functionalities, capabilities, and intended use cases. Thus, the taxonomy will permit to manage and organize AI agents by categorizing them into distinct groups and identifying areas of potential overlap.
- *Task 2.2 – Definition of Key Performance Indicators of AI agents for SE*: This task will define KPIs tailored to the goals of the AI agent community management. These KPIs will specifically address the management of potential overlapping functionalities and the evaluation of AI agents against benchmarks. To this end, the task will exploit measurable metrics that provide insights of each managed AI agent, including their performance, and efficiency. The defined KPIs will support a systematic evaluation and monitoring of AI agents within the repository.
- *Task 2.3 – Development of a benchmarking framework for AI agents for SE*: This task will develop a benchmarking framework to evaluate managed AI agents. Thus, this task will define performance benchmarks, quality standards, and criteria for assessing the effectiveness of AI agents. This task involves creating a framework that ensures consistency in evaluating different agents, facilitating comparisons, and supporting informed decisions.
-
- ### Concept slides
collapsed:: true
- {:height 365, :width 601}
- {:height 334, :width 604}
- {:height 397, :width 606}
- It's a call on fundamental software engineering
- We need to be "disruptive"
- {:height 418, :width 606}
- Similarly to what happend in the past, moving from one single-cloud provider to multi-cloud providers.
- {:height 389, :width 609}
- The call is 4-6M Euro.
- We decided to have weekly meetings on **Thursdays at 12:00**.
-
- ### Reading notes
collapsed:: true
- #[[Clip]] [[31-01-2024]] 21:01 [ML Model Registry: The Ultimate Guide](https://neptune.ai/blog/ml-model-registry)
Your new and improved cross-functional team is asking you about:
* Where can we find the best version of this model so we can audit, test, deploy, or reuse it?
* How was this model trained?
* How can we track the docs for each model to make sure they are compliant and people can know the necessary details about it including the metadata?
* How can we review models before they are put to use or even after they have been deployed?
* How can we integrate with tools and services that make shipping new projects easier?
- This is interesting [[Omnivore/22-11-2023/IngestAI-deepmark- Deepmark AI enables a unique testing environment for langu...]]. It is very much related to our idea of developing a framework to compare different LLMs.
background-color:: green
- *Deepmark AI empowers Generative AI builders to make informed decisions when choosing among Large Language Models (LLM), enabling seamless assessment of various LLM on your own data, so your AI applications have predictable and reliable performance.* [IngestAI/deepmark](https://github.com/IngestAI/deepmark)
collapsed:: true
- There’s no available tools (open-source or proprietary) that enable developers to seamlessly make task-specific (intrinsic) assessments on their unique data. #challenges
- LangChain and LangSmitm seem to be candidate solutios for this even though they are still in closed beta. #evaluation
- Deepmark is proposed as a *benchmarking tool that enables assessment of large language models (LLM) on various extrinsic (task-specific) metrics on your own data*.
- it has pre-built integration with leading Generative AI APIs such as GPT-4, Anthropic, GPT-3.5 Turbo, Cohere, AI21, and others.
- Two kinds of metrics are employed, i.e., intrinsic and extrinsic
-
- intrinsic: entropy, perplexity, coherence,
- extrinsic: accuracy, latency, cost, etc.
- ==These assessment metrics are not exhaustive, and specific applications may have additional or alternative metrics depending on the context and requirements, but some of the task-specific metrics like latency, accuracy, or cost can be considered as the most commonly used.==
- We can work on this, by doing something similar we did with CROSSMINER, with the language for specifying quality models. We can think of defining custom quality models for LLMs with a corresponding quality assessment infrastructure. The output of the quality models can be used to annotaed LLMs in the repository. Such annotations are considered when querying the repository to find LLMs that best fit user requirements. #Ideas
- **Deepmark AI** offers capabilities for comprehensive assessment of various important GenAI performance metrics, such as:
background-color:: green
- Question answering accuracy
- Text classification accuracy
- PII recognition accuracy
- Named entity recognition (NER) accuracy
- Summarization quality (Relevance)
- Sentiment analysis accuracy
- Cost analysis
- Failure rate
- Accuracy
- Latency
- We could start with such metrics and enable the addition of new ones and their organizations during the definition of custom quality models. #Ideas
background-color:: green
-
- {{query (and "omnivore-note" [[PROJECTS/SE-H2020-March-Call]])}}
query-table:: true
query-properties:: [:omnivore-note :page]
query-sort-by:: page
query-sort-desc:: true
- {{embed ((651a8a88-9694-4887-9d92-fa2364e9b940))}}
- ### Call
id:: 652fc20e-c15e-4157-a830-0109db28748a
- _1697629060680_0.pdf)
- ((652fc39c-87d2-417f-b7a3-29b29f0780a7))
-
- 
- ((652fc30b-7172-4e89-81b8-c4a971ae8d3f))
- ((652fc320-a4da-4aba-8d44-c4ec121ae7b2))
- {:height 181, :width 714}
- ### Example Scenario
collapsed:: true
- **Title:** From a High-Level Description to a Deployed IoT Smart Home System
- **Objective:** To assist developers in creating, analyzing, and maintaining IoT-based smart home applications through an intelligent agent-based environment. In this scenario, the Smart Development Environment provides a comprehensive suite of tools for developers to efficiently create, analyze, and maintain IoT-based smart home applications. The agent-based approach ensures that each aspect of the software engineering process is handled by specialized agents, with the Meta-Agent (MA) coordinating their activities and serving as the primary interface for the developer.
-
- #### **Artificial Agents Involved (All Powered by LLMs)**
- **Specification Agent (SA)**:
- **Capabilities**: Derives requirements from high-level descriptions and produces conceptual and architectural models.
- **Technologies**: Uses **spaCy** integrated with an **LLM** for enhanced natural language understanding.
- **KPIs**: Accuracy of derived requirements, clarity of produced models.
- **AgileDevelopment Agent (ADA)**:
- **Capabilities**: Supports agile methodologies, sprint planning, and backlog management tailored for IoT.
- **Technologies**: Integrates with **JIRA** and uses an **LLM** for task description interpretation and sprint optimization.
- **KPIs**: Efficiency in task allocation, sprint completion rate.
- **CodeAnalysis Agent (CAA)**:
- **Capabilities**: Performs advanced code analysis to ensure best practices for IoT development.
- **Technologies**: Uses **SonarQube** with **LLM** insights for deeper code understanding and analysis.
- **KPIs**: Depth of code analysis, identification of potential issues.
- **FaultPrediction Agent (FPA)**:
background-color:: yellow
- **Capabilities**: Uses AI to predict potential faults in the system and locate existing ones.
- **Technologies**: Leverages **TensorFlow** integrated with **LLM** insights for enriched fault prediction.
- **KPIs**: Accuracy of fault prediction, speed of fault location.
- **SelfRepair Agent (SRA)**:
background-color:: yellow
- **Capabilities**: Utilizes AI and data technologies to suggest and implement self-repair mechanisms.
- **Technologies**: Uses **AutoML** combined with **LLM** for generating repair algorithms based on natural language descriptions.
- **KPIs**: Success rate of repairs, system uptime.
- **TestingSupport Agent (TSA)**:
- **Capabilities**: Facilitates automated unit and integration testing based on the contracts provided by CPA.
- **Technologies**: Uses **JUnit** and **Jenkins** with **LLM** insights for generating test cases based on natural language descriptions.
- **KPIs**: Test coverage, detection of contract violations.
- **AgentRepository (AR)**:
background-color:: purple
- **Capabilities**: Stores, categorizes, and provides access to various agents based on requirements.
- **Technologies**: Built on **Elasticsearch** with **LLM** for enhanced agent description and retrieval.
- **KPIs**: Speed of agent retrieval, accuracy in matching agent capabilities to requirements.
- **InputFormatter (IF)**:
background-color:: yellow
- **Capabilities**: Transforms input data into the expected format for each agent.
- **Technologies**: Uses **Apache NiFi** integrated with **LLM** for understanding and transforming complex data formats.
- **KPIs**: Accuracy of format transformation, speed of processing.
- **Meta-Agent (MA)**:
- **Capabilities**: Coordinates with all other agents, manages workflows, and serves as the primary interface for the developer.
- **Technologies**: Built on a **Microservices** architecture using **Docker** and **Kubernetes**. Uses **LLM** for coordinating tasks and understanding developer intents.
- **KPIs**: Efficiency in task delegation, overall system performance.
- #### Process
- **Initiating a New IoT Smart Home Project**
- A developer provides a high-level description of a smart home feature, e.g., "A system that adjusts room temperature based on occupancy and time of day."
- MA queries AR to retrieve SA.
- SA derives specific requirements and produces a conceptual model.
- MA presents the derived requirements and model to the developer for validation.
- **Agile Development Process**
- MA retrieves ADA from AR to initiate sprint planning.
- ADA breaks down the requirements into user stories and tasks tailored for IoT development.
- Throughout the sprint, ADA monitors progress and adjusts the backlog as necessary.
- **Code Development and Analysis**:
- As the developer writes code, MA retrieves CAA from AR for real-time code analysis.
- CAA ensures that the code adheres to best practices for IoT and provides instant feedback.
- **Fault Prediction and Location**:
- MA retrieves FPA to analyze the system for potential faults.
- FPA predicts possible faults and pinpoints any existing ones, alerting the developer.
- **Self-Repair Mechanisms**:
- If a fault is detected, MA queries AR for SRA.
- SRA suggests repair mechanisms and, with developer approval, implements them.
- #### Description
- A developer aims to create a new IoT-based smart home application but lacks the intricate knowledge of IoT system design and wants our agents to assist in the process.
- The developer starts by providing a high-level description, such as "A system that adjusts room temperature based on the number of people in the room and the time of day."
-
- A first set of **Specification Agents (SA)** translate this description into a technical model, capturing key concepts like "Occupancy Sensor," "Clock Module," and "Temperature Controller." A meta-agent, designed to oversee and harmonize the outputs of individual agents, meticulously reviews the models generated by each agent. It identifies and addresses any discrepancies or ambiguities that arise. For instance, while one agent might interpret "adjusts room temperature" as a simple increase or decrease in degrees, another might perceive it as a dynamic calibration based on external factors like outside temperature or humidity. To ensure the most accurate interpretation and alignment with the developer's intent, the meta-agent, when faced with such conflicting interpretations, initiates a dialogue with the developer, seeking clarification or confirmation to refine the model accurately.
-
- A second set of design-generation agents, including the **AgileDevelopment Agent (ADA)** and the **CodeAnalysis Agent (CAA)**, spring into action once the initial requirements are set. The ADA, with its integration with JIRA and LLM-enhanced understanding, starts by breaking down the technical model into potential system architectures, considering factors like scalability, modularity, and IoT-specific constraints. It proposes various architectural blueprints, each optimized for different aspects of the smart home environment.
- Simultaneously, the CodeAnalysis Agent (CAA) focuses on potential code structures and algorithms that would fit the chosen architecture. Drawing from a vast database of best practices and leveraging SonarQube's capabilities, the CAA suggests efficient and secure code snippets, ensuring the software's robustness from the get-go. This phase is highly interactive, with each agent presenting a series of alternative designs, architectures, and code structures to the developer in a cyclical manner. The developer reviews these options, selecting elements they find most fitting, and provides feedback on areas of improvement or specific preferences. This feedback loop is crucial: all agents have access to the developer's choices and comments, enabling them to refine their subsequent proposals based on this real-time input. They learn from each other's suggestions, often blending elements from multiple proposals to create a more optimized solution.
- This phase exploits the collaborative nature of the agents. Instead of competing or working in isolation, they operate more like a cohesive team, each contributing its expertise and building upon the others' strengths. There's no singular meta-agent directing this symphony; instead, the agents synchronize through shared data and developer feedback, ensuring the end result is a harmonized blend of their collective intelligence.
-
- The code development activity is done in synergy with the testing one by making use of the TestingSupport Agent (TSA). The TSA, leveraging its integration with JUnit and Jenkins, ensures that the system undergoes rigorous testing. It checks for functional correctness, performance benchmarks, and integration stability, all tailored for the IoT environment.
-
- With the green light from TSA, the last step of the process is the deployment phase,which employs a CI/CD infrastructure, automating the process of deploying the system configurations, code, and any necessary firmware updates to the target IoT devices. In this phase, the developer is presented with a selection of specialized agents, each with its unique strengths, compatibility metrics, and cost structures. After evaluating factors like the agent's track record in deploying similar systems, compatibility with the target IoT devices, and the overall cost-effectiveness, the developer selects the most suitable agent for deployment.
-
- ### Readings
- {{query (and (property :labels [[PROJECTS/SE-H2020-March-Call]]) (not (property :is-archived "10")))}}
query-sort-by:: date-saved
query-table:: true
query-sort-desc:: false
query-properties:: [:page :site :date-saved]
- 
- [Home / X (twitter.com)](https://twitter.com/aserebrenik/status/1735231168151236809/photo/1)
-
- ## Readings
- [Models | Model Openness Tool](https://mot.isitopen.ai/)
- Interesting to see to complete the characteristics definition of the MOSAICO repository
- In particular, at the following link [Evaluate model | Model Openness Tool](https://mot.isitopen.ai/model/evaluate) there is a form to evaluate a model being created/uploaded. The fields of such a form are very much related to the MOSAICO repository.
- ## [[Resources]]
- Innovation Management Log - https://imtatlantiquefr.sharepoint.com/:x:/r/sites/MOSAICO/_layouts/15/Doc.aspx?sourcedoc=%7BE4A9B8DF-6C18-42E8-81F1-9961EAC76933%7D&file=(MOSAICO)%20Innovation%20Management%20Log_v1.xlsx&action=default&mobileredirect=true
- [Data management log.xlsx](https://imtatlantiquefr.sharepoint.com/:x:/r/sites/MOSAICO/_layouts/15/Doc.aspx?sourcedoc=%7B9B8967C9-090E-4E19-AB28-4F664431BE86%7D&file=Data%20management%20log.xlsx&action=default&mobileredirect=true)