tags:: #todoist-task, [[PROJECTS/MOSAICO]] date:: [[25-08-2025]] - 08:38 progress:: {{renderer :todomaster}} - ### References - [Explore models in AI Toolkit](https://code.visualstudio.com/docs/intelligentapps/models) - [AI Toolkit for Visual Studio Code - Visual Studio Marketplace](https://marketplace.visualstudio.com/items?itemName=ms-windows-ai-studio.windows-ai-studio) - [Specification - Agent2Agent (A2A) Protocol](https://a2a-protocol.org/latest/specification/#554-agentskill-object) - {{embed ((6863b98d-74e6-435a-8631-c03155c89300))}} - ### Tasks - TODO Guardare i TODOs da [[Missione ROMA Mosaico]] #todoist-task #PROJECTS/MOSAICO #meeting - DONE Finalizzare revisione di Section 3 - [[07-08-2025]] id:: 6894a6d1-0613-4b32-ac49-0c3a3fd0753d - DONE Rivedere Section 5 - [[07-08-2025]] id:: 6894a6a2-bbab-4ec6-bcf4-1821588feb0f - DONE Da finalizzare Tabella 4 con lista di marketplace e agent repositories. id:: 68a423d6-ddac-4f6f-b162-51318a682db6 :LOGBOOK: CLOCK: [2025-09-09 Tue 15:38:36]--[2025-09-09 Tue 15:38:36] => 00:00:00 :END: - DONE Raffinare il metamodello in Section 6 - [[08-08-2025]] id:: 68960139-341c-40fa-9a35-04c20f74ad14 :LOGBOOK: CLOCK: [2025-08-19 Tue 14:43:06]--[2025-08-19 Tue 14:43:07] => 00:00:01 :END: - DONE Aggiornare il metamodello guardando quanto da Antonio ne suo deliverable di WP1 id:: 68960139-34fc-4ac4-8745-f8a2bb7cb6e9 - [imtatlantiquefr.sharepoint.com/sites/MOSAICO/Shared Documents/Forms/AllItems.aspx?id=%2Fsites%2FMOSAICO%2FShared Documents%2FWork Packages%2FWP1%2FD1%2E1%2F20250801 draft - only Sections 2 and 3%2Epdf&parent=%2Fsites%2FMOSAICO%2FShared Documents%2FWork Packages%2FWP1%2FD1%2E1](https://imtatlantiquefr.sharepoint.com/sites/MOSAICO/Shared%20Documents/Forms/AllItems.aspx?id=%2Fsites%2FMOSAICO%2FShared%20Documents%2FWork%20Packages%2FWP1%2FD1%2E1%2F20250801%20draft%20%2D%20only%20Sections%202%20and%203%2Epdf&parent=%2Fsites%2FMOSAICO%2FShared%20Documents%2FWork%20Packages%2FWP1%2FD1%2E1) - Da guardare particolarmente e' [Agntcy](https://docs.agntcy.org/#vision) - DONE Inserire in Section 7 una prima bozza dell'architettura del Repository id:: 68960139-bbd3-44c1-a39a-5f0782e5bfad :LOGBOOK: CLOCK: [2025-08-19 Tue 18:09:34]--[2025-08-26 Tue 23:06:08] => 172:56:34 :END: - Partire da qualche slide in WP5? - DONE Scrivere le conclusioni id:: 68960139-c4ea-441c-9d45-89381bddcfb9 - DONE Lavorare su capitolo 4 a partire dal contenuto del paper EMSE id:: 68960139-1246-4fb7-a933-cc75804483b7 :LOGBOOK: CLOCK: [2025-08-19 Tue 18:09:37]--[2025-08-21 Thu 16:42:09] => 46:32:32 :END: - DONE Rimuovere dalla bib tutte le occorrenze univaq id:: 68ac6e66-ba4e-41e7-9637-5fe847dfc8ba :LOGBOOK: CLOCK: [2025-08-25 Mon 16:26:24]--[2025-08-25 Mon 16:26:24] => 00:00:00 :END: - DONE Ripassata finale prima di mandare ai revisori collapsed:: true - ### Notes id:: 689467c8-88b1-45b0-b58c-99fa0d0b73a8 - Review di Antonio Garcia ( ![D2_1-InternalReview_version_20250827-agd220250907.pdf](../assets/D2_1-InternalReview_version_20250827-agd220250907_1758720457899_0.pdf) ) - DONE Quick note about the English: I see a lot of "agents monitoring" which should be "agent monitoring", for example. In general, when you have "noun verb-ing", "noun" tends to be singular. id:: 68d3ea95-45ab-43bb-86c6-506a00ca5420 - DONE Did you not consider LangGraph for Ch3? If so, why? (It's a big player, so this was somewhat surprising - did you run into issues with the license, or did you not consider it as a MAS framework?) id:: 68d3ea95-6c8a-40f9-93ff-5d930e6a7174 :LOGBOOK: CLOCK: [2025-09-24 Wed 16:01:00]--[2025-09-24 Wed 16:01:02] => 00:00:02 :END: - DONE Figure 3 doesn't really explain the notation used. id:: 68d3ea95-7a33-412c-a9e0-effaf3e484a0 - DONE For FC3.3 (continual evolution), have you considered options like Letta or MemGPT for self-improving agents? - DONE For the METAGENTE study: in the optimisation loop, do you evaluate the new prompt based on how it does across a population of repositories, or is it only against one repository? The latter sounds like it could have a risk of over-fitting. id:: 68d3ea95-2936-4efb-8e99-d9282435fff9 :LOGBOOK: CLOCK: [2025-09-24 Wed 16:01:22]--[2025-09-24 Wed 16:01:23] => 00:00:01 :END: - DONE Figure 13(a) has an odd "5" before "ROUGE-1". - DONE In Section 4.3.4, besides doing the Wilcoxon paired test for statistical significance, I'd also suggest looking at effect sizes. My intuition is that even if the differences are statistically significant, the effect sizes will be very small. id:: 68d3ea95-00ff-47d6-b92f-0243f944f2c0 - TODO In Figure 9, it feels like each spider chart has its own axis limits. They should all be using the same axis limits, so that the spider charts can be compared to each other. - DONE The report never defines what PTM stands for (used in Ch5). id:: 68d3ea95-f822-42f4-a4f6-6da2d5adfb75 - DONE Given the talk we attended from Barahona at LLMA4SE 2025 on the various levels of "openness" of LLMs, your discussion of AI Agent List [77] sounds like this marketplace may be oversimplifying things. It may be worth mentioning the need for more nuance in that regard? id:: 68d3ea95-8858-468b-9c3a-4895f500a9f9 - ((68bac2bb-1db8-41cb-81ed-7ef7f76e1491)) - DONE Please take a close look at my comments on Figure 24. I think the metamodel should be expanded in a number of ways. For example, in OpenTelemetry, spans are organised into a tree - this doesn't seem to be reflected in TelemetryRecord. id:: 68d3ea95-f018-4b23-9c51-b0ee94287661 :LOGBOOK: CLOCK: [2025-09-26 Fri 18:20:20]--[2025-09-26 Fri 18:20:21] => 00:00:01 :END: - TODO I also think that Figure 25 needs some improvements: it should stay closer to UML object diagram notation, and it may be best to split it into static parts (about the agents themselves and their relationships) and dynamic parts (about the traces for specific executions of those agents). In fact, I would divide the trace for the separate phases, and I'd suggest expanding further the instantiation (making it more detailed and using all the types in the taxonomy). - TODO For the MOSAICO agent repository architecture: would it make sense to integrate some type of vector database to allow for semantic search over the descriptions of the agents? It's quite likely that we will leave it up to the LLM in the reference agent to invoke your MCP tool for finding the most relevant agents, and that will most likely require semantic search. - DONE Incidentally, I don't think MCP is documented via OpenAPI: MCP is JSON-RPC based rather than REST. id:: 68d3ea95-b046-49bf-9509-651f1e88b06e - TODO For telemetry dashboards, you mention Grafana. That's good for metrics, but can it display traces as well, or should we use LangFuse for that? - Da considerare anche il prompt? - {{renderer :mermaid_68668a24-ce43-472e-b787-a53b895f7c75, 3}} collapsed:: true - ```mermaid classDiagram class Agent { id name version owner } class Capability { domain taskType supportedLanguages inputFormats outputFormats } class PerformanceKPI { accuracy latencyMs resourceConsumption robustnessMetrics } class Governance { license dataResidency GDPRCompliance auditTrailAvailable } class FairnessEthics { biasDetected fairnessConstraints explanationCapabilities } class RuntimeEnvironment { runtime requiredLibraries hardwareAcceleration } class Provenance { trainingDataSources modelLineage lastUpdated } class ContactInfo { maintainerEmail documentationUrl } class AgentRepository { <> } Agent --> Capability : has Agent --> PerformanceKPI : provides Agent --> Governance Agent --> FairnessEthics Agent --> RuntimeEnvironment Agent --> Provenance Agent --> ContactInfo AgentRepository --> Agent : manages ``` - Agents should be also marked if they keep information or not for IP purposes. - - {{renderer :plantuml_wqrztfdo}} - ```plantuml @startuml ' Enumerations enum FeedbackType { EXPLICIT IMPLICIT } enum MemoryType { SHORT_TERM LONG_TERM } enum TelemetryOutput { LOG TRACE BAGGAGE } ' Core entities class Agent { ID: int name: string modelCard: string messageContent: string hyperparameter: string dependencies: Agent } class SolutionAgent { ID: int intentions: string desires: string backStory: string promptType: string role: string } class CollaborationAgent { ID: int collaborationPattern: Pattern } class SupervisionAgent { ID: int } class ConsensusAgent { ID: int } class Task { id: int description: string input: string output: string } class Tool { ID: int name: string API_key: string } class Memory { ID: int type: MemoryType db: string agent: Agent } ' Governance and usage class Provider { name: string } class Usage { ID: int name: string } class Remote { ID: int url: string provider: Provider } class Local { ID: int hardwareRequirement: string } class A2AProtocol { ID: int } ' Evaluation and monitoring class Benchmark { ID: int metadata: string features: string } class Metric { ID: int name: string threshold: double } class TelemetryData { ID: int tool: TelemetrySource outputFormat: TelemetryOutput } class TelemetrySource { ID: int } class HumanFeedback { ID: int userID: int feedback: FeedbackType } ' Relationships Agent <|-- SolutionAgent Agent <|-- CollaborationAgent Agent <|-- SupervisionAgent Agent <|-- ConsensusAgent Agent --> "1..*" Task : accomplishes Agent --> "0..*" Tool : exploits Agent --> "1..*" Memory : has Agent --> "0..*" Benchmark : evaluated by Agent --> "1" Provider : provided by Agent --> "0..*" Usage : uses Agent --> "0..*" A2AProtocol : complies with Benchmark --> "1..*" Metric : includes TelemetryData --> Benchmark : collected on TelemetryData --> HumanFeedback : stores HumanFeedback --> Agent : evaluates Usage <|-- Remote Usage <|-- Local @enduml ``` - collapsed:: true {{renderer :plantuml_ickelqar}} - ```plantuml @startuml "summarization-teacher-student-example" ' ====== STYLE (optional, safe to remove) ====== hide methods hide stereotypes skinparam shadowing false skinparam packageStyle rectangle skinparam class { BackgroundColor White BorderColor Black } skinparam object { BackgroundColor White BorderColor Black } skinparam note { BackgroundColor #F9FAFB BorderColor #E5E7EB } ' ====== CAPABILITY & PROTOCOL ====== object Capability_Summarization as "Capability: Summarization" { id = C-SUMM description = "Generate concise, accurate summaries of technical text" supportedLanguages = "EN, IT" } object Proto_TeacherStudent as "InteractionProtocol: Teacher–Student v1" { specUrl = "https://example.org/protos/teacher-student" } ' ====== TRAINING PHASE ====== package "Training Phase" as Training { object TeacherA as "SupervisionAgent: Teacher A" { ID = A-T1 name = "TeacherA" role = "Prompt Explorer" objective = "Generate/critique prompt candidates" } object TeacherB as "SupervisionAgent: Teacher B" { ID = A-T2 name = "TeacherB" role = "Prompt Critic" objective = "Score and refine prompts" } object BenchTrain as "Benchmark: README Summarization (Train)" { ID = B-TRAIN datasetRef = "gh-readme-train" protocolVersion = "1.0" } object KPI_Rouge as "PerformanceKPI: ROUGE" object KPI_BERT as "PerformanceKPI: BERTScore" object KPI_Lat as "PerformanceKPI: Latency" object M_Rouge1 as "Metric {name=ROUGE-1, unit=score}" object M_RougeL as "Metric {name=ROUGE-L, unit=score}" object M_BERT as "Metric {name=BERTScore, unit=score}" object M_Lat as "Metric {name=Latency, unit=ms}" object UsageTrain1 as "AgentUsage: TrainRun#1" { timestamp = 2025-08-05T10:12:00Z durationMs = 8420 cost = 0.12 } object TeleToolTrain as "TelemetryTool: TrainerLogger {format=JSON}" object TeleRecTrain as "TelemetryRecord {kind=TRACE}" ' Tools and memory used in training object ToolRetriever as "Tool: DomainRetriever" { authMethod = "API Key" scopes = "read" } object STMem as "Memory {type=SHORT_TERM, scope=AGENT}" object LTMem as "Memory {type=LONG_TERM, scope=SHARED, db=vectorDB}" ' Links within training TeacherA --> Capability_Summarization : exposes TeacherB --> Capability_Summarization : exposes TeacherA --> ToolRetriever : exploits TeacherB --> ToolRetriever : exploits TeacherA --> STMem : has TeacherB --> STMem : has TeacherA --> LTMem : has TeacherB --> LTMem : has TeacherA --> Proto_TeacherStudent : supports TeacherB --> Proto_TeacherStudent : supports BenchTrain --> TeacherA : evaluates BenchTrain --> TeacherB : evaluates BenchTrain --> KPI_Rouge : measures BenchTrain --> KPI_BERT : measures BenchTrain --> KPI_Lat : measures KPI_Rouge --> M_Rouge1 : includes KPI_Rouge --> M_RougeL : includes KPI_BERT --> M_BERT : includes KPI_Lat --> M_Lat : includes UsageTrain1 --> TeacherA : agent TeleToolTrain --> UsageTrain1 : collects TeleToolTrain --> TeleRecTrain : produces note right of UsageTrain1 Prompt candidates explored: - "Summarize in 3 sentences..." - "Provide a factual abstract..." Stored with scores (ROUGE/BERT). end note } ' ====== TESTING PHASE ====== package "Testing Phase" as Testing { object Student1 as "SolutionAgent: Summarizer-1" { ID = A-S1 name = "Summarizer-Abstractive" role = "Apply best prompt" objective = "Produce concise factual summary" } object Student2 as "SolutionAgent: Summarizer-2" { ID = A-S2 name = "Summarizer-Extractive" role = "Apply best prompt" objective = "Produce concise factual summary" } ' Best prompt produced in training (abstracted as context stored in memory) object BestPrompt as "Memory {type=LONG_TERM, scope=SHARED}" { db = "vectorDB: best_prompt_embeddings" } object BenchTest as "Benchmark: README Summarization (Test)" { ID = B-TEST datasetRef = "gh-readme-test" protocolVersion = "1.0" } object KPI_Rouge_T as "PerformanceKPI: ROUGE" object KPI_Lat_T as "PerformanceKPI: Latency" object M_RougeL_T as "Metric {name=ROUGE-L, unit=score}" object M_Lat_T as "Metric {name=Latency, unit=ms}" object UsageTest1 as "AgentUsage: TestRun#S1" { timestamp = 2025-08-12T15:44:00Z durationMs = 5100 cost = 0.08 } object UsageTest2 as "AgentUsage: TestRun#S2" { timestamp = 2025-08-12T15:45:00Z durationMs = 4300 cost = 0.06 } object TeleToolTest as "TelemetryTool: RuntimeLogger {format=JSON}" object TeleRecTest1 as "TelemetryRecord {kind=LOG}" object TeleRecTest2 as "TelemetryRecord {kind=TRACE}" ' Tools used in testing object ToolGlossary as "Tool: TechGlossary" { authMethod = "None" scopes = "public" } Student1 --> Capability_Summarization : exposes Student2 --> Capability_Summarization : exposes Student1 --> BestPrompt : has Student2 --> BestPrompt : has Student1 --> ToolGlossary : exploits Student2 --> ToolGlossary : exploits Student1 --> Proto_TeacherStudent : supports Student2 --> Proto_TeacherStudent : supports BenchTest --> Student1 : evaluates BenchTest --> Student2 : evaluates BenchTest --> KPI_Rouge_T : measures BenchTest --> KPI_Lat_T : measures KPI_Rouge_T --> M_RougeL_T : includes KPI_Lat_T --> M_Lat_T : includes UsageTest1 --> Student1 : agent UsageTest2 --> Student2 : agent TeleToolTest --> UsageTest1 : collects TeleToolTest --> UsageTest2 : collects TeleToolTest --> TeleRecTest1 : produces TeleToolTest --> TeleRecTest2 : produces note right of BestPrompt Best prompt selected from training (Teacher–Student protocol): "Summarize in ≤3 sentences, preserve factual references; avoid speculation." end note } ' ====== CONSENSUS & GOVERNANCE (applied after testing) ====== object Cons as "ConsensusAgent: Summarization-Consensus" { ID = A-C1 name = "ConsensusSummarizer" } object Gov as "GovernancePolicy: Accuracy-First" { rules = "Prefer factual consistency over brevity; break ties by ROUGE-L" } Cons --> Gov : implements Cons --> Student1 : aggregates candidate Cons --> Student2 : aggregates candidate note right of Cons Applies GovernancePolicy across candidate summaries from Student1/2: - Compare ROUGE-L + factuality signals - Select winner under policy rules end note @enduml ```