535 lines
22 KiB
Markdown
535 lines
22 KiB
Markdown
tags:: #todoist-task, [[PROJECTS/MOSAICO]]
|
||
date:: [[25-08-2025]] - 08:38
|
||
progress:: {{renderer :todomaster}}
|
||
|
||
- ### References
|
||
- [Explore models in AI Toolkit](https://code.visualstudio.com/docs/intelligentapps/models)
|
||
- [AI Toolkit for Visual Studio Code - Visual Studio Marketplace](https://marketplace.visualstudio.com/items?itemName=ms-windows-ai-studio.windows-ai-studio)
|
||
- [Specification - Agent2Agent (A2A) Protocol](https://a2a-protocol.org/latest/specification/#554-agentskill-object)
|
||
- {{embed ((6863b98d-74e6-435a-8631-c03155c89300))}}
|
||
- ### Tasks
|
||
- TODO Guardare i TODOs da [[Missione ROMA Mosaico]] #todoist-task #PROJECTS/MOSAICO #meeting
|
||
- DONE Finalizzare revisione di Section 3 - [[07-08-2025]]
|
||
id:: 6894a6d1-0613-4b32-ac49-0c3a3fd0753d
|
||
- DONE Rivedere Section 5 - [[07-08-2025]]
|
||
id:: 6894a6a2-bbab-4ec6-bcf4-1821588feb0f
|
||
- DONE Da finalizzare Tabella 4 con lista di marketplace e agent repositories.
|
||
id:: 68a423d6-ddac-4f6f-b162-51318a682db6
|
||
:LOGBOOK:
|
||
CLOCK: [2025-09-09 Tue 15:38:36]--[2025-09-09 Tue 15:38:36] => 00:00:00
|
||
:END:
|
||
- DONE Raffinare il metamodello in Section 6 - [[08-08-2025]]
|
||
id:: 68960139-341c-40fa-9a35-04c20f74ad14
|
||
:LOGBOOK:
|
||
CLOCK: [2025-08-19 Tue 14:43:06]--[2025-08-19 Tue 14:43:07] => 00:00:01
|
||
:END:
|
||
- DONE Aggiornare il metamodello guardando quanto da Antonio ne suo deliverable di WP1
|
||
id:: 68960139-34fc-4ac4-8745-f8a2bb7cb6e9
|
||
- [imtatlantiquefr.sharepoint.com/sites/MOSAICO/Shared Documents/Forms/AllItems.aspx?id=%2Fsites%2FMOSAICO%2FShared Documents%2FWork Packages%2FWP1%2FD1%2E1%2F20250801 draft - only Sections 2 and 3%2Epdf&parent=%2Fsites%2FMOSAICO%2FShared Documents%2FWork Packages%2FWP1%2FD1%2E1](https://imtatlantiquefr.sharepoint.com/sites/MOSAICO/Shared%20Documents/Forms/AllItems.aspx?id=%2Fsites%2FMOSAICO%2FShared%20Documents%2FWork%20Packages%2FWP1%2FD1%2E1%2F20250801%20draft%20%2D%20only%20Sections%202%20and%203%2Epdf&parent=%2Fsites%2FMOSAICO%2FShared%20Documents%2FWork%20Packages%2FWP1%2FD1%2E1)
|
||
- Da guardare particolarmente e' [Agntcy](https://docs.agntcy.org/#vision)
|
||
- DONE Inserire in Section 7 una prima bozza dell'architettura del Repository
|
||
id:: 68960139-bbd3-44c1-a39a-5f0782e5bfad
|
||
:LOGBOOK:
|
||
CLOCK: [2025-08-19 Tue 18:09:34]--[2025-08-26 Tue 23:06:08] => 172:56:34
|
||
:END:
|
||
- Partire da qualche slide in WP5?
|
||
- DONE Scrivere le conclusioni
|
||
id:: 68960139-c4ea-441c-9d45-89381bddcfb9
|
||
- DONE Lavorare su capitolo 4 a partire dal contenuto del paper EMSE
|
||
id:: 68960139-1246-4fb7-a933-cc75804483b7
|
||
:LOGBOOK:
|
||
CLOCK: [2025-08-19 Tue 18:09:37]--[2025-08-21 Thu 16:42:09] => 46:32:32
|
||
:END:
|
||
- DONE Rimuovere dalla bib tutte le occorrenze univaq
|
||
id:: 68ac6e66-ba4e-41e7-9637-5fe847dfc8ba
|
||
:LOGBOOK:
|
||
CLOCK: [2025-08-25 Mon 16:26:24]--[2025-08-25 Mon 16:26:24] => 00:00:00
|
||
:END:
|
||
- DONE Ripassata finale prima di mandare ai revisori
|
||
collapsed:: true
|
||
- ### Notes
|
||
id:: 689467c8-88b1-45b0-b58c-99fa0d0b73a8
|
||
- Review di Antonio Garcia (  )
|
||
- DONE Quick note about the English: I see a lot of "agents monitoring" which should be "agent monitoring", for example. In general, when you have "noun verb-ing", "noun" tends to be singular.
|
||
id:: 68d3ea95-45ab-43bb-86c6-506a00ca5420
|
||
- DONE Did you not consider LangGraph for Ch3? If so, why? (It's a big player, so this was somewhat surprising - did you run into issues with the license, or did you not consider it as a MAS framework?)
|
||
id:: 68d3ea95-6c8a-40f9-93ff-5d930e6a7174
|
||
:LOGBOOK:
|
||
CLOCK: [2025-09-24 Wed 16:01:00]--[2025-09-24 Wed 16:01:02] => 00:00:02
|
||
:END:
|
||
- DONE Figure 3 doesn't really explain the notation used.
|
||
id:: 68d3ea95-7a33-412c-a9e0-effaf3e484a0
|
||
- DONE For FC3.3 (continual evolution), have you considered options like Letta or MemGPT for self-improving agents?
|
||
- DONE For the METAGENTE study: in the optimisation loop, do you evaluate the new prompt based on how it does across a population of repositories, or is it only against one repository? The latter sounds like it could have a risk of over-fitting.
|
||
id:: 68d3ea95-2936-4efb-8e99-d9282435fff9
|
||
:LOGBOOK:
|
||
CLOCK: [2025-09-24 Wed 16:01:22]--[2025-09-24 Wed 16:01:23] => 00:00:01
|
||
:END:
|
||
- DONE Figure 13(a) has an odd "5" before "ROUGE-1".
|
||
- DONE In Section 4.3.4, besides doing the Wilcoxon paired test for statistical significance, I'd also suggest looking at effect sizes. My intuition is that even if the differences are statistically significant, the effect sizes will be very small.
|
||
id:: 68d3ea95-00ff-47d6-b92f-0243f944f2c0
|
||
- TODO In Figure 9, it feels like each spider chart has its own axis limits. They should all be using the same axis limits, so that the spider charts can be compared to each other.
|
||
- DONE The report never defines what PTM stands for (used in Ch5).
|
||
id:: 68d3ea95-f822-42f4-a4f6-6da2d5adfb75
|
||
- DONE Given the talk we attended from Barahona at LLMA4SE 2025 on the various levels of "openness" of LLMs, your discussion of AI Agent List [77] sounds like this marketplace may be oversimplifying things. It may be worth mentioning the need for more nuance in that regard?
|
||
id:: 68d3ea95-8858-468b-9c3a-4895f500a9f9
|
||
- ((68bac2bb-1db8-41cb-81ed-7ef7f76e1491))
|
||
- DONE Please take a close look at my comments on Figure 24. I think the metamodel should be expanded in a number of ways. For example, in OpenTelemetry, spans are organised into a tree - this doesn't seem to be reflected in TelemetryRecord.
|
||
id:: 68d3ea95-f018-4b23-9c51-b0ee94287661
|
||
:LOGBOOK:
|
||
CLOCK: [2025-09-26 Fri 18:20:20]--[2025-09-26 Fri 18:20:21] => 00:00:01
|
||
:END:
|
||
- TODO I also think that Figure 25 needs some improvements: it should stay closer to UML object diagram notation, and it may be best to split it into static parts (about the agents themselves and their relationships) and dynamic parts (about the traces for specific executions of those agents). In fact, I would divide the trace for the separate phases, and I'd suggest expanding further the instantiation (making it more detailed and using all the types in the taxonomy).
|
||
- TODO For the MOSAICO agent repository architecture: would it make sense to integrate some type of vector database to allow for semantic search over the descriptions of the agents? It's quite likely that we will leave it up to the LLM in the reference agent to invoke your MCP tool for finding the most relevant agents, and that will most likely require semantic search.
|
||
- DONE Incidentally, I don't think MCP is documented via OpenAPI: MCP is JSON-RPC based rather than REST.
|
||
id:: 68d3ea95-b046-49bf-9509-651f1e88b06e
|
||
- TODO For telemetry dashboards, you mention Grafana. That's good for metrics, but can it display traces as well, or should we use LangFuse for that?
|
||
- Da considerare anche il prompt?
|
||
- {{renderer :mermaid_68668a24-ce43-472e-b787-a53b895f7c75, 3}}
|
||
collapsed:: true
|
||
- ```mermaid
|
||
classDiagram
|
||
class Agent {
|
||
id
|
||
name
|
||
version
|
||
owner
|
||
}
|
||
|
||
class Capability {
|
||
domain
|
||
taskType
|
||
supportedLanguages
|
||
inputFormats
|
||
outputFormats
|
||
}
|
||
|
||
class PerformanceKPI {
|
||
accuracy
|
||
latencyMs
|
||
resourceConsumption
|
||
robustnessMetrics
|
||
}
|
||
|
||
class Governance {
|
||
license
|
||
dataResidency
|
||
GDPRCompliance
|
||
auditTrailAvailable
|
||
}
|
||
|
||
class FairnessEthics {
|
||
biasDetected
|
||
fairnessConstraints
|
||
explanationCapabilities
|
||
}
|
||
|
||
class RuntimeEnvironment {
|
||
runtime
|
||
requiredLibraries
|
||
hardwareAcceleration
|
||
}
|
||
|
||
class Provenance {
|
||
trainingDataSources
|
||
modelLineage
|
||
lastUpdated
|
||
}
|
||
|
||
class ContactInfo {
|
||
maintainerEmail
|
||
documentationUrl
|
||
}
|
||
|
||
class AgentRepository {
|
||
<<service>>
|
||
}
|
||
|
||
Agent --> Capability : has
|
||
Agent --> PerformanceKPI : provides
|
||
Agent --> Governance
|
||
Agent --> FairnessEthics
|
||
Agent --> RuntimeEnvironment
|
||
Agent --> Provenance
|
||
Agent --> ContactInfo
|
||
AgentRepository --> Agent : manages
|
||
```
|
||
- Agents should be also marked if they keep information or not for IP purposes.
|
||
-
|
||
- <img src="https://www.plantuml.com/plantuml/png/VLNTRzem47_tNs7rfT8agcsFqJILKagHw42GIdkhJ78b5ewT-OCERVlVTuuXSKB42zZd-_ZTtUykbDuRoxHrfI2tT2fT2PfPhgGXW1Vw191lMRPFZnNGFuJIwS_LOZwPftYSlvoE_qXGVe5IwMEZkdbUhjEtTBf-mSjY-MCMpcVj50IKOFLnwMpbR6soMCxmDrsF9rFyVnZFPkFPr5lSqedIG45QRZaOaWbc31tlK50RpXyJogN5asGb9DHOpUKEhwNAGKoOpcCP6CDsC57IedtqK21qNJ6DBYpew2672cGECiFOIGZhKGKK6oMSBzaG6bw7yFNiU3DSGopn9TvOB5uagxGgA-kh6Gcr4ksrGJ1HGh2j2fqRWf752YjcCJkPqDCXoiHX_WTk1hr4qQG1QPovefCoiwzVU7x6W5bdcbSUGPGGbzZxwAvgCbocc2ebhdTwl9g_xU5uQHfesJEsTLrRndg0syWRszNKQ5fzIsVgWCLYCWFAP4wTfz0fmagh0yz1rp4wi1eGhrxzMWADvXensRwgqw9BZJfUqaHkpHSgO_rA5SZ_3wPX3RySCg-CMTyOZh-EqPjLMQ_GTVBJ0nEkPa-TVAaaHxfw-s3yWBDHb4plUv5na5ZEB8l0lmEpBgP_r2WKPDUxR0iqBPH0MkNARGL453alb4SCs6ytqYTfDJRAwGmQkZqfNJARz5VIWEjWECpvPrSoULwO_VuPqEtj_QIKTFPhgFGQH5Z01QyC2IFsxUzer5qpySFbz7VCUZFTjOmdcPoUHgFlzERBtTsdcp362MLPXhj8S5E0YPK-1oK_cWc5tvLGt9fBBwS1J9270-OjTT17e1daT7li-BbfPomvqx-l5Do5GKjynGU2nIHFQCWA3FtWjY2aHT81NvFIjouJ3fSOwVBCgyOf-5KBcKz1oG7LBaayiV6xPaXNx1K3zBOaXeJCwiQ57H5Bwh4dv1u_K_vRJiX_" />
|
||
{{renderer :plantuml_wqrztfdo}}
|
||
- ```plantuml
|
||
@startuml
|
||
' Enumerations
|
||
enum FeedbackType {
|
||
EXPLICIT
|
||
IMPLICIT
|
||
}
|
||
|
||
enum MemoryType {
|
||
SHORT_TERM
|
||
LONG_TERM
|
||
}
|
||
|
||
enum TelemetryOutput {
|
||
LOG
|
||
TRACE
|
||
BAGGAGE
|
||
}
|
||
|
||
' Core entities
|
||
class Agent {
|
||
ID: int
|
||
name: string
|
||
modelCard: string
|
||
messageContent: string
|
||
hyperparameter: string
|
||
dependencies: Agent
|
||
}
|
||
|
||
class SolutionAgent {
|
||
ID: int
|
||
intentions: string
|
||
desires: string
|
||
backStory: string
|
||
promptType: string
|
||
role: string
|
||
}
|
||
|
||
class CollaborationAgent {
|
||
ID: int
|
||
collaborationPattern: Pattern
|
||
}
|
||
|
||
class SupervisionAgent {
|
||
ID: int
|
||
}
|
||
|
||
class ConsensusAgent {
|
||
ID: int
|
||
}
|
||
|
||
class Task {
|
||
id: int
|
||
description: string
|
||
input: string
|
||
output: string
|
||
}
|
||
|
||
class Tool {
|
||
ID: int
|
||
name: string
|
||
API_key: string
|
||
}
|
||
|
||
class Memory {
|
||
ID: int
|
||
type: MemoryType
|
||
db: string
|
||
agent: Agent
|
||
}
|
||
|
||
' Governance and usage
|
||
class Provider {
|
||
name: string
|
||
}
|
||
|
||
class Usage {
|
||
ID: int
|
||
name: string
|
||
}
|
||
|
||
class Remote {
|
||
ID: int
|
||
url: string
|
||
provider: Provider
|
||
}
|
||
|
||
class Local {
|
||
ID: int
|
||
hardwareRequirement: string
|
||
}
|
||
|
||
class A2AProtocol {
|
||
ID: int
|
||
}
|
||
|
||
' Evaluation and monitoring
|
||
class Benchmark {
|
||
ID: int
|
||
metadata: string
|
||
features: string
|
||
}
|
||
|
||
class Metric {
|
||
ID: int
|
||
name: string
|
||
threshold: double
|
||
}
|
||
|
||
class TelemetryData {
|
||
ID: int
|
||
tool: TelemetrySource
|
||
outputFormat: TelemetryOutput
|
||
}
|
||
|
||
class TelemetrySource {
|
||
ID: int
|
||
}
|
||
|
||
class HumanFeedback {
|
||
ID: int
|
||
userID: int
|
||
feedback: FeedbackType
|
||
}
|
||
|
||
' Relationships
|
||
Agent <|-- SolutionAgent
|
||
Agent <|-- CollaborationAgent
|
||
Agent <|-- SupervisionAgent
|
||
Agent <|-- ConsensusAgent
|
||
|
||
Agent --> "1..*" Task : accomplishes
|
||
Agent --> "0..*" Tool : exploits
|
||
Agent --> "1..*" Memory : has
|
||
Agent --> "0..*" Benchmark : evaluated by
|
||
Agent --> "1" Provider : provided by
|
||
Agent --> "0..*" Usage : uses
|
||
Agent --> "0..*" A2AProtocol : complies with
|
||
|
||
Benchmark --> "1..*" Metric : includes
|
||
TelemetryData --> Benchmark : collected on
|
||
TelemetryData --> HumanFeedback : stores
|
||
HumanFeedback --> Agent : evaluates
|
||
Usage <|-- Remote
|
||
Usage <|-- Local
|
||
|
||
@enduml
|
||
|
||
|
||
```
|
||
- <img src="https://www.plantuml.com/plantuml/png/jLZDSkCs4hxhAL2QgiourhGbPhmxovItGiaSZJRwAv5EAhce81AIk2O9BW0grhgcAjUSznNoP7cID02I8YdAzcHhv-2HWA-RNpSQtP-8lkK2Cv754UhoB8un2_-3HKWJGn3ixmWpkCW2aWY3FE8uZKYtqtcBRjK_v7e_JXpqZgRI04UNYECDGO8YHcAw9nSvhhCB0u9Y8dOquFeB5uGHAWufuHt-42OfPZX6V8S3-dEORD46HvnKTbBiF-0jSSKX8k3T5pZPHbM07s7EqLC7eG50juncIJ2a4MNeXrqeY5odB21ChmqY07K-L-pf-b_WzNzma511MitVVFpxH_lZeE7WZNFZ_CqPW8jZEeVsmXwC9sFlH_GLMYpdtdmudnGvp0aESOhNOHIAmygj7XV277MFcnQgxNOLip10y133lPzEuLj0kCz2TNImt1sHX30CCVWqyKDEBX7s_Kojv7L1EA8R98Y_Iq8VH_3fKNJ14S_Ib391WWaSIWQdnAK_PtQ9nbvN1fXJNp0gwChJPUNggbAqnmbK0_Ob4uNnQMIX7FVxB_yjeFk-3eEdnBzdaNp8JeYKMzVNUMrUKRQzJgKBVjqetsujqzxI7i_6in5QVB9TfqXnNcIewp4S9h8C5plCILUIB9SaWpoWdAAjWd2pbB1zo248UmiFB2D0jgQDqFWEEDk6rrVV4XmJ6KFXfQjM6OtKAYGYJWLo7bEe5iBqfdvkk2VL0xk6KnJXlpE2KctZuoG80zZXqWh2RX8Ul41uq2HijX4Uj18UIZ9-2rtNXpWGK8FhkmcJWkq9nm59_9tAjcAflaBrFLXewTXtKwTUsEYTWbxK60yCTRvg0V80HoYMP2D9R7S68pY8YI6acQQPvZNtFM4ylmpzgrwJrtUByMf9CoWFIMj1s8Qo62S-WGtWDhyVETqwUk0ilLQmt53fQ1XCi6Z5mpfauT2ji9bgBds5dnB1GXyzoGEwLKoC_YNAab3SSlcOpzqJmyavmybvmpAUcbaPp7d38gwQNHvKRXNppzNmxZdSGtMoEa9Le6hHqZTncILl-iMXYp0cCCNY58xEx9atHk-3qRln-ZshRrgzta-w46GlW-ETokxquRtPKwi-vGA-zwxwvibTYOX7QNGiHRa2GumTv7BEWx09tMxXpZof4nEt_tJdishqqcX9_1Oli0hp03t110bkeM27Zax1MoJzStLLOfYZx80oJW84zY9lGXNt05tAj98za53-5Rax6iii5Jj5fd0cTbCrXsMLsuin-euSz2s0SqjrvvOtH5T_-HZNcv8uFqF5w4cExbltqtpfhJndERtKzhVso9bvrVWdxPQJ-MnKCtG_sKldxX85wzizs55sDoZICGcJ1uv-3iMkdeAo_Hh6FyxFHGkHnvHoq1desF--r48_evvi1OjeA4uylupJ_dHMBGH3fkcYTKjRJSvRLRUQPgtJrog6zuchb-1IpXmxjOgwU1e4lCTH9WVGMTJWUTInsLfm1p3Fs3cKQaelWMG3gc6E_YMaRAOMN38_WbZFOYQd6CL0Gt8o3O1ykjxNFCfjsAyskaOIiUnvwbYgZKY2gaOMjB0eWXl3Mw6rzcF9KHTaldwq4gmitEw4b7KLdweTB9fQGbSn8u6bzWqf8FIj8R8rVOquq8NK4tvrTTKjCE1cB_K-1aNlYmnK8rvp8SLUYNA5zAhkDrB3WwDtQXPTotHUN068G2kKTEiQpd6zPoGSJ8I6WijNAW8khsizOrmQPU8eXihecD5lw26tBk0gI3iF3ZHFNHtPQHeTq1eevBgdHI4jzE4K0hpCc9RVXwOWoScRBv0tc-JDS-ITn_y_zxTmFKi_HJdMPXjwLrG8h4Deu57-q81VZAfC0Ac7uaLLCdAHr-jh1ytd8Yl1Mj8lHewbmbnfUYiIhqaG0ASp8bL6yZgD2iWJYGhb-xn21QC_9L1NtgibAZIaC-XISvuenvNt9xJZyK5VAW8X2oqQ45Qb17IVrO1zq-lVME_Vjsl0ctwleG5x7-ggfoHWdYTWleB0JJk1zr-V4FZhEH5Qb5f3Wm85-TnNQj0obUqQT38VdR4olqovbd9LTzowMXr5b7De2HMnwX5_LootAzKPJKXJfgRPEjA_D5N6obx-QcrNDj0ljT3Fg7ITGepLF9xVxvSgiKn6JIJMtBm8wxzItjMylbhUoS9JWhHuM9jkQu3CPq7LFjKYtAemtJNQbTlAg-coLRlmAb65grFnfMv58yH2UrKwG2E-4tLMyAuuRTTc9q3p1M3j-bOqtFDGyrcvTonETQkgGvaJoGHky0Q-Ltzl8Jd8cY_bYjbqeMLXNHJ-_kjl5Lru2M322TiJ1Iqr0YCRmXJY6uJtD0pK6xuiKlTFYSHM_JUSprndvjwxw2iqcd_lB6VsREY0V02v4ahji17G4_CUTD5yZKeJhb-PmWUIy8mtb9D-1rxkwhQKIwYXr7-5W2eXHoKbtpNcpndHlNeC_4zO8gVhWaBNEbZ8LczN_OFnCMGmw9L_baMwjorKJieCWGRY8LUZ4ab7QCt87bhLD_83Ve3P0sRh0yh7i7hNgIAKDI4fGDtBzwCnXCYFExM2tpAoLJg_bFmdGFCSi5ClBca7qTZg83Xgnewmpw2V0QBqK7d5hAgkO7Pjoh8om6ESOaQA0D5VYjH04X0FjmcEk0AwgdZXnqIIG9go982_gNwgIcwdBANEj_0nYwCEwlm1" />
|
||
collapsed:: true
|
||
{{renderer :plantuml_ickelqar}}
|
||
- ```plantuml
|
||
@startuml "summarization-teacher-student-example"
|
||
|
||
' ====== STYLE (optional, safe to remove) ======
|
||
hide methods
|
||
hide stereotypes
|
||
skinparam shadowing false
|
||
skinparam packageStyle rectangle
|
||
skinparam class {
|
||
BackgroundColor White
|
||
BorderColor Black
|
||
}
|
||
skinparam object {
|
||
BackgroundColor White
|
||
BorderColor Black
|
||
}
|
||
skinparam note {
|
||
BackgroundColor #F9FAFB
|
||
BorderColor #E5E7EB
|
||
}
|
||
|
||
' ====== CAPABILITY & PROTOCOL ======
|
||
object Capability_Summarization as "Capability: Summarization" {
|
||
id = C-SUMM
|
||
description = "Generate concise, accurate summaries of technical text"
|
||
supportedLanguages = "EN, IT"
|
||
}
|
||
|
||
object Proto_TeacherStudent as "InteractionProtocol: Teacher–Student v1" {
|
||
specUrl = "https://example.org/protos/teacher-student"
|
||
}
|
||
|
||
' ====== TRAINING PHASE ======
|
||
package "Training Phase" as Training {
|
||
object TeacherA as "SupervisionAgent: Teacher A" {
|
||
ID = A-T1
|
||
name = "TeacherA"
|
||
role = "Prompt Explorer"
|
||
objective = "Generate/critique prompt candidates"
|
||
}
|
||
|
||
object TeacherB as "SupervisionAgent: Teacher B" {
|
||
ID = A-T2
|
||
name = "TeacherB"
|
||
role = "Prompt Critic"
|
||
objective = "Score and refine prompts"
|
||
}
|
||
|
||
object BenchTrain as "Benchmark: README Summarization (Train)" {
|
||
ID = B-TRAIN
|
||
datasetRef = "gh-readme-train"
|
||
protocolVersion = "1.0"
|
||
}
|
||
|
||
object KPI_Rouge as "PerformanceKPI: ROUGE"
|
||
object KPI_BERT as "PerformanceKPI: BERTScore"
|
||
object KPI_Lat as "PerformanceKPI: Latency"
|
||
|
||
object M_Rouge1 as "Metric {name=ROUGE-1, unit=score}"
|
||
object M_RougeL as "Metric {name=ROUGE-L, unit=score}"
|
||
object M_BERT as "Metric {name=BERTScore, unit=score}"
|
||
object M_Lat as "Metric {name=Latency, unit=ms}"
|
||
|
||
object UsageTrain1 as "AgentUsage: TrainRun#1" {
|
||
timestamp = 2025-08-05T10:12:00Z
|
||
durationMs = 8420
|
||
cost = 0.12
|
||
}
|
||
|
||
object TeleToolTrain as "TelemetryTool: TrainerLogger {format=JSON}"
|
||
object TeleRecTrain as "TelemetryRecord {kind=TRACE}"
|
||
|
||
' Tools and memory used in training
|
||
object ToolRetriever as "Tool: DomainRetriever" {
|
||
authMethod = "API Key"
|
||
scopes = "read"
|
||
}
|
||
object STMem as "Memory {type=SHORT_TERM, scope=AGENT}"
|
||
object LTMem as "Memory {type=LONG_TERM, scope=SHARED, db=vectorDB}"
|
||
|
||
' Links within training
|
||
TeacherA --> Capability_Summarization : exposes
|
||
TeacherB --> Capability_Summarization : exposes
|
||
TeacherA --> ToolRetriever : exploits
|
||
TeacherB --> ToolRetriever : exploits
|
||
TeacherA --> STMem : has
|
||
TeacherB --> STMem : has
|
||
TeacherA --> LTMem : has
|
||
TeacherB --> LTMem : has
|
||
|
||
TeacherA --> Proto_TeacherStudent : supports
|
||
TeacherB --> Proto_TeacherStudent : supports
|
||
|
||
BenchTrain --> TeacherA : evaluates
|
||
BenchTrain --> TeacherB : evaluates
|
||
BenchTrain --> KPI_Rouge : measures
|
||
BenchTrain --> KPI_BERT : measures
|
||
BenchTrain --> KPI_Lat : measures
|
||
KPI_Rouge --> M_Rouge1 : includes
|
||
KPI_Rouge --> M_RougeL : includes
|
||
KPI_BERT --> M_BERT : includes
|
||
KPI_Lat --> M_Lat : includes
|
||
|
||
UsageTrain1 --> TeacherA : agent
|
||
TeleToolTrain --> UsageTrain1 : collects
|
||
TeleToolTrain --> TeleRecTrain : produces
|
||
|
||
note right of UsageTrain1
|
||
Prompt candidates explored:
|
||
- "Summarize in 3 sentences..."
|
||
- "Provide a factual abstract..."
|
||
Stored with scores (ROUGE/BERT).
|
||
end note
|
||
}
|
||
|
||
' ====== TESTING PHASE ======
|
||
package "Testing Phase" as Testing {
|
||
object Student1 as "SolutionAgent: Summarizer-1" {
|
||
ID = A-S1
|
||
name = "Summarizer-Abstractive"
|
||
role = "Apply best prompt"
|
||
objective = "Produce concise factual summary"
|
||
}
|
||
|
||
object Student2 as "SolutionAgent: Summarizer-2" {
|
||
ID = A-S2
|
||
name = "Summarizer-Extractive"
|
||
role = "Apply best prompt"
|
||
objective = "Produce concise factual summary"
|
||
}
|
||
|
||
' Best prompt produced in training (abstracted as context stored in memory)
|
||
object BestPrompt as "Memory {type=LONG_TERM, scope=SHARED}" {
|
||
db = "vectorDB: best_prompt_embeddings"
|
||
}
|
||
|
||
object BenchTest as "Benchmark: README Summarization (Test)" {
|
||
ID = B-TEST
|
||
datasetRef = "gh-readme-test"
|
||
protocolVersion = "1.0"
|
||
}
|
||
|
||
object KPI_Rouge_T as "PerformanceKPI: ROUGE"
|
||
object KPI_Lat_T as "PerformanceKPI: Latency"
|
||
object M_RougeL_T as "Metric {name=ROUGE-L, unit=score}"
|
||
object M_Lat_T as "Metric {name=Latency, unit=ms}"
|
||
|
||
object UsageTest1 as "AgentUsage: TestRun#S1" {
|
||
timestamp = 2025-08-12T15:44:00Z
|
||
durationMs = 5100
|
||
cost = 0.08
|
||
}
|
||
object UsageTest2 as "AgentUsage: TestRun#S2" {
|
||
timestamp = 2025-08-12T15:45:00Z
|
||
durationMs = 4300
|
||
cost = 0.06
|
||
}
|
||
|
||
object TeleToolTest as "TelemetryTool: RuntimeLogger {format=JSON}"
|
||
object TeleRecTest1 as "TelemetryRecord {kind=LOG}"
|
||
object TeleRecTest2 as "TelemetryRecord {kind=TRACE}"
|
||
|
||
' Tools used in testing
|
||
object ToolGlossary as "Tool: TechGlossary" {
|
||
authMethod = "None"
|
||
scopes = "public"
|
||
}
|
||
|
||
Student1 --> Capability_Summarization : exposes
|
||
Student2 --> Capability_Summarization : exposes
|
||
Student1 --> BestPrompt : has
|
||
Student2 --> BestPrompt : has
|
||
Student1 --> ToolGlossary : exploits
|
||
Student2 --> ToolGlossary : exploits
|
||
Student1 --> Proto_TeacherStudent : supports
|
||
Student2 --> Proto_TeacherStudent : supports
|
||
|
||
BenchTest --> Student1 : evaluates
|
||
BenchTest --> Student2 : evaluates
|
||
BenchTest --> KPI_Rouge_T : measures
|
||
BenchTest --> KPI_Lat_T : measures
|
||
KPI_Rouge_T --> M_RougeL_T : includes
|
||
KPI_Lat_T --> M_Lat_T : includes
|
||
|
||
UsageTest1 --> Student1 : agent
|
||
UsageTest2 --> Student2 : agent
|
||
TeleToolTest --> UsageTest1 : collects
|
||
TeleToolTest --> UsageTest2 : collects
|
||
TeleToolTest --> TeleRecTest1 : produces
|
||
TeleToolTest --> TeleRecTest2 : produces
|
||
|
||
note right of BestPrompt
|
||
Best prompt selected from training
|
||
(Teacher–Student protocol):
|
||
"Summarize in ≤3 sentences, preserve
|
||
factual references; avoid speculation."
|
||
end note
|
||
}
|
||
|
||
' ====== CONSENSUS & GOVERNANCE (applied after testing) ======
|
||
object Cons as "ConsensusAgent: Summarization-Consensus" {
|
||
ID = A-C1
|
||
name = "ConsensusSummarizer"
|
||
}
|
||
object Gov as "GovernancePolicy: Accuracy-First" {
|
||
rules = "Prefer factual consistency over brevity; break ties by ROUGE-L"
|
||
}
|
||
|
||
Cons --> Gov : implements
|
||
Cons --> Student1 : aggregates candidate
|
||
Cons --> Student2 : aggregates candidate
|
||
|
||
note right of Cons
|
||
Applies GovernancePolicy across
|
||
candidate summaries from Student1/2:
|
||
- Compare ROUGE-L + factuality signals
|
||
- Select winner under policy rules
|
||
end note
|
||
|
||
@enduml
|
||
|
||
``` |