ddiruscio/logseq
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
master
logseq/pages/Deliverable D2.1.md
T

22 KiB
Raw Permalink Blame History

tags:: #todoist-task, PROJECTS/MOSAICO date:: 25-08-2025 - 08:38 progress:: {{renderer :todomaster}}

  • References

  • Tasks

    • TODO Guardare i TODOs da Missione ROMA Mosaico #todoist-task #PROJECTS/MOSAICO #meeting
    • DONE Finalizzare revisione di Section 3 - 07-08-2025 id:: 6894a6d1-0613-4b32-ac49-0c3a3fd0753d
    • DONE Rivedere Section 5 - 07-08-2025 id:: 6894a6a2-bbab-4ec6-bcf4-1821588feb0f
      • DONE Da finalizzare Tabella 4 con lista di marketplace e agent repositories. id:: 68a423d6-ddac-4f6f-b162-51318a682db6 :LOGBOOK: CLOCK: [2025-09-09 Tue 15:38:36]--[2025-09-09 Tue 15:38:36] => 00:00:00 :END:
    • DONE Raffinare il metamodello in Section 6 - 08-08-2025 id:: 68960139-341c-40fa-9a35-04c20f74ad14 :LOGBOOK: CLOCK: [2025-08-19 Tue 14:43:06]--[2025-08-19 Tue 14:43:07] => 00:00:01 :END:
    • DONE Inserire in Section 7 una prima bozza dell'architettura del Repository id:: 68960139-bbd3-44c1-a39a-5f0782e5bfad :LOGBOOK: CLOCK: [2025-08-19 Tue 18:09:34]--[2025-08-26 Tue 23:06:08] => 172:56:34 :END:
      • Partire da qualche slide in WP5?
    • DONE Scrivere le conclusioni id:: 68960139-c4ea-441c-9d45-89381bddcfb9
    • DONE Lavorare su capitolo 4 a partire dal contenuto del paper EMSE id:: 68960139-1246-4fb7-a933-cc75804483b7 :LOGBOOK: CLOCK: [2025-08-19 Tue 18:09:37]--[2025-08-21 Thu 16:42:09] => 46:32:32 :END:
    • DONE Rimuovere dalla bib tutte le occorrenze univaq id:: 68ac6e66-ba4e-41e7-9637-5fe847dfc8ba :LOGBOOK: CLOCK: [2025-08-25 Mon 16:26:24]--[2025-08-25 Mon 16:26:24] => 00:00:00 :END:
    • DONE Ripassata finale prima di mandare ai revisori collapsed:: true

  • Notes

    id:: 689467c8-88b1-45b0-b58c-99fa0d0b73a8

    • Review di Antonio Garcia ( D2_1-InternalReview_version_20250827-agd220250907.pdf )
      • DONE Quick note about the English: I see a lot of "agents monitoring" which should be "agent monitoring", for example. In general, when you have "noun verb-ing", "noun" tends to be singular. id:: 68d3ea95-45ab-43bb-86c6-506a00ca5420
      • DONE Did you not consider LangGraph for Ch3? If so, why? (It's a big player, so this was somewhat surprising - did you run into issues with the license, or did you not consider it as a MAS framework?) id:: 68d3ea95-6c8a-40f9-93ff-5d930e6a7174 :LOGBOOK: CLOCK: [2025-09-24 Wed 16:01:00]--[2025-09-24 Wed 16:01:02] => 00:00:02 :END:
      • DONE Figure 3 doesn't really explain the notation used. id:: 68d3ea95-7a33-412c-a9e0-effaf3e484a0
      • DONE For FC3.3 (continual evolution), have you considered options like Letta or MemGPT for self-improving agents?
      • DONE For the METAGENTE study: in the optimisation loop, do you evaluate the new prompt based on how it does across a population of repositories, or is it only against one repository? The latter sounds like it could have a risk of over-fitting. id:: 68d3ea95-2936-4efb-8e99-d9282435fff9 :LOGBOOK: CLOCK: [2025-09-24 Wed 16:01:22]--[2025-09-24 Wed 16:01:23] => 00:00:01 :END:
      • DONE Figure 13(a) has an odd "5" before "ROUGE-1".
      • DONE In Section 4.3.4, besides doing the Wilcoxon paired test for statistical significance, I'd also suggest looking at effect sizes. My intuition is that even if the differences are statistically significant, the effect sizes will be very small. id:: 68d3ea95-00ff-47d6-b92f-0243f944f2c0
      • TODO In Figure 9, it feels like each spider chart has its own axis limits. They should all be using the same axis limits, so that the spider charts can be compared to each other.
      • DONE The report never defines what PTM stands for (used in Ch5). id:: 68d3ea95-f822-42f4-a4f6-6da2d5adfb75
      • DONE Given the talk we attended from Barahona at LLMA4SE 2025 on the various levels of "openness" of LLMs, your discussion of AI Agent List [77] sounds like this marketplace may be oversimplifying things. It may be worth mentioning the need for more nuance in that regard? id:: 68d3ea95-8858-468b-9c3a-4895f500a9f9
        • ((68bac2bb-1db8-41cb-81ed-7ef7f76e1491))
      • DONE Please take a close look at my comments on Figure 24. I think the metamodel should be expanded in a number of ways. For example, in OpenTelemetry, spans are organised into a tree - this doesn't seem to be reflected in TelemetryRecord. id:: 68d3ea95-f018-4b23-9c51-b0ee94287661 :LOGBOOK: CLOCK: [2025-09-26 Fri 18:20:20]--[2025-09-26 Fri 18:20:21] => 00:00:01 :END:
      • TODO I also think that Figure 25 needs some improvements: it should stay closer to UML object diagram notation, and it may be best to split it into static parts (about the agents themselves and their relationships) and dynamic parts (about the traces for specific executions of those agents). In fact, I would divide the trace for the separate phases, and I'd suggest expanding further the instantiation (making it more detailed and using all the types in the taxonomy).
      • TODO For the MOSAICO agent repository architecture: would it make sense to integrate some type of vector database to allow for semantic search over the descriptions of the agents? It's quite likely that we will leave it up to the LLM in the reference agent to invoke your MCP tool for finding the most relevant agents, and that will most likely require semantic search.
      • DONE Incidentally, I don't think MCP is documented via OpenAPI: MCP is JSON-RPC based rather than REST. id:: 68d3ea95-b046-49bf-9509-651f1e88b06e
      • TODO For telemetry dashboards, you mention Grafana. That's good for metrics, but can it display traces as well, or should we use LangFuse for that?
    • Da considerare anche il prompt?
    • {{renderer :mermaid_68668a24-ce43-472e-b787-a53b895f7c75, 3}} collapsed:: true 
      • classDiagram
    class Agent {
        id
        name
        version
        owner
    }

    class Capability {
        domain
        taskType
        supportedLanguages
        inputFormats
        outputFormats
    }

    class PerformanceKPI {
        accuracy
        latencyMs
        resourceConsumption
        robustnessMetrics
    }

    class Governance {
        license
        dataResidency
        GDPRCompliance
        auditTrailAvailable
    }

    class FairnessEthics {
        biasDetected
        fairnessConstraints
        explanationCapabilities
    }

    class RuntimeEnvironment {
        runtime
        requiredLibraries
        hardwareAcceleration
    }

    class Provenance {
        trainingDataSources
        modelLineage
        lastUpdated
    }

    class ContactInfo {
        maintainerEmail
        documentationUrl
    }

    class AgentRepository {
        <<service>>
    }

    Agent --> Capability : has
    Agent --> PerformanceKPI : provides
    Agent --> Governance
    Agent --> FairnessEthics
    Agent --> RuntimeEnvironment
    Agent --> Provenance
    Agent --> ContactInfo
    AgentRepository --> Agent : manages
      • Agents should be also marked if they keep information or not for IP purposes.
  • {{renderer :plantuml_wqrztfdo}} - ```plantuml @startuml ' Enumerations enum FeedbackType { EXPLICIT IMPLICIT } 
    enum MemoryType {
  SHORT_TERM
  LONG_TERM
}

enum TelemetryOutput {
  LOG
  TRACE
  BAGGAGE
}

' Core entities
class Agent {
  ID: int
  name: string
  modelCard: string
  messageContent: string
  hyperparameter: string
  dependencies: Agent
}

class SolutionAgent {
  ID: int
  intentions: string
  desires: string
  backStory: string
  promptType: string
  role: string
}

class CollaborationAgent {
  ID: int
  collaborationPattern: Pattern
}

class SupervisionAgent {
  ID: int
}

class ConsensusAgent {
  ID: int
}

class Task {
  id: int
  description: string
  input: string
  output: string
}

class Tool {
  ID: int
  name: string
  API_key: string
}

class Memory {
  ID: int
  type: MemoryType
  db: string
  agent: Agent
}

' Governance and usage
class Provider {
  name: string
}

class Usage {
  ID: int
  name: string
}

class Remote {
  ID: int
  url: string
  provider: Provider
}

class Local {
  ID: int
  hardwareRequirement: string
}

class A2AProtocol {
  ID: int
}

' Evaluation and monitoring
class Benchmark {
  ID: int
  metadata: string
  features: string
}

class Metric {
  ID: int
  name: string
  threshold: double
}

class TelemetryData {
  ID: int
  tool: TelemetrySource
  outputFormat: TelemetryOutput
}

class TelemetrySource {
  ID: int
}

class HumanFeedback {
  ID: int
  userID: int
  feedback: FeedbackType
}

' Relationships
Agent <|-- SolutionAgent
Agent <|-- CollaborationAgent
Agent <|-- SupervisionAgent
Agent <|-- ConsensusAgent

Agent --> "1..*" Task : accomplishes
Agent --> "0..*" Tool : exploits
Agent --> "1..*" Memory : has
Agent --> "0..*" Benchmark : evaluated by
Agent --> "1" Provider : provided by
Agent --> "0..*" Usage : uses
Agent --> "0..*" A2AProtocol : complies with

Benchmark --> "1..*" Metric : includes
TelemetryData --> Benchmark : collected on
TelemetryData --> HumanFeedback : stores
HumanFeedback --> Agent : evaluates
Usage <|-- Remote
Usage <|-- Local

@enduml


```
  • collapsed:: true {{renderer :plantuml_ickelqar}} - ```plantuml @startuml "summarization-teacher-student-example" 
    ' ====== STYLE (optional, safe to remove) ======
hide methods
hide stereotypes
skinparam shadowing false
skinparam packageStyle rectangle
skinparam class {
  BackgroundColor White
  BorderColor Black
}
skinparam object {
  BackgroundColor White
  BorderColor Black
}
skinparam note {
  BackgroundColor #F9FAFB
  BorderColor #E5E7EB
}

' ====== CAPABILITY & PROTOCOL ======
object Capability_Summarization as "Capability: Summarization" {
  id = C-SUMM
  description = "Generate concise, accurate summaries of technical text"
  supportedLanguages = "EN, IT"
}

object Proto_TeacherStudent as "InteractionProtocol: Teacher–Student v1" {
  specUrl = "https://example.org/protos/teacher-student"
}

' ====== TRAINING PHASE ======
package "Training Phase" as Training {
  object TeacherA as "SupervisionAgent: Teacher A" {
    ID = A-T1
    name = "TeacherA"
    role = "Prompt Explorer"
    objective = "Generate/critique prompt candidates"
  }

  object TeacherB as "SupervisionAgent: Teacher B" {
    ID = A-T2
    name = "TeacherB"
    role = "Prompt Critic"
    objective = "Score and refine prompts"
  }

  object BenchTrain as "Benchmark: README Summarization (Train)" {
    ID = B-TRAIN
    datasetRef = "gh-readme-train"
    protocolVersion = "1.0"
  }

  object KPI_Rouge as "PerformanceKPI: ROUGE"
  object KPI_BERT as "PerformanceKPI: BERTScore"
  object KPI_Lat as "PerformanceKPI: Latency"

  object M_Rouge1 as "Metric {name=ROUGE-1, unit=score}"
  object M_RougeL as "Metric {name=ROUGE-L, unit=score}"
  object M_BERT as "Metric {name=BERTScore, unit=score}"
  object M_Lat as "Metric {name=Latency, unit=ms}"

  object UsageTrain1 as "AgentUsage: TrainRun#1" {
    timestamp = 2025-08-05T10:12:00Z
    durationMs = 8420
    cost = 0.12
  }

  object TeleToolTrain as "TelemetryTool: TrainerLogger {format=JSON}"
  object TeleRecTrain as "TelemetryRecord {kind=TRACE}"

  ' Tools and memory used in training
  object ToolRetriever as "Tool: DomainRetriever" {
    authMethod = "API Key"
    scopes = "read"
  }
  object STMem as "Memory {type=SHORT_TERM, scope=AGENT}"
  object LTMem as "Memory {type=LONG_TERM, scope=SHARED, db=vectorDB}"

  ' Links within training
  TeacherA --> Capability_Summarization : exposes
  TeacherB --> Capability_Summarization : exposes
  TeacherA --> ToolRetriever : exploits
  TeacherB --> ToolRetriever : exploits
  TeacherA --> STMem : has
  TeacherB --> STMem : has
  TeacherA --> LTMem : has
  TeacherB --> LTMem : has

  TeacherA --> Proto_TeacherStudent : supports
  TeacherB --> Proto_TeacherStudent : supports

  BenchTrain --> TeacherA : evaluates
  BenchTrain --> TeacherB : evaluates
  BenchTrain --> KPI_Rouge : measures
  BenchTrain --> KPI_BERT : measures
  BenchTrain --> KPI_Lat : measures
  KPI_Rouge --> M_Rouge1 : includes
  KPI_Rouge --> M_RougeL : includes
  KPI_BERT --> M_BERT : includes
  KPI_Lat --> M_Lat : includes

  UsageTrain1 --> TeacherA : agent
  TeleToolTrain --> UsageTrain1 : collects
  TeleToolTrain --> TeleRecTrain : produces

  note right of UsageTrain1
    Prompt candidates explored:
    - "Summarize in 3 sentences..."
    - "Provide a factual abstract..."
    Stored with scores (ROUGE/BERT).
  end note
}

' ====== TESTING PHASE ======
package "Testing Phase" as Testing {
  object Student1 as "SolutionAgent: Summarizer-1" {
    ID = A-S1
    name = "Summarizer-Abstractive"
    role = "Apply best prompt"
    objective = "Produce concise factual summary"
  }

  object Student2 as "SolutionAgent: Summarizer-2" {
    ID = A-S2
    name = "Summarizer-Extractive"
    role = "Apply best prompt"
    objective = "Produce concise factual summary"
  }

  ' Best prompt produced in training (abstracted as context stored in memory)
  object BestPrompt as "Memory {type=LONG_TERM, scope=SHARED}" {
    db = "vectorDB: best_prompt_embeddings"
  }

  object BenchTest as "Benchmark: README Summarization (Test)" {
    ID = B-TEST
    datasetRef = "gh-readme-test"
    protocolVersion = "1.0"
  }

  object KPI_Rouge_T as "PerformanceKPI: ROUGE"
  object KPI_Lat_T as "PerformanceKPI: Latency"
  object M_RougeL_T as "Metric {name=ROUGE-L, unit=score}"
  object M_Lat_T as "Metric {name=Latency, unit=ms}"

  object UsageTest1 as "AgentUsage: TestRun#S1" {
    timestamp = 2025-08-12T15:44:00Z
    durationMs = 5100
    cost = 0.08
  }
  object UsageTest2 as "AgentUsage: TestRun#S2" {
    timestamp = 2025-08-12T15:45:00Z
    durationMs = 4300
    cost = 0.06
  }

  object TeleToolTest as "TelemetryTool: RuntimeLogger {format=JSON}"
  object TeleRecTest1 as "TelemetryRecord {kind=LOG}"
  object TeleRecTest2 as "TelemetryRecord {kind=TRACE}"

  ' Tools used in testing
  object ToolGlossary as "Tool: TechGlossary" {
    authMethod = "None"
    scopes = "public"
  }

  Student1 --> Capability_Summarization : exposes
  Student2 --> Capability_Summarization : exposes
  Student1 --> BestPrompt : has
  Student2 --> BestPrompt : has
  Student1 --> ToolGlossary : exploits
  Student2 --> ToolGlossary : exploits
  Student1 --> Proto_TeacherStudent : supports
  Student2 --> Proto_TeacherStudent : supports

  BenchTest --> Student1 : evaluates
  BenchTest --> Student2 : evaluates
  BenchTest --> KPI_Rouge_T : measures
  BenchTest --> KPI_Lat_T : measures
  KPI_Rouge_T --> M_RougeL_T : includes
  KPI_Lat_T --> M_Lat_T : includes

  UsageTest1 --> Student1 : agent
  UsageTest2 --> Student2 : agent
  TeleToolTest --> UsageTest1 : collects
  TeleToolTest --> UsageTest2 : collects
  TeleToolTest --> TeleRecTest1 : produces
  TeleToolTest --> TeleRecTest2 : produces

  note right of BestPrompt
    Best prompt selected from training
    (Teacher–Student protocol):
    "Summarize in ≤3 sentences, preserve
     factual references; avoid speculation."
  end note
}

' ====== CONSENSUS & GOVERNANCE (applied after testing) ======
object Cons as "ConsensusAgent: Summarization-Consensus" {
  ID = A-C1
  name = "ConsensusSummarizer"
}
object Gov as "GovernancePolicy: Accuracy-First" {
  rules = "Prefer factual consistency over brevity; break ties by ROUGE-L"
}

Cons --> Gov : implements
Cons --> Student1 : aggregates candidate
Cons --> Student2 : aggregates candidate

note right of Cons
  Applies GovernancePolicy across
  candidate summaries from Student1/2:
  - Compare ROUGE-L + factuality signals
  - Select winner under policy rules
end note

@enduml

```
Reference in New Issue View Git Blame Copy Permalink