81 lines
8.0 KiB
Markdown
81 lines
8.0 KiB
Markdown
collapsed:: true
|
||
type:: [[REVIEWS]]
|
||
tags::
|
||
year:: 2026
|
||
venue:: [[ICSE]]
|
||
full-title:: Multi-Agent LLM Collaboration for Enhancing Unit Test Generation Using Repository-Aware Knowledge Graphs
|
||
date-start:: [[09-09-2025]] - 15:31
|
||
date-submitted::
|
||
external-links::
|
||
status:: [[DONE]]
|
||
deadline-submission::
|
||
file:: [[@Multi-Agent LLM Collaboration for Enhancing Unit Test Generation Using Repository-Aware Knowledge Graphs]]
|
||
parent::
|
||
todoist:: https://app.todoist.com/app/task/1656-multi-agent-llm-collaboration-for-enhancing-unit-test-generation-using-repo-6cVHmvX8F6V23H48
|
||
|
||
- ### [[Highlights]]
|
||
- ### [[Comments]]
|
||
- #.tabular
|
||
- ### Paper summary
|
||
- The paper presents TestAgent, a multi-agent system (MAS) based on LLMs for the automated unit test generation. The proposed MAS consists of three agents (requirement planner, test generator, reviewer) that collaborate while relying on external tool APIs for executing generated tests, and a knowledge graph that includes fine-grained dependencies of the repository storing the code of the method under analysis. The authors have performed experiments on Java and Python projects, showing that TestAgent outperforms existing approaches under different dimensions.
|
||
- ### Strengths
|
||
- + Interesting and effective use of a multi-agent system for a critical software engineering task
|
||
- + Rich experiments including an ablation study showing the contribution of each agent to the unit test generation process
|
||
- + Extensive comparison with existing approaches
|
||
- ### Weaknesses
|
||
- - Some claims are overstated, especially concerning outperforming all baselines across all metrics (e.g., EvoSuite outperforms TestAgent on some dimensions).
|
||
- - The multi-agent collaboration protocol is not specified. How agents interact, share state, and resolve conflicts is not well articulated.
|
||
- - Several terminologies (e.g., “practicality”, “adaptive states”, “real-world industrial projects”) are ambiguous and need better definition
|
||
- - The mutation process is unclear; more details on fault injection are necessary to validate fairness of the evaluation.
|
||
- - The scalability of "node-per-line" approach is not discussed
|
||
- ### Detailed comments for authors
|
||
- Novelty: The usage of the repository-aware knowledge graph sounds promising, even though it is not clear how agents technical exploit it. Many existing tools can extract AST or dependency graphs out of source code. It is not clear if authors essentially rely on existing graph-construction techniques, or if they had to devise a new one because of some specific requirements that are not elaborated in the text. Nevertheless, the usage of MAS for unit test generation is novel.
|
||
- Rigor: The evaluation is extensive, covering multiple metrics and configurations (LLMs, baselines, languages). That said, certain evaluation design choices are unclear:
|
||
- How exactly mutation faults are injected? ([p.7, “Mutation score”])
|
||
- How were datasets split to mitigate LLM memorization or contamination?
|
||
- What constitutes a “real-world industrial project”? ([p.6])
|
||
- “Adaptability” in RQ3 is misleading. In particular, RQ3 does not demonstrate adaptability. What is shown is essentially that TestAgent can be executed with different models, but without clarifying what “robustness” (as mentioned in RQ3) actually means in this context. Could you specify the concrete efforts required to integrate new models? For instance, what operations or modifications were needed in the system to make different LLMs work within TestAgent? While Table 5 compares models on the same tasks and metrics, it does not provide insights into the potential challenges, limitations, or architectural adjustments necessary when switching models.
|
||
- **Relevance**: The planner–generator–reviewer design, together with tool APIs, is potentially relevant to practitioners. The focus on realistic workflows, integration with external tools, and support for multiple LLMs and languages (Java, Python) enhance applicability.
|
||
- Verifiability & transparency: A replication package is cited and online. However, key implementation details are missing in the text:
|
||
- What MAS framework is used?
|
||
- How are the tools (e.g., check_syntax, calculate_coverage) integrated and invoked autonomously?
|
||
- What steps are required to plug a new LLM into the system?
|
||
- Presentation: Overall, the paper is well written. However, I have some suggestions for improvement. In particular, the motivation for the graph should appear earlier. Statements like "substantially outperforms” (pag. 7) should be moderated especially when EvoSuite wins in different cases.
|
||
- Detailed comments:
|
||
- page 1 - "captures fine-grained dependency relations through graph edges": How are these dependencies used in practice? Clarify their concrete usage.
|
||
page 2 - "voting-based diagnostic mechanism": Mechanism unclear; how is voting implemented? Among agents or heuristics?
|
||
page 3 - "graph serves as … representation": The goal of the graph should be presented earlier.
|
||
page 3 - "node corresponds to a line of code": Scalability of this design is questionable.
|
||
page 3 - "Extensive Evaluation … industrial scenarios": What qualifies as “industrial” here? Real companies? Internal repositories?
|
||
page 4 - "dynamic analysis … adequacy checks…": How do agents access runtime info? Through what interface or protocol?
|
||
page 5 - "adaptive states": Adaptive in what sense? Not explained.
|
||
page 5 - "assesses whether current context is sufficient…": Define what “sufficient” means and how it is determined.
|
||
page 7 - "Mutation score": How is mutation done? What tools or techniques are used? Potential bias?
|
||
page 7 - "Readability and Usability … aggregated": How is the aggregation done? What weights? Based on what criteria?
|
||
page 7 - "TestAgent using GPT-4o … two alternative LLMs": Clarify whether only RQ3 used alternative LLMs.
|
||
page 7 - "substantially outperforms baselines": This should be rephrased. On some metrics, EvoSuite is better.
|
||
page 8 - "Model Adaptability": The name of the RQ is misleading. This is model portability. What changes are needed to support a new model?
|
||
page 8 - "data leakage risk": Why is this a concern? What kind of leakage?
|
||
page 8 - "information is completely unseen": Unverifiable claim unless explicit evidence is given.
|
||
page 10 - "Taxonomy of Root Causes and Bug Impacts": Could you reflect on how baselines fare in terms of these root causes?
|
||
page 10 - "private industrial dataset (UTXXX)": Need more details about this dataset. Company? Domain? Lines of code?
|
||
- Typos / Errors
|
||
page 3 - "s[": Missing space and typo.
|
||
page 5 - "n T": Should be “n. T.” or reformulated.
|
||
page 7 - "Appraoch": Should be “Approach”.
|
||
page 7 - "97.46%" in bold: Not necessary, especially when it is not the best result.
|
||
page 8 - "↑": Inconsistent with performance drop. Arrows should point down.
|
||
- ### Questions
|
||
- How exactly is the multi-agent collaboration orchestrated? Is there a predefined protocol (e.g., queue-based, blackboard architecture), and which MAS framework (if any) is used to support the coordination, communication, and tool invocation?
|
||
- What is the novelty and added value of the repository-aware knowledge graph? How does it compare to existing code representation structures used in similar works (e.g., code property graphs, AST-based models)? Can you provide a motivating example early in the paper?
|
||
- What operations are required to integrate a new LLM model into TestAgent? Does the system abstract away model-specific APIs? What challenges arise when using non-OpenAI models (e.g., context window, latency, inference mode)?
|
||
- ### [[REVIEWS/Notes]]
|
||
- ### YELLOW CONCERNS
|
||
background-color:: yellow
|
||
collapsed:: true
|
||
- {{query (and [[ffd400]] [[ICSE2026-paper1656]] )}}
|
||
collapsed:: true
|
||
- ### ❓️Questions
|
||
- {{query (and [[question]] [[ICSE2026-paper1656]] )[[question]]}}
|
||
query-table:: true
|
||
query-properties:: [:block] |