logseq/pages/ICSE2026-paper2859.md

collapsed:: true
type:: [[REVIEWS]]
tags:: 
year:: 2026
venue:: [[ICSE]] 
full-title:: TraceCoder: A Trace-Driven Multi-Agent Framework for Automated Debugging of LLM-Generated Code
date-start:: [[17-09-2025]] - 15:05
date-submitted::
external-links::
status:: [[DONE]] 
deadline-submission:: 
file:: [[@TraceCoder: A Trace-Driven Multi-Agent Framework for Automated Debugging of LLM-Generated Code]] 
parent::
todoist:: https://app.todoist.com/app/task/2859-trace-coder-a-trace-driven-multi-agent-framework-for-automated-debugging-of-6cVHmvW5WjQw4vmg

	- ### [[Highlights]]
	- ### [[Comments]]
		- #.tabular
			- ### Paper summary
				- This paper presents TraceCoder, a multi-agent system to support the automated debugging of source code generated by LLMs. The approach is based on three specialized agents on Instrumentation, Analysis, and Repair. To guide the automated debugging process, the approach makes use of two introduced techniques, ie., Historical Lesson Learning (HLLM) and Rollback Mechanism (RM). They permi to avoid redundant failures. The authors claim significant improvements over state-of-the-art methods, achieving up to 34.43% relative gain in Pass@1 on benchmark datasets.
			- ### Strengths
				- + well-motivated and relevant problem
				- + interesting multi-agent architecture
			- ### Weaknesses
				- - Experimental setup lacks detail. The granularity of the initial code generation (method/class/project), the test suite’s role and coverage, and how baselines were configured/applied are insufficiently explained
				- - No agent orchestration framework is employed or discussed
				- - Instrumentation phase is critical but underspecified. In particular, semantic-preserving behavior is assumed but not formally verified or tested
			- ### Detailed comments for authors
				- Novelty: The paper is about an interesting and novel technique based on MAS to deal with a relevant problem. The proposed combination of runtime tracing, historical error learning, and modular multi-agent design is novel and well-motivated. The paper contributes original ideas such as HLLM and RM in the context of LLM-driven code repair.
				- Rigor: While the architecture is conceptually solid, key assumptions are not empirically validated, e.g., semantic preservation during instrumentation. Moreover, the experiments related to RQ1 lacks depths. Examples, setup details (class/function/project), and baseline configurations are missing.
				- Relevance: The work is highly relevant given the increasing reliance on LLMs for code generation and the frequent presence of logic bugs in such code. Debugging automation remains a practical bottleneck in applying LLMs for software engineering at scale.
				- Verifiability & transparency: Authors provide a link with the implementation of the proposed approach. However, details on the form and source of input prompts, testing strategies/coverage, and baseline configurations are missing.
				- Presentation: Writing is generally clear, though certain technical claims (e.g., semantic integrity preservation) are not critically assessed. In addition, some terms like “minimal adjustments”, and “strategic print statements” are ambiguous and require proper definitions. Moreover, authors should give details on test cases, e.g., if they play some roles during the initial generation of source code.
				- Detailed comments:
					- p.3: “Instrumentation Agent inserts diagnostic probes”
					  Does it mean that it changes the previously generated code? This is a critical point. Clarify whether the instrumentation alters control flow or computational semantics, especially in complex programs.
					- p.3: “The Instrumentation Agent employs a dedicated prompt…”
					  Is it sure that the instrumentation agent does not wrongly add statements that change the semantics of the code? You should explain how you ensure the correctness of the modified code, and whether any formal guarantees or empirical tests were performed.
					- p.3: “with strategically placed print statements...” Confirm whether print statements are the only instrumentation method.
					- p.3: “must not modify logic, comment out code, or introduce new variables” This is crucial. How do you ensure that? How is this enforced at generation time? Please elaborate.
					- p.5: “allowed to make minimal adjustments”
					  How can you define such "minimal" adjustments? Define what “minimal” means (e.g., token delta, AST edit distance) and how it's enforced across iterations.
					- p.5: “Communication follows a structured, sequential pattern…”
					  Please discuss why you didn’t use an existing MAS framework, especially when others provide orchestration, coordination, and policy enforcement.
					- p.5: “instrumentation suggestions in subsequent iterations”
					  Is this always necessary?
					- p.5: “Agents do not communicate directly...” But how can convergence be guaranteed in this loosely coupled setup?
					- p.10: Conclusion: Several high-level contributions are claimed. However, the practical usage of the system is hard to infer, especially in terms of granularity of code, role of test cases in initial generation, and setup of generation pipeline.
				- QUESTIONS:
					- Q1: How is semantic preservation ensured during the instrumentation process?
					- Q2: Why was a dedicated MAS framework not used to orchestrate agent interactions and maintain state across iterations?
					- Can you provide more details about the experimental setup for RQ1? In particular:
						- What is the granularity of the code used (function, class, full project)?
						- What is the role of test suites in the initial generation?
						- How were baselines configured and executed?
						- How was test case coverage measured and controlled?
	- ### [[REVIEWS/Notes]]
	- ### YELLOW CONCERNS
	  background-color:: yellow
		- {{query (and [[ffd400]] [[ICSE2026-paper2859]] )}}
		  collapsed:: true
	- ### ❓️Questions
		- {{query (and [[question]] [[ICSE2026-paper2859]] )[[question]]}}
		  query-table:: true
		  query-properties:: [:block]