[logseq-plugin-git:commit] 2025-06-26T07:32:55.532Z

pages/ESEM25_paper_239.md

@@ -0,0 +1,24 @@
+type:: [[REVIEWS]]
+tags:: 
+year:: 2025
+venue:: [[ESEM]] 
+full-title::
+date-start:: [[26-06-2025]] - 09:20
+date-submitted::
+external-links::
+status:: [[DOING]]
+deadline-submission:: 
+file::
+parent::
+todoist::
+collapsed:: true
+
+	- ### [[Highlights]]
+	- ### [[Comments]]
+		- SUMMARY: The paper presents C3Gen, an LLM-based approach to generate commit messages. The goal is enhancing the input given to large language models by incorporating additional context to the committed code diff. The authors argue that enriching the input with code context (i.e., code the uses the functions or classes that are the subject of the commit under analysis) leads to semantically richer and more complete commit messages. Alongside this, authors introduce ApacheCM, a dataset designed to support CMG research. The paper includes both automatic metric-based evaluations and human assessments to measure the effectiveness of the proposed approach by employing several state-of-the-art LLMs.
+		- COMMENTS: The paper is about a relevant problem, even though I identified a number of issues that undermine the validity of some drawn conclusions.
+		- One key issue concerns the absence of a concrete and operationalized definition of what constitutes a “good” commit message. Although the paper discusses dimensions such as clarity, completeness, and correctness, they are discussed only at the end of the paper and not upfront as requirements that any CMG approch should implement.
+		- The paper makes several methodological choices, such as including 25 lines of context around an invocation or removal of duplicates, without justifying these decisions empirically or referencing prior work. Several parts of the proposed method remain underspecified or ambiguous. For instance, the concept of “Code Structure Graphs” (CSGs) needs to be better distinguished from known representations like abstract syntax trees. Moreover, key operations such as the augmentation of nodes with diff information, or the retrieval of relevant contextual segments, are described at a high level but would benefit from concrete examples to aid comprehension.
+		- The authors state that the proposed approach significantly improves performance on the CMG task, but the quantitative results show limited differences across baselines, and sometimes mixed or even negative trends. Rather than framing these results as improvements, it would be more accurate to discuss them as preliminary or exploratory outcomes. Moreover, the discussion highlights that context retrieval may actually introduce a mismatch between what is relevant for the message and what is available in the surrounding code, thereby affecting the semantic quality or relevance. This point is critically important and suggests that more investigation into when and how additional context helps is necessary.
+		- The evaluation protocol also has room for improvement. While human evaluation is mentioned, it appears to have been applied only to the C3Gen-generated messages without a corresponding assessment of the baseline outputs.
+		- Finally, there are minor issues such as typos ("particular,cular," “GitHub Specifically.” missing a period, and “et.”), which should be corrected for clarity and professionalism.