ddiruscio/logseq
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

[logseq-plugin-git:commit] 2025-06-26T07:39:28.977Z

Browse Source
This commit is contained in:
Davide Di Ruscio
2025-06-26 09:39:29 +02:00
parent 89725cd2d6
commit ee21166d7c
2 changed files with 406 additions and 5 deletions
Download Patch File Download Diff File Expand all files Collapse all files
pages/@ESEM25_paper_239.md
+401
View File
@@ -0,0 +1,401 @@
tags:: [[#zotero]]
title:: @ESEM25_paper_239
item-type:: [[document]]
original-title:: ESEM25_paper_239
links:: [Local library](zotero://select/library/items/DMQKA893), [Web library](https://www.zotero.org/users/1039502/items/DMQKA893)
- ### Attachments
- [PDF](zotero://select/library/items/P9AYY9MP) {{zotero-imported-file P9AYY9MP, "ESEM25_paper_239.pdf"}}
- ### Notes
- # Annotazioni
(26/6/2025, 09:15:21)
- “Code Retrieval” (“ESEM25_paper_239”, p. 1) #5fb236
* *
- “Commit Message Generation” (“ESEM25_paper_239”, p. 1) #5fb236
* *
- “A commit message describes the main code changes in a commit and plays a crucial role in software maintenance” (“ESEM25_paper_239”, p. 1) #5fb236
* *
- “Existing commit message generation (CMG) approaches typically frame it as a direct mapping that takes a code diff as input and produces a brief descriptive sentence as output.” (“ESEM25_paper_239”, p. 1) #a28ae5
* *
- “argue” (“ESEM25_paper_239”, p. 1) #f19837
* *
- “relying solely on the code diff is insufficient,” (“ESEM25_paper_239”, p. 1) #a28ae5
* *
- “C3Gen to enhance CMG” (“ESEM25_paper_239”, p. 1) #a28ae5
* *
- “richer contextual information at the repository scope.” (“ESEM25_paper_239”, p. 1) #5fb236
* *
- “conduct a human evaluation” (“ESEM25_paper_239”, p. 1) #e56eee
* *
- “commits serve as the fundamental unit for recording the evolution of code” (“ESEM25_paper_239”, p. 1) #5fb236
* *
- “The commit message is intended to describe the nature, motivation, and potential impact of the corresponding code changes” (“ESEM25_paper_239”, p. 1) #a28ae5
* *
- “facilitate effective collaboration within development teams” (“ESEM25_paper_239”, p. 1) #f19837
* *
- “composing informative commit messages can be both timeconsuming and labor-intensive for developers” (“ESEM25_paper_239”, p. 1) #5fb236
* *
- “As a result, in real-world development settings, the overall quality of commit messages is often suboptimal” (“ESEM25_paper_239”, p. 1) #5fb236
* *
- “CMG has emerged as a prominent topic in automated software engineering, attracting growing attention from the research community” (“ESEM25_paper_239”, p. 1) #ffd400
*Necessary a reference to support this statement. *
- “f that maps a code diff to a message, i.e., f (diff) → message.” (“ESEM25_paper_239”, p. 1) #e56eee
* *
- “the .diff file in Git only contains a limited window of code around the chang” (“ESEM25_paper_239”, p. 1) #5fb236
* *
- “enhancing the input context, such that the model can generate semantically rich commit messages that better capture the global development intent” (“ESEM25_paper_239”, p. 1) #a28ae5
* *
- “we propose C3Gen, a retrieval-augmented framework designed to enhance commit message generation by incorporating relevant contextual code snippets alongside the code diff” (“ESEM25_paper_239”, p. 1) #5fb236
* *
- “C3Gen enables models to effectively leverage additional contextual information, thereby improving the completeness of the generated commit message” (“ESEM25_paper_239”, p. 1) #e56eee
* *
- “predefined rules or templates” (“ESEM25_paper_239”, p. 2) #e56eee
* *
- “Retrieval-based approaches leverage information retrieval (IR) techniques to suggest commit messages from similar code diff.” (“ESEM25_paper_239”, p. 2) #e56eee
* *
- “These methods leverage large-scale diff-message datasets to train deep neural networks capable of learning how to generate commit messages.” (“ESEM25_paper_239”, p. 2) #5fb236
* *
- “However, none of these approaches incorporate commit-relevant code snippets to provide the model with repository-level code context, which we argue is important for capturing the global scope of changes and the underlying development intent.” (“ESEM25_paper_239”, p. 2) #5fb236
* *
- “Contextual Code-aware Commit Message Generation” (“ESEM25_paper_239”, p. 2) #5fb236
* *
- “identifying program entities t” (“ESEM25_paper_239”, p. 2) #ffd400
* *
- “incorporating richer contextual information into the generation process” (“ESEM25_paper_239”, p. 2) #ffd400
*why locating and considering program entities that invoke or instantiate the motified functions should imply having a better commit message? Commit messages should describe the changes being committed and not the code related to them. *
- “(1) constructing Code Structure Graphs (CSGs)” (“ESEM25_paper_239”, p. 2) #2ea8e5
* *
- “(2) enriching the CSG with diff-related modifications” (“ESEM25_paper_239”, p. 2) #2ea8e5
* *
- “(3) extracting contextually relevant code snippets for the later commit message generation” (“ESEM25_paper_239”, p. 2) #2ea8e5
* *
- “CSGs” (“ESEM25_paper_239”, p. 2) #ffd400
*Code Structure Graphs: is it like abstract syntax trees or something different? *
- “extract the names of classes or functions, their file paths, as well as the start and end lines of the class or function definitions.” (“ESEM25_paper_239”, p. 2) #5fb236
* *
- “These elements extracted from each sourcecode file are serialized and stored in one JSON file, and the JSON files of all sourcecode files in the codebase are referred to DefinitionIndex” (“ESEM25_paper_239”, p. 2) #5fb236
* *
- “CSG of each source code file in the codebases is constructed based on the corresponding JSON file in DefinitionIndex” (“ESEM25_paper_239”, p. 2) #5fb236
* *
- “The edges between S-node and F-node/C-node denote that the functions and/or classes are defined in this source code file.” (“ESEM25_paper_239”, p. 2) #5fb236
* *
- “augmented with information related to code changes, which point to the specified code diff.” (“ESEM25_paper_239”, p. 2) #a28ae5
* *
- “removing duplicates” (“ESEM25_paper_239”, p. 3) #ffd400
*why duplicates? The example shown in Fig. 1 should be referred during the discussion to make the sentence easy to understand. *
- “identify the source code files that are invoked by the functions or instantiate the classes defined in this source code file” (“ESEM25_paper_239”, p. 3) #5fb236
* *
- “the corresponding C-nodes or Fnodes in the CSG of this source code file will be augmented with code diff information.” (“ESEM25_paper_239”, p. 3) #ffd400
*how is this augmentation is done? What does it mean? What does the augmentation consist of? *
- “D-nodes to cover the names and paths of the invoked source code files as well as the line number referring to the locations of instantiation, invocation, etc.” (“ESEM25_paper_239”, p. 3) #5fb236
* *
- “For each entity in ModifiedEntityList, we locate its corresponding information nodes in the CSG to retrieve all the recorded invocations or instantiations. If the invocation occurs within the definition of one function or class, the entire body of the enclosing function or class will be extracted by using the elements from DefinitionIndex.” (“ESEM25_paper_239”, p. 3) #ffd400
*here we need an examples. It's difficult to grasp the details of what you are doing here *
- “If the invocation is located in a global scope or unstructured block, 25 lines before and after the invocation will be selected to as a heuristic for contextual relevance of the specified code diff.” (“ESEM25_paper_239”, p. 3) #ffd400
*Is this number defined empirically, or how? *
- “extracted segments is treated as the Relevant Code Context corresponding to the commit message, and will be used to enrich the input to the commit message generation model.” (“ESEM25_paper_239”, p. 3) #ffd400
* *
- “context,C3Gen” (“ESEM25_paper_239”, p. 3) #ff6666
* *
- “only by the immediate code changes but also by their usage, dependencies, and interactions across the broader codebase. This contributes to more semantically accurate and contextaware natural language summaries of code commits.” (“ESEM25_paper_239”, p. 3) #ffd400
*Can you give an example of what is supposed to be the right commit message compare to one that is generated by currently available CMG approaches?
Can you define the quality of commit messages? What is it expected to be a good commit message? How have you driven the work towards the achievement of such quality factors? *
- “Furthermore, persistent issues with respect to data quality and construction methodologies continue to plague existing datasets” (“ESEM25_paper_239”, p. 3) #5fb236
* *
- “Zhang et al. have revealed that specific datasets did not implement rigorous quality filtering on source repositories, resulting in commit messages that exhibit marked inconsistent clarity and adherence to conventions [3].” (“ESEM25_paper_239”, p. 3) #5fb236
* *
- “incomplete data fields” (“ESEM25_paper_239”, p. 3) #ffd400
*which one, can you give some examples? *
- “A. Data Selection and Collection” (“ESEM25_paper_239”, p. 3) #2ea8e5
* *
- “GitHub Specifically” (“ESEM25_paper_239”, p. 3) #ff6666
*missing "." *
- “et.” (“ESEM25_paper_239”, p. 3) #ff6666
* *
- “the diff and its associated retrieved code snippets as input,” (“ESEM25_paper_239”, p. 4) #5fb236
* *
- “In our experiments, four representative LLMs, i.e., GPT4o, GPT-4.1, DeepSeek V3, and DeepSeek R1, are selected to perform generation” (“ESEM25_paper_239”, p. 4) #5fb236
* *
- “We access all models via their official APIs and uniformly set the temperature parameter to 0.0, which minimizes randomness.” (“ESEM25_paper_239”, p. 4) #5fb236
* *
- “quantitatively measure the similarity between the commit messages automatically generated by LLMs and those written by human developers.” (“ESEM25_paper_239”, p. 4) #5fb236
* *
- “a) BLEU” (“ESEM25_paper_239”, p. 4) #2ea8e5
* *
- “b) ROUGE-L” (“ESEM25_paper_239”, p. 4) #2ea8e5
* *
- “c) METEOR” (“ESEM25_paper_239”, p. 4) #2ea8e5
* *
- “Clarity, Completness, and Correctness.” (“ESEM25_paper_239”, p. 4) #5fb236
* *
- “Clarity is defined to evaluate how easily the commit message can be understood, considering its wording, structure, and grammar. Completeness assesses how thoroughly the commit message captures all changes in the code diff, including important and relevant contextual details. Correctness assesses how accurately the commit message reflects the actual code changes, ensuring it avoids hallucinations or misinterpretations.” (“ESEM25_paper_239”, p. 4) #ffd400
*Who assessed such metrics? Who and how measured them? *
- “retrieve code segments relevant to the diff as additional information to enrich the input context” (“ESEM25_paper_239”, p. 4) #5fb236
* *
- “assumption that providing more contextual information will help the LLM generate semantically richer commit messages, enabling it to better articulate why the change was made and what was changed.” (“ESEM25_paper_239”, p. 4) #5fb236
* *
- “the one enhanced with our proposed C3Gen input context augmentation.” (“ESEM25_paper_239”, p. 4) #5fb236
* *
- “These mixed results suggest that different metrics emphasize distinct aspects of similarity. It is important to note that all of these metrics approximate quality based on heuristic similarity to the ground truth.” (“ESEM25_paper_239”, p. 5) #5fb236
* *
- “This highlights the limitations of relying solely on automatic metrics to assess semantic adequacy and human preference in the context of CMG tasks.” (“ESEM25_paper_239”, p. 5) #5fb236
* *
- “We identified several cases that shed light on the potential reasons behind this observation. One possible reason lies in the complexity of the code diff.” (“ESEM25_paper_239”, p. 5) #5fb236
* *
- “In many cases, the original commit message primarily describes changes to a key function, whereas the retrieval module might retrieve relevant code based on other, less central modifications.” (“ESEM25_paper_239”, p. 5) #5fb236
* *
- “This mismatch between the source of the retrieved code and the focus of the commit message can negatively impact the evaluation metrics.” (“ESEM25_paper_239”, p. 5) #ffd400
*Yes, but this can be a sign that maybe including context information is not beneficial, because commits might not have nothing to do directly with the code that is not the subject of the code, but that uses functions or classes that have been changed! *
- “This additional context may shift the generation toward a more detailed and implementationoriented style, resulting in lower similarity scores due to a mismatch in writing style rather than semantic quality.” (“ESEM25_paper_239”, p. 5) #5fb236
* *
- “due to the inherent randomness in the output of large language models, different generations of commit messages may vary slightly in word choice.” (“ESEM25_paper_239”, p. 5) #5fb236
* *
- “we argue that existing similarity-based objective metrics only provide heuristic approximations of generation quality. This calls for complementary human evaluation to gain deeper insights into the actual quality and semantic improvements of the generated commit messages.” (“ESEM25_paper_239”, p. 5) #5fb236
* *
- “Further analysis reveals that these objective metrics provide only heuristic approximations and fail to effectively capture the semantic improvements introduced by additional contextual code.” (“ESEM25_paper_239”, p. 6) #ffd400
*I think that's the point. The first thing to investigate is define what's a good commit message, whatever is the values of objective metrics. Once defined, then it is necessary to conceive metrics accordingly and then you can assess the quality of the proposed tool. At the current state, what can we conclude? what's the takeaway message? Is your approaching accurate? Since the metric scores for the considered balisenes do not change singnificantly, does it mean that existing tools are alrady good at the task of commit message generation? *
- “C3Gen-generated commit messages” (“ESEM25_paper_239”, p. 6) #5fb236
* *
- “For each commit instance, participants were provided with the code diff and a set of nine commit message candidates generated by different methods including the developer-written reference message” (“ESEM25_paper_239”, p. 6) #5fb236
* *
- “Key Finding 2” (“ESEM25_paper_239”, p. 6) #ffd400
*To be unbiased and valid it is necessary to do the same quality assessment also with the baseline. *
- “particular,cular,” (“ESEM25_paper_239”, p. 6) #ff6666
* *
- “Our experimental results demonstrate that the proposed approach significantly improves performance on the CMG task” (“ESEM25_paper_239”, p. 6) #ffd400
*This statement is not supported by the experiments that instead for some aspects represent negative results, which are ok but need to be presented as such. *
- “ApacheCM, a high-quality and context-rich dataset tailored for commit message generation research.” (“ESEM25_paper_239”, p. 6) #ffd400
*This is the main contribution of the paper. C3Gen still needs improvements by following the process that I mentioned before by starting with the definition of what's a good commit message? *
pages/ESEM25_paper_239.md
+5 -5
View File
@@ -1,17 +1,17 @@
collapsed:: true
type:: [[REVIEWS]]
tags::
year:: 2025
venue:: [[ESEM]]
full-title::
full-title:: Contextual Code Retrieval for Commit Message Generation: A Preliminary Study
date-start:: [[26-06-2025]] - 09:20
date-submitted::
external-links::
status:: [[DOING]]
status:: [[DONE]]
deadline-submission::
file::
file:: [[@ESEM25_paper_239]]
parent::
todoist::
collapsed:: true
todoist:: https://app.todoist.com/app/task/239-contextual-code-retrieval-for-commit-message-generation-a-preliminary-study-6c6VQJ226jG98g56
- ### [[Highlights]]
- ### [[Comments]]