logseq/pages/@SRC-Retrieval%3A Semantic-Based GitHub Repository and Code Retrieval.md

tags:: [[#zotero]]
title:: @SRC-Retrieval: Semantic-Based GitHub Repository and Code Retrieval
item-type:: [[document]]
original-title:: SRC-Retrieval: Semantic-Based GitHub Repository and Code Retrieval
links:: [Local library](zotero://select/library/items/NH39F2BD), [Web library](https://www.zotero.org/users/1039502/items/NH39F2BD)

- ### Attachments
	- [PDF](zotero://select/library/items/MWXK9CXR) {{zotero-imported-file MWXK9CXR, "icse2026-paper3125.pdf"}}
- ### Notes
	- I'm reviewing a research paper and I took the following notes:
	  
	  # Annotations 
	   (16/09/2025, 16:05:09)
	  
	  - “Semantic-Based” (“SRC-Retrieval: Semantic-Based GitHub Repository and Code Retrieval”, p. 1) #5fb236
	  
	  - “existing search methods on GitHub and general search engines like Bing often fail to capture the semantic intent behind developers' queries, so the retrieved repositories may not satisfy developers’ requirements.” (“SRC-Retrieval: Semantic-Based GitHub Repository and Code Retrieval”, p. 1) #a28ae5
	  
	  - “Moreover, the search results of current methods are at the repository level rather than at the level of individual code ;iles, making it challenging for beginners to locate speci;ic code implementations.” (“SRC-Retrieval: Semantic-Based GitHub Repository and Code Retrieval”, p. 1) #ffd400
	  *Do you actually need to do so via existing search engines like Bing when we have the availability of advanced technologies like ChatGPT and Copilot?*
	  
	  - “high-level functional requirements and concrete code implementations.” (“SRC-Retrieval: Semantic-Based GitHub Repository and Code Retrieval”, p. 1) #a28ae5
	  
	  - “A transformer-based retrieval model leverages text embeddings to recommend not only relevant repositories but also speci;ic code ;iles aligned with the user’s intended functionalities” (“SRC-Retrieval: Semantic-Based GitHub Repository and Code Retrieval”, p. 1) #5fb236
	  
	  - “A common use case involves searching for repositories that implement specific functionalities, such as “user authentication” or “data visualization”, to serve as references for their own software projects.” (“SRC-Retrieval: Semantic-Based GitHub Repository and Code Retrieval”, p. 1) #5fb236
	  
	  - “GitHub’s search mainly relies on keyword matching, which often fails to capture the semantic intent behind developers’ queries.” (“SRC-Retrieval: Semantic-Based GitHub Repository and Code Retrieval”, p. 1) #5fb236
	  
	  - “While general-purpose search engines offer alternative solutions [1], they also typically lack a nuanced understanding of the semantic relationship between queries and repositories, often returning broad and imprecise results.” (“SRC-Retrieval: Semantic-Based GitHub Repository and Code Retrieval”, p. 1) #e56eee
	  
	  - “As a result, even when developers find a relevant repository, locating the exact code snippets they need can be difficult, especially for those who are new to programming.” (“SRC-Retrieval: Semantic-Based GitHub Repository and Code Retrieval”, p. 1) #ffd400
	  
	  - “we propose SRC-Retrieval, a GitHub repository and code retrieval method that semantically aligns developers’ functional requirements with relevant repositories and specific code files.” (“SRC-Retrieval: Semantic-Based GitHub Repository and Code Retrieval”, p. 1) #ffd400
	  *Intersting to see later, what do the author mean with semantic alignment and the corresponding granularity.*
	  
	  - “Text embeddings are leveraged to represent both user queries and repositories, as they effectively capture contextual and semantic information and have been widely applied in various tasks such as semantic search, question answering, and document classification” (“SRC-Retrieval: Semantic-Based GitHub Repository and Code Retrieval”, p. 1) #a28ae5
	  
	  - “code search is an important research topic in software engineering.” (“SRC-Retrieval: Semantic-Based GitHub Repository and Code Retrieval”, p. 2) #5fb236
	  
	  - “Large Language Models (LLMs) may generate code containing hallucinations, which can mislead beginners lacking the expertise to assess correctness. Based on these considerations and supported by our experimental results, we adopt transformer-based encoders for embedding generation.” (“SRC-Retrieval: Semantic-Based GitHub Repository and Code Retrieval”, p. 2) #a28ae5
	  *This is a good point!*
	  
	  - “Unlike existing tools that primarily recommend repositories, our approach identifies specific code files within repositories that directly implement the queried functionality, providing beginners with coding references.” (“SRC-Retrieval: Semantic-Based GitHub Repository and Code Retrieval”, p. 2) #a28ae5
	  
	  - “We propose a semantic retrieval method that aligns user queries with repository functionalities.” (“SRC-Retrieval: Semantic-Based GitHub Repository and Code Retrieval”, p. 2) #ffd400
	  *"repository functionality" is a good point, even though tricky to mange and sustain. I guess it is based on README information, isn't it?
	  
	  Let's go ahead with the reading and let's see.*
	  
	  - “BERT-based model to link functional requirements to specific code files” (“SRC-Retrieval: Semantic-Based GitHub Repository and Code Retrieval”, p. 2) #ffd400
	  *How functional requirements are defined? Are they automatically created from README files? What else? Who create such links?*
	  
	  - “To the best of our knowledge, this is the first approach to semantically match user requirement to both repositories and source code files, providing practical references for software development” (“SRC-Retrieval: Semantic-Based GitHub Repository and Code Retrieval”, p. 2) #ffd400
	  *I think so, even though the created of the training data is crucial here! Not clear yet how the dataset has been created.*
	  
	  - “Experimental results of 3 tasks demonstrate that our method consistently outperforms the baseline methods in metrics such as precision, recall, MRR and repository and code success rate.” (“SRC-Retrieval: Semantic-Based GitHub Repository and Code Retrieval”, p. 2) #5fb236
	  
	  - “open-source software repository retrieval and recommendation” (“SRC-Retrieval: Semantic-Based GitHub Repository and Code Retrieval”, p. 2) #2ea8e5
	  
	  - “API recommendation” (“SRC-Retrieval: Semantic-Based GitHub Repository and Code Retrieval”, p. 2) #2ea8e5
	  
	  - “requirementto-code traceability link recovery” (“SRC-Retrieval: Semantic-Based GitHub Repository and Code Retrieval”, p. 2) #2ea8e5
	  
	  - “their” (“SRC-Retrieval: Semantic-Based GitHub Repository and Code Retrieval”, p. 2) #ff6666
	  *the*
	  
	  - “While API recommendation provides more fine-grained assistance than repository-level retrieval or recommendation, developers searching for references on implementing specific functionalities often require concrete code examples to understand how these functionalities are implemented in practice.” (“SRC-Retrieval: Semantic-Based GitHub Repository and Code Retrieval”, p. 3) #ffd400
	  *These two options are not exclusive. API recommendation is still valid even when developers have found a reference GitHub repository.*
	  
	  - “This paper solves this problem by proposing a method for recommending specific code files that directly demonstrate the implementations of desired functionalities.” (“SRC-Retrieval: Semantic-Based GitHub Repository and Code Retrieval”, p. 3) #ffd400
	  *See my previous comment. I would smooth this point.*
	  
	  - “a technique that automatically refines coarsegrained requirement-to-class traces into method-level links.” (“SRC-Retrieval: Semantic-Based GitHub Repository and Code Retrieval”, p. 3) #5fb236
	  
	  - “On GitHub, issues often contain valuable descriptions of desired functionalities, effectively serving as high-level requirements that developers aim to implement. As such, they are well-suited for application of traceability link recovery techniques.” (“SRC-Retrieval: Semantic-Based GitHub Repository and Code Retrieval”, p. 3) #5fb236
	  
	  - “While existing methods like BTLink and PromptLink focus on linking issues to commits, they fall short of establishing direct connections between issues and the corresponding code files” (“SRC-Retrieval: Semantic-Based GitHub Repository and Code Retrieval”, p. 3) #5fb236
	  
	  - “In this paper, we propose a novel model that extends existing methods by identifying issues that represent specific functionalities and mapping them directly to relevant code files.” (“SRC-Retrieval: Semantic-Based GitHub Repository and Code Retrieval”, p. 3) #a28ae5
	  
	  - “Then, functionalities are extracted from the README files and issues of the repositories” (“SRC-Retrieval: Semantic-Based GitHub Repository and Code Retrieval”, p. 3) #5fb236
	  
	  - “Figure 1: The architecture of the GitHub repository and code retrieval method” (“SRC-Retrieval: Semantic-Based GitHub Repository and Code Retrieval”, p. 3) #ffd400
	  *The link creation between Issue and code is more intuitive than that between readme and code. The latter seems to be more coarse grained. Not enought details are available to link specific parts of the code base with a functionality description that should be in the readme file.*
	  
	  - “collect all other repositories starred by that user.” (“SRC-Retrieval: Semantic-Based GitHub Repository and Code Retrieval”, p. 4) #ffd400
	  
	  - “their starred repositories are likely related to software development as well.” (“SRC-Retrieval: Semantic-Based GitHub Repository and Code Retrieval”, p. 4) #ffd400
	  *what kind of relation do you expect? What do you want to do with such a relation?*
	  
	  - “we apply a filtering criterion that remains only repositories with at least 150 stars. This results in a refined dataset comprising over 180,000 repositories.” (“SRC-Retrieval: Semantic-Based GitHub Repository and Code Retrieval”, p. 4) #5fb236
	  
	  - “we identify and select 40,133 repositories tagged with the top 50 most frequently used topics, ensuring that our dataset reflects both emerging trends and well-established areas within the software development ecosystem.” (“SRC-Retrieval: Semantic-Based GitHub Repository and Code Retrieval”, p. 4) #5fb236
	  
	  - “In this section, we present our approach to extracting functionalities from GitHub repositories.” (“SRC-Retrieval: Semantic-Based GitHub Repository and Code Retrieval”, p. 4) #5fb236
	  
	  - “identify feature request entries,” (“SRC-Retrieval: Semantic-Based GitHub Repository and Code Retrieval”, p. 4) #ffd400
	  *So, you considered only closed issues. But in some cases it can be that the issues is closed not because the functionality has been actually implemented, but because it is no longer interesting to implement, or something similar.*
	  
	  - “3.2.1 Functionalities in README Files.” (“SRC-Retrieval: Semantic-Based GitHub Repository and Code Retrieval”, p. 4) #2ea8e5
	  
	  - “repository descriptions inherently reflect a repository’s functionalities.” (“SRC-Retrieval: Semantic-Based GitHub Repository and Code Retrieval”, p. 4) #a28ae5
	  
	  - “Therefore, we use repository descriptions to identify functionalities in README files.” (“SRC-Retrieval: Semantic-Based GitHub Repository and Code Retrieval”, p. 4) #ffd400
	  *This step is not clear at all!*
	  
	  - “For negative samples, we randomly select sentences from README files that do not match any predefined requirement patterns.” (“SRC-Retrieval: Semantic-Based GitHub Repository and Code Retrieval”, p. 4) #ffd400
	  *This is also unclear!*
	  
	  - “s[” (“SRC-Retrieval: Semantic-Based GitHub Repository and Code Retrieval”, p. 5) #ff6666
	  *missing space*
	  
	  - “improve the accuracy of functionality extraction from README files, by capturing the semantic relationships between the text and the repository’s intended functionalities.” (“SRC-Retrieval: Semantic-Based GitHub Repository and Code Retrieval”, p. 5) #ffd400
	  *Since readme files and repository descriptions play a key role in the model training phase, does it make sense to impose come constraints on the length or structure of README files and/or repository descriptions?*
	  
	  - “After training, the classifier is applied to the candidate sentences, resulting in the extraction of 570,978 functionalities from README files.” (“SRC-Retrieval: Semantic-Based GitHub Repository and Code Retrieval”, p. 5) #ffd400
	  *To make things clear, it would be great to have a representative example of README file and corresponding text with the retrieved functionalities. Currently such a mapping is vague.*
	  
	  - “3.2.2 Functionalities in Issues.” (“SRC-Retrieval: Semantic-Based GitHub Repository and Code Retrieval”, p. 5) #2ea8e5
	  
	  - “2. A feature request issue must be marked as “closed” with the status “completed”. On GitHub, when closing an issue, collaborators can categorize it as “completed”, “not planned” or “duplicate”, with “close as completed” being the default option. In contrast, an open issue indicates that the feature hasn’t yet been implemented in the repository.” (“SRC-Retrieval: Semantic-Based GitHub Repository and Code Retrieval”, p. 5) #5fb236
	  
	  - “In some cases, although an issue may not be explicitly linked to a commit, contributors may confirm that the feature has been accepted by the issue’s comments.” (“SRC-Retrieval: Semantic-Based GitHub Repository and Code Retrieval”, p. 5) #ffd400
	  *how frequent is this case. How many of this case have you managed during the data preparation phase?*
	  
	  - “To automate this verification process, we train a classifier to assess whether a feature request issue has been accepted, based on the comments provided by the contributors. This classifier helps ensure that only genuinely accepted and implemented feature requests are considered as functionalities of the repository.” (“SRC-Retrieval: Semantic-Based GitHub Repository and Code Retrieval”, p. 5) #ffd400
	  *does it introduce some bias in the data creation phase? I'm not sure about this automation step.*
	  
	  - “We fine-tune a BERT-based classifier on this dataset, enabling the model to learn to distinguish between accepted and rejected feature requests.” (“SRC-Retrieval: Semantic-Based GitHub Repository and Code Retrieval”, p. 6) #5fb236
	  
	  - “After training, the classifier is applied to feature request issues that are marked as “closed” with the status “completed”, identifying 350,413 issues as accepted. These accepted issues, along with the 22,270 issues that are directly linked to milestones or commits, are considered as valid functionalities in the repositories. To represent these functionalities, we use the titles of the issues, as they concisely express the main functionality described in the issue.” (“SRC-Retrieval: Semantic-Based GitHub Repository and Code Retrieval”, p. 6) #ffd400
	  *The creation of the dataset by means of the classifier, can introduce some noise. Have you investigated the effects of not correctly classified issues to the creation phase of the dataset?*
	  
	  - “specific code files that are likely to implement those requirements.” (“SRC-Retrieval: Semantic-Based GitHub Repository and Code Retrieval”, p. 6) #a28ae5
	  
	  - “establish direct links between the extracted functionalities and the concrete code files within each repository, which enables developers to not only discover repositories that align with their requirements but also quickly pinpoint the precise code files that serve as useful references for their own implementations” (“SRC-Retrieval: Semantic-Based GitHub Repository and Code Retrieval”, p. 6) #5fb236
	  
	  - “GitHub issues are more closely related to implementation than README files, as commits that implement features often reference the original issues requesting those features.” (“SRC-Retrieval: Semantic-Based GitHub Repository and Code Retrieval”, p. 6) #ffd400
	  *It is necessary to add such a clarification early in the paper. In the previous section, README files and issues are considered similar for the sake of functionality characterization of repositories.*
	  
	  - “identify connections between feature request issues and the specific code files that implement the requested functionalities.” (“SRC-Retrieval: Semantic-Based GitHub Repository and Code Retrieval”, p. 6) #5fb236
	  
	  - “issuecommit pairs” (“SRC-Retrieval: Semantic-Based GitHub Repository and Code Retrieval”, p. 6) #5fb236
	  
	  - “To construct negative samples, we randomly select commits that do not reference any issue in the positive set.” (“SRC-Retrieval: Semantic-Based GitHub Repository and Code Retrieval”, p. 6) #5fb236
	  
	  - “ce 8” (“SRC-Retrieval: Semantic-Based GitHub Repository and Code Retrieval”, p. 6) #ff6666
	  
	  - “the classifier is applied to identify issuecommit links within repositories.” (“SRC-Retrieval: Semantic-Based GitHub Repository and Code Retrieval”, p. 6) #5fb236
	  
	  - “However, a single commit may address multiple features, making it inappropriate to assume that all modified code files are relevant to the linked issue.” (“SRC-Retrieval: Semantic-Based GitHub Repository and Code Retrieval”, p. 6) #e56eee
	  
	  - “design a secondary classifier to determine whether a specific code file implements the feature described in a given issue.” (“SRC-Retrieval: Semantic-Based GitHub Repository and Code Retrieval”, p. 6) #ffd400
	  *Another classifier? Too many and their impact/contribution to the whole process is not discussed at this stage, See my previous comments on the effect of the error propagation through the whole process.*
	  
	  - ““You are a data tagger who needs to read a GitHub issue describing a specific feature and a code file to decide whether the code is related to the issue, meaning the code implements (or partially implements) the feature. Use 1 for related and 0 for not” (“SRC-Retrieval: Semantic-Based GitHub Repository and Code Retrieval”, p. 7) #a28ae5
	  
	  - “e resulting labeled dataset is then used to finetune a BERT-based classifier, further improving the model's ability to accurately link issues to relevant code files.” (“SRC-Retrieval: Semantic-Based GitHub Repository and Code Retrieval”, p. 7) #ffd400
	  *I would improve figure 4 to make more explicit the different steps including a graphical representation of the process constructing the different datasets.*
	  
	  - “source code features of a code file as the concatenation of method names, class names, variable names, and comments found within the code.” (“SRC-Retrieval: Semantic-Based GitHub Repository and Code Retrieval”, p. 7) #5fb236
	  
	  - “For each issue-code pair in the dataset, the issue text and the extracted source code features are concatenated, and the combined sequence is tokenized into individual tokens.” (“SRC-Retrieval: Semantic-Based GitHub Repository and Code Retrieval”, p. 7) #5fb236
	  
	  - “To optimize model performance, we use Cross Entropy as the loss function to train the model, with the Adam optimizer employed for optimization. The learning rate is set to 1 × 10!" to ensure stable and effective convergence. The model is based on the bert-base-uncased varian” (“SRC-Retrieval: Semantic-Based GitHub Repository and Code Retrieval”, p. 7) #ffd400
	  *This has been repeated four times. It's better to say once and refer the different steps where such an optimization has been operated.*
	  
	  - “After fine-tuning, this model is applied to all issue-code pairs to identify the links between issues and the corresponding code files that implement them. This approach allows us to efficiently establish the relevant connections between issues and the code that addresses them, helping developers quickly locate the most relevant code files for specific functionalities.” (“SRC-Retrieval: Semantic-Based GitHub Repository and Code Retrieval”, p. 7) #ffd400
	  *Repeated.*
	  
	  - “We randomly select 1,000 README functionalities from different repositories and pair them with relevant code files from the same repository, resulting in a dataset of 48,107 functionality-code pairs.” (“SRC-Retrieval: Semantic-Based GitHub Repository and Code Retrieval”, p. 7) #ffd400
	  *The granularity of README files vs functionality vs code is questionable. Not convincing at this stage.*
	  
	  - “Use 1 for related and 0 for not.” The model temperature is set to zero.” (“SRC-Retrieval: Semantic-Based GitHub Repository and Code Retrieval”, p. 7) #5fb236
	  
	  - “dataset generated by DeepSeek is then used to fine-tune a BERT model, enhancing its ability to accurately capture the relationships between high-level README functionalities and the code that implements them.” (“SRC-Retrieval: Semantic-Based GitHub Repository and Code Retrieval”, p. 7) #ffd400
	  *This is not clear. A README file can be linked to several lines of code and potentially to the entire repository. This kind of link is not clear to me.*
	  
	  - “For every README functionalitycode file pair in the dataset, the README functionality and the extracted code features are concatenated and tokenized into individual tokens. Following the BERT architecture, special tokens [CLS] and [SEP] are added at the beginning and end of the token sequence, respectively. The tokenized sequence is passed through the encoding layer, and the resulting embeddings are fed into a fully connected layer followed by a sigmoid layer for classification. Cross Entropy is used as the loss function during training, and the Adam optimizer is applied to optimize the model. The learning rate is set to 1 × 10!" to ensure stable and effective convergence. The model is based on the bert-base-uncased variant from Hugging Face and is implemented using PyTorch.” (“SRC-Retrieval: Semantic-Based GitHub Repository and Code Retrieval”, p. 7) #ffd400
	  *Again, this is repeated many times.*
	  
	  - “To enable efficient and accurate search, we use the Software Entity Recognition (SER) model proposed by Nguyen et al. [39] to identify software entities in the extracted functionalities. Additionally, we use Sentence-BERT [40] to generate embeddings for all the functionalities.” (“SRC-Retrieval: Semantic-Based GitHub Repository and Code Retrieval”, p. 7) #5fb236
	  
	  - “We then match the software entities against those in the functionality dataset and calculate the cosine similarity between the query embedding and all functionality embeddings in the database” (“SRC-Retrieval: Semantic-Based GitHub Repository and Code Retrieval”, p. 7) #5fb236
	  
	  - “This hierarchical fusion approach ensures that the top 20 most relevant functionalities are ranked and returned.” (“SRC-Retrieval: Semantic-Based GitHub Repository and Code Retrieval”, p. 8) #5fb236
	  
	  - “specifying their software requirements, and the tool will retrieve the top 20 related repositories, along with the corresponding code files that implement the requested functionalities for reference” (“SRC-Retrieval: Semantic-Based GitHub Repository and Code Retrieval”, p. 8) #a28ae5
	  
	  - “This paper presents a GitHub repository and code retrieval method that extracts functionalities from GitHub repositories, establishes links between these functionalities and the corresponding code files, and retrieves repositories and code files based on user queries” (“SRC-Retrieval: Semantic-Based GitHub Repository and Code Retrieval”, p. 8) #ffd400
	  *This paragraph is not needed at this stage. You have already said what the paper is about. In any case, you are not presenting a GitHub repository in the end, isn't it?*
	  
	  - “RQ1: How effective is our method in extracting functionalities from README files and issues to support the functionality extraction component of the method?” (“SRC-Retrieval: Semantic-Based GitHub Repository and Code Retrieval”, p. 8) #ffd400
	  *I'm curious to see the contribution of README files in the overall process. An ablation study would help here. I'm not convinced that README files contribute in a significant manner.*
	  
	  - “two baseline models for comparison.” (“SRC-Retrieval: Semantic-Based GitHub Repository and Code Retrieval”, p. 8) #ffd400
	  *The considered baselins are not appropriate in my opinion. Authors should compare the proposed approach with technologies like Copilot or alike.
	  
	  Moreover, it's not clear how the two baselines have been applied and compared with the proposed approach. 
	  
	  How have you validated that the lines of code that have been retrieved starting for user requests are correctly retrieved and meet the user requirements?*
	  
	  - “Keyword-Driven Hierarchical Classification of GitHub Repositories,” (“SRC-Retrieval: Semantic-Based GitHub Repository and Code Retrieval”, p. 11) #5fb236
	  
	  - “[3] Y. Zhang, D. Lo, P. S. Kochhar, X. Xia, Q. Li, and J. Sun, “Detecting similar repositories on GitHub,” in 2017 IEEE 24th International Conference on Software Analysis, Evolution and Reengineering (SANER), Feb. 2017, pp. 13–23. doi: 10.1109/SANER.2017.7884605.” (“SRC-Retrieval: Semantic-Based GitHub Repository and Code Retrieval”, p. 11) #5fb236
	  
	  - “Approaching code search for python as a translation retrieval problem with dual encoders,” (“SRC-Retrieval: Semantic-Based GitHub Repository and Code Retrieval”, p. 11) #ffd400
	  *This might be a related work to compare with, isn't it?*
	  
	  - “Information Retrieval Approaches Applied to Requirements Trace Recovery,” (“SRC-Retrieval: Semantic-Based GitHub Repository and Code Retrieval”, p. 11) #5fb236
	  
	  - “TraceRefiner: An Automated Technique for Refining Coarse-Grained Requirement-toClass Traces,” (“SRC-Retrieval: Semantic-Based GitHub Repository and Code Retrieval”, p. 11) #5fb236
	  
	  - “Extracting Requirements Patterns from Software Repositories” (“SRC-Retrieval: Semantic-Based GitHub Repository and Code Retrieval”, p. 11) #5fb236
	  
	  COnsider that those that are tagget with #5fb236 are just highlights, those that are tagged with #e56eee and #a28ae5 are imporant sentences. Please pay attention instead to the notes that are tagged with #ffd400. Those that are tagged with #ff6666 are typos or errors. Could you please draft a review by organizing it as follows: 
	  
	  SUMMARY: Just a few sentence to summarize the work 
	  
	  STRENGHTS:
	  
	  WEAKNESSES:
	  
	  COMMENTS: Organize the notes with respect to the following criteria:
	  
	  - 
	  `Novelty`
	  
	  - 
	  `Rigor`
	  
	  - 
	  `Relevance (of the contribution)`
	  
	  - 
	  `Verifiability and Transparency`
	  
	  - 
	  `Presentation`
	  
	  And then add a Detailed Comments section to report the notes that contain issues or typos. 
	  Can you also formulate three explicit questions by considering the comments above?