4.7 KiB
4.7 KiB
collapsed:: true type:: REVIEWS tags:: year:: 2026 venue:: ICSE full-title:: SRC-Retrieval: Semantic-Based GitHub Repository and Code Retrieval date-start:: 16-09-2025 - 16:09 date-submitted:: external-links:: status:: DONE deadline-submission:: file:: @SRC-Retrieval: Semantic-Based GitHub Repository and Code Retrieval parent:: todoist:: https://app.todoist.com/app/task/3125-src-retrieval-semantic-based-git-hub-repository-and-code-retrieval-6cVHmvhGJfhPWjQg
- ### [[Highlights]]
- ### [[Comments]]
- #.tabular
- ### Paper summary
- The paper presents an approach to retrieving GitHub repositories and specific code files starting from functional requirements given by developers. Functionalities are extracted from README files and GitHub issues, which are about implemented functionalities. The approach has been compared with two different baselines.
- ### Strengths
- + Interesting and relevant topic
- + Linking functionalities with source code is indeed important
- ### Weaknesses
- Problems in the way dataset was constructed, especially regarding the role of classifiers in labeling large amounts of data, potentially introducing noise and cascading errors
- The approach relies heavily on README files for functionality extraction, but the validity and precision of these mappings are questionable.
- The evaluation section does not convincingly show that the retrieved code files actually meet user requirements in a semantically meaningful way.
- The baselines chosen for comparison appear weak, and the omission of modern tools like GitHub Copilot undermines the credibility of the evaluation.
- The presentation is redundant in different parts, especially with repeated technical details on model training.
- ### Detailed comments for authors
- Novelty: The paper presents a retrieval pipeline that integrates multiple transformers to map functional requirements to GitHub repositories and specific code files. While this integration is interesting, the paper is not convincing with respect to the quality of the obtained results especially in comparison with existing supporting technologies like Copilot. The considered baselines are not appropriate because they do not permit to actually stress the novelty of the proposed approach with respect to existing development assistance tools.
- Rigor: The approach heavly rely on multiple trained classifiers, each operating on noisy or weakly supervised data. For instance, the extraction of functionalities from README files lacks solid grounding and examples. The same applies to the issue-to-code linkage process, where the labeling process has been automated without discussing the accuracy of such an automated process. Thus the risk of compounded errors due to the different classifiers that have been used to create the training dataset is not addressed.
- Relevance: The paper is highly relevant to software engineering practice, particularly for developers seeking concrete examples of functionality implementations. However, the lack of comparison with widely adopted tools like Copilot reduces the practical significance of the proposed approach.
- Verifiability & transparency: How functional requirements are defined and extracted remains ambiguous. The use of classifiers to assess whether a GitHub issue represents a valid functionality or not is prone to mislabeling, and the paper does not provide error analysis or a discussion on its impact. The absence of ablation studies, particularly to separate the contribution of README-derived vs. issue-derived functionalities, is an important omission. Moreover, the evaluation lacks a user-centered validation of whether retrieved code files actually satisfy the intended requirement.
- Presentation: The paper is generally readable but suffers from redundancy. Many paragraphs related to the data creation steps are repeated, Some conceptual terms like “semantic alignment” or “repository functionality” are used early on but only vaguely defined.
- QUESTIONS:
- Q1: How have you quantified the effect of classification errors (e.g., incorrectly labeled issues or README sentences) on the overall retrieval performance?
- Q2: What is the actual contribution of README files to the overall retrieval pipeline? Have you considered an ablation study that separates the impact of README-derived vs. issue-derived functionalities?
- Q3: Why have you not compared your method against modern tools like GitHub Copilot?
- ### [[REVIEWS/Notes]]
- ### YELLOW CONCERNS
background-color:: yellow
- {{query (and [[ffd400]] [[ICSE2026-paper3125]] )}}
collapsed:: true
- ### ❓️Questions
- {{query (and [[question]] [[ICSE2026-paper3125]] )[[question]]}}
query-table:: true
query-properties:: [:block]