23 KiB
23 KiB
type:: REVIEWS tags:: year:: 2024 venue:: ICSE full-title:: Knowledge-guided Large Language Models are Trustworthy API Recommenders date-start:: 16-04-2024 - 13:34 date-submitted:: external-links:: status:: DONE deadline-submission:: file:: @icse2025-paper215.pdf parent:: todoist:: https://app.todoist.com/app/task/215-knowledge-guided-large-language-models-are-trustworthy-api-recommenders-7858696695
- ### [[Highlights]]
collapsed:: true
-
- # Annotazioni
- (4/6/2024, 14:15:07)
- - “Application Programming Interface (API) recommendation aims to recommend APIs for developers that meet their functional requirements, which can compensate for developers’ lack of API knowledge” ([“icse2025-paper215.pdf”, p. 1](zotero://select/library/items/9ZBQY9SL)) ([pdf](zotero://open-pdf/library/items/WDJYB643?page=1&annotation=6QSLXJQN)) #a28ae5
- - “Instances of this include recommending fictitious APIs, APIs whose calling conditions cannot be satisfied, or API sequences failure to conform to the interface parameter types.” ([“icse2025-paper215.pdf”, p. 1](zotero://select/library/items/9ZBQY9SL)) ([pdf](zotero://open-pdf/library/items/WDJYB643?page=1&annotation=L6IA6DAP)) #a28ae5
- - “Knowledge-guided framework for LLM-based API Recommendation (KG4LLM)” ([“icse2025-paper215.pdf”, p. 1](zotero://select/library/items/9ZBQY9SL)) ([pdf](zotero://open-pdf/library/items/WDJYB643?page=1&annotation=2I366KAQ)) #f0ff00
- *is it similarto RAG?*
- - “Java Development Kit (JDK) documentation to” ([“icse2025-paper215.pdf”, p. 1](zotero://select/library/items/9ZBQY9SL)) ([pdf](zotero://open-pdf/library/items/WDJYB643?page=1&annotation=H6GBTVPW)) #f0ff00
- *maybe the jdk doumentaion is already in the training set of the used LLM?*
- - “even the most skilled developer can hardly master using all the APIs. Therefore, this paper focuses on the API recommendation task to compensate for developers’ lack of API knowledge.” ([“icse2025-paper215.pdf”, p. 1](zotero://select/library/items/9ZBQY9SL)) ([pdf](zotero://open-pdf/library/items/WDJYB643?page=1&annotation=88KC5QLG)) #a28ae5
- - “specific interface parameter types predefined by the software architect” ([“icse2025-paper215.pdf”, p. 1](zotero://select/library/items/9ZBQY9SL)) ([pdf](zotero://open-pdf/library/items/WDJYB643?page=1&annotation=E69WA4N4)) #e56eee
- - “API Recommendation under specific Interface Parameter Types (APIRIP)” ([“icse2025-paper215.pdf”, p. 1](zotero://select/library/items/9ZBQY9SL)) ([pdf](zotero://open-pdf/library/items/WDJYB643?page=1&annotation=LDI9WT77)) #a28ae5
- - “During APIRIP, the APIrecommender system takes functional description and interface parameter types (if no interface parameter types, set them to “void”) as the query and recommends API sequences that conform to the interface parameter types to fulfill the functional requirement.” ([“icse2025-paper215.pdf”, p. 1](zotero://select/library/items/9ZBQY9SL)) #00b036
- - “The parameter types “Desktop” and “URL”, on which the recommended API sequence relies, conform to the interface parameter types.” ([“icse2025-paper215.pdf”, p. 1](zotero://select/library/items/9ZBQY9SL)) ([pdf](zotero://open-pdf/library/items/WDJYB643?page=1&annotation=UQSRGBMW)) #a28ae5
- - “APIRID requires API recommender systems to be sensitive enough to the parameter types defined in the interface.” ([“icse2025-paper215.pdf”, p. 1](zotero://select/library/items/9ZBQY9SL)) ([pdf](zotero://open-pdf/library/items/WDJYB643?page=1&annotation=U9GWJBTL)) #e56eee
- - “In this condition, to “open URL by Desktop object”, the API recommender system should recommend an additional API named “Desktop.getDesktop()” to generate a “Desktop” object. As a result, the recommended API sequence only relies on the parameter “URL”, which conforms to the interface parameter types. Similarly, if the interface parameter type becomes “void”, an additional API “URL.URL(String)” will be added to the recommended API sequence to create an “URL” object” ([“icse2025-paper215.pdf”, p. 1](zotero://select/library/items/9ZBQY9SL)) ([pdf](zotero://open-pdf/library/items/WDJYB643?page=1&annotation=5YVJ67KE)) #a28ae5
- - “Fig. 1. Examples of APIRID.” ([“icse2025-paper215.pdf”, p. 1](zotero://select/library/items/9ZBQY9SL)) ([pdf](zotero://open-pdf/library/items/WDJYB643?page=1&annotation=NVK7N6J4)) #f0ff00
- *it's not completely cleary. Let's see if later gets better.*
- - “previous” ([“icse2025-paper215.pdf”, p. 1](zotero://select/library/items/9ZBQY9SL)) ([pdf](zotero://open-pdf/library/items/WDJYB643?page=1&annotation=Y3E6WG48)) #f0ff00
- *previous and then you put a reference to the next section?*
- - “UTIs” ([“icse2025-paper215.pdf”, p. 1](zotero://select/library/items/9ZBQY9SL)) ([pdf](zotero://open-pdf/library/items/WDJYB643?page=1&annotation=TXAYK2DM)) #ffd400
- *what about hallucinations that you previously mentioned?*
- - “fictitious APIs” ([“icse2025-paper215.pdf”, p. 1](zotero://select/library/items/9ZBQY9SL)) ([pdf](zotero://open-pdf/library/items/WDJYB643?page=1&annotation=RH29NPY6)) #a28ae5
- - “unsatisfied calling conditions” ([“icse2025-paper215.pdf”, p. 1](zotero://select/library/items/9ZBQY9SL)) ([pdf](zotero://open-pdf/library/items/WDJYB643?page=1&annotation=4IGAJNDL)) #a28ae5
- - “interface parameter types.” ([“icse2025-paper215.pdf”, p. 1](zotero://select/library/items/9ZBQY9SL)) ([pdf](zotero://open-pdf/library/items/WDJYB643?page=1&annotation=CN6M2TND)) #a28ae5
- - “beam search” ([“icse2025-paper215.pdf”, p. 2](zotero://select/library/items/9ZBQY9SL)) ([pdf](zotero://open-pdf/library/items/WDJYB643?page=2&annotation=YNH6RR6Y)) #a28ae5
- - “fine-tuning” ([“icse2025-paper215.pdf”, p. 2](zotero://select/library/items/9ZBQY9SL)) ([pdf](zotero://open-pdf/library/items/WDJYB643?page=2&annotation=5IGS7HY3)) #f0ff00
- - “UTI 3” ([“icse2025-paper215.pdf”, p. 2](zotero://select/library/items/9ZBQY9SL)) ([pdf](zotero://open-pdf/library/items/WDJYB643?page=2&annotation=H4W2WS9P)) #e56eee
- - “data augmentation,” ([“icse2025-paper215.pdf”, p. 2](zotero://select/library/items/9ZBQY9SL)) ([pdf](zotero://open-pdf/library/items/WDJYB643?page=2&annotation=TJJ8EQ9K)) #f0ff00
- *check this how it works later.*
- - “generate more training instances with more matched combinations of interface parameter types and API sequences to fine-tune LLM” ([“icse2025-paper215.pdf”, p. 2](zotero://select/library/items/9ZBQY9SL)) ([pdf](zotero://open-pdf/library/items/WDJYB643?page=2&annotation=GHHWIW79)) #a28ae5
- - “UTI 1, UTI 2 and further alleviate UTI 3,” ([“icse2025-paper215.pdf”, p. 2](zotero://select/library/items/9ZBQY9SL)) ([pdf](zotero://open-pdf/library/items/WDJYB643?page=2&annotation=FPPYFSEU)) #e56eee
- - “Finally, we compile a Java dataset tailored for the APIRIP task and introduce a set of evaluation metrics designed to assess the trustworthiness of LLM recommendations.” ([“icse2025-paper215.pdf”, p. 2](zotero://select/library/items/9ZBQY9SL)) ([pdf](zotero://open-pdf/library/items/WDJYB643?page=2&annotation=J5ENY8SH)) #a28ae5
- - “benchmark LLMs’ T R@1” ([“icse2025-paper215.pdf”, p. 2](zotero://select/library/items/9ZBQY9SL)) ([pdf](zotero://open-pdf/library/items/WDJYB643?page=2&annotation=EP5ZV595)) #a28ae5
- - “new variant of the API recommendation task named APIRIP for scenarios where developers program under specific interface parameter types.” ([“icse2025-paper215.pdf”, p. 2](zotero://select/library/items/9ZBQY9SL)) ([pdf](zotero://open-pdf/library/items/WDJYB643?page=2&annotation=BV938QAR)) #a28ae5
- - “first study on the trustworthiness of LLM for API recommendation task.” ([“icse2025-paper215.pdf”, p. 2](zotero://select/library/items/9ZBQY9SL)) ([pdf](zotero://open-pdf/library/items/WDJYB643?page=2&annotation=VIV7QB3J)) #a28ae5
- - “KG4LLM can effectively improve the trustworthiness of LLM” ([“icse2025-paper215.pdf”, p. 2](zotero://select/library/items/9ZBQY9SL)) ([pdf](zotero://open-pdf/library/items/WDJYB643?page=2&annotation=UTCPSIXI)) #a28ae5
- - “CodeBERT” ([“icse2025-paper215.pdf”, p. 2](zotero://select/library/items/9ZBQY9SL)) ([pdf](zotero://open-pdf/library/items/WDJYB643?page=2&annotation=83RXB5YX)) #ffd400
- - “When UTI 3 occurs, it often means that the recommended APIs do not meet the developer’s functional requirements.” ([“icse2025-paper215.pdf”, p. 2](zotero://select/library/items/9ZBQY9SL)) ([pdf](zotero://open-pdf/library/items/WDJYB643?page=2&annotation=98NF3NN9)) #5fb236
- - “Since the generation process of existing LLMs is mainly based on probabilistic sampling of tokens without any constraints for trustworthiness, the untrustworthy issue is common in almost all varieties of LLM.” ([“icse2025-paper215.pdf”, p. 2](zotero://select/library/items/9ZBQY9SL)) ([pdf](zotero://open-pdf/library/items/WDJYB643?page=2&annotation=L3GHDPTQ)) #5fb236
- - “Fig. 3. Overall framework of KG4LLM.” ([“icse2025-paper215.pdf”, p. 3](zotero://select/library/items/9ZBQY9SL)) ([pdf](zotero://open-pdf/library/items/WDJYB643?page=3&annotation=UZ7TNL6P)) #ffd400
- *It's not clear how the proposed approach has been implemented. Fig. 3 is fairly clear, even though later in the paper it is not clear how the different components of Fig. 3 have been implemented. It seems that CodeT5 underpins the proposed approach, even though it is not clear how Algorithm 1 and Algorithm 2 have been developed in synergy with CodetT5.*
- - “generate more training instances containing more combinations of matched interface parameter types and API sequences for fine-tuning LLMs” ([“icse2025-paper215.pdf”, p. 3](zotero://select/library/items/9ZBQY9SL)) ([pdf](zotero://open-pdf/library/items/WDJYB643?page=3&annotation=2HIZD8AT)) #5fb236
- - “interface parameter types after the functional description and adding a separator between them as the input.” ([“icse2025-paper215.pdf”, p. 3](zotero://select/library/items/9ZBQY9SL)) ([pdf](zotero://open-pdf/library/items/WDJYB643?page=3&annotation=E4ZY2NYC)) #5fb236
- - “vocabulary constraints to guarantee that LLMs can only generate real-world APIs whose calling conditions can be met.” ([“icse2025-paper215.pdf”, p. 3](zotero://select/library/items/9ZBQY9SL)) ([pdf](zotero://open-pdf/library/items/WDJYB643?page=3&annotation=D6B3UB7W)) #ffd400
- *What does it mean. How can you ensure that calling conditions are met?*
- - “ontology of API as shown in Figure 4.” ([“icse2025-paper215.pdf”, p. 3](zotero://select/library/items/9ZBQY9SL)) ([pdf](zotero://open-pdf/library/items/WDJYB643?page=3&annotation=QYMAUJHN)) #5fb236
- - “dimension of the API parameter, API return type, or member type.” ([“icse2025-paper215.pdf”, p. 3](zotero://select/library/items/9ZBQY9SL)) ([pdf](zotero://open-pdf/library/items/WDJYB643?page=3&annotation=SSPYE28F)) #ffd400
- *Why dim is relevant?*
- - “method annotations as the functional descriptions” ([“icse2025-paper215.pdf”, p. 3](zotero://select/library/items/9ZBQY9SL)) ([pdf](zotero://open-pdf/library/items/WDJYB643?page=3&annotation=PVID2R8I)) #ffd400
- *This is a potential threat to validity*
- - “Fig. 6” ([“icse2025-paper215.pdf”, p. 4](zotero://select/library/items/9ZBQY9SL)) ([pdf](zotero://open-pdf/library/items/WDJYB643?page=4&annotation=3FEWU7UZ)) #ffd400
- *byte should be byte[] in the "interface parameter types" section, isn't it?*
- - “Knowledge-Guided Data Augmentation” ([“icse2025-paper215.pdf”, p. 4](zotero://select/library/items/9ZBQY9SL)) ([pdf](zotero://open-pdf/library/items/WDJYB643?page=4&annotation=IVZ8RWWN)) #5fb236
- - “knowledge-guided data augmentation” ([“icse2025-paper215.pdf”, p. 4](zotero://select/library/items/9ZBQY9SL)) ([pdf](zotero://open-pdf/library/items/WDJYB643?page=4&annotation=LUBBJRTF)) #5fb236
- - “we treat the unsatisfied parameter types of the augmented API sequence as interface parameter types to ensure that enhanced API sequences can conform to these interface parameter types.” ([“icse2025-paper215.pdf”, p. 4](zotero://select/library/items/9ZBQY9SL)) ([pdf](zotero://open-pdf/library/items/WDJYB643?page=4&annotation=6PHJ7IZ9)) #ffd400
- *While such generation can be beneficial to avoid calling conditions that are not satisfied, this my contribute to the generation of fictitious APIs, isn't it?*
- - “Fig” ([“icse2025-paper215.pdf”, p. 4](zotero://select/library/items/9ZBQY9SL)) ([pdf](zotero://open-pdf/library/items/WDJYB643?page=4&annotation=STXNDA7G)) #ffd400
- *Why FileFilter has been renamed as FilenameFilter in Fig.7?*
- - “Each triple consists of a interface parameter types-API sequence pair with a score that represents the quality of the pair” ([“icse2025-paper215.pdf”, p. 4](zotero://select/library/items/9ZBQY9SL)) ([pdf](zotero://open-pdf/library/items/WDJYB643?page=4&annotation=2QDDISYV)) #5fb236
- - “number of APIs whose return value is not used by subsequent APIs, which is designed based on the following intuition: After calling an API, if it returns a object, it is likely to be used by the subsequent APIs. The smaller the score, the higher the quality. Next, we sort all the triples according to the score in the ascending order, and output top 5 of them. Finally, we combine each generated interface parameter typesAPI sequence pair with the original functional description text and add them to the training set.” ([“icse2025-paper215.pdf”, p. 4](zotero://select/library/items/9ZBQY9SL)) ([pdf](zotero://open-pdf/library/items/WDJYB643?page=4&annotation=RWPL4W3E)) #ffd400
- *"The smaller the score, the higher the quality" What's the concept of quality that is used here?*
- - “Knowledge-Guided Beam Search” ([“icse2025-paper215.pdf”, p. 4](zotero://select/library/items/9ZBQY9SL)) ([pdf](zotero://open-pdf/library/items/WDJYB643?page=4&annotation=NMT268QK)) #5fb236
- - “vocabulary constraints and beam reordering.” ([“icse2025-paper215.pdf”, p. 4](zotero://select/library/items/9ZBQY9SL)) ([pdf](zotero://open-pdf/library/items/WDJYB643?page=4&annotation=X3YA7W2I)) #a28ae5
- - “we propose vocabulary constraints to guide LLMs only to generate real-world APIs with satisfied calling conditions by constraining the output vocabulary based on the API knowledge base.” ([“icse2025-paper215.pdf”, p. 4](zotero://select/library/items/9ZBQY9SL)) ([pdf](zotero://open-pdf/library/items/WDJYB643?page=4&annotation=Y7YL4MHA)) #5fb236
- - “generate” ([“icse2025-paper215.pdf”, p. 5](zotero://select/library/items/9ZBQY9SL)) ([pdf](zotero://open-pdf/library/items/WDJYB643?page=5&annotation=CN8U7GKV)) #ff6666
- - “If yes, we multiply a manually set weight r on the score of the API sequence given by beam search and reorder all the API sequences in the beam list” ([“icse2025-paper215.pdf”, p. 5](zotero://select/library/items/9ZBQY9SL)) ([pdf](zotero://open-pdf/library/items/WDJYB643?page=5&annotation=R7U6RE5N)) #ffd400
- *This is not clear. What's the score r?*
- - “We set the beam size to 2 and r to 2 (because we change the log softmax to softmax, r must be greater than 1 to increase a positive score in this illustrative example). We use the average of the probabilities of generating each token (marked in purple in Figure 8 ) to denote the score of each API sequence. For example, the score of “file.read<e>” is p(f ile.read < e >) = 1 4 (p(f ile| < s >) + p(.| < s > f ile) + p(read| < s > f ile.) + p(< e > | < s > f ile.read)) = 1 4 (0.3 + 0.6 + 0.4 + 0.5) = 0.45.” ([“icse2025-paper215.pdf”, p. 5](zotero://select/library/items/9ZBQY9SL)) ([pdf](zotero://open-pdf/library/items/WDJYB643?page=5&annotation=9RF9ZQ4K)) #ffd400
- *This is not very clear. The description of Fig. 8 should be improved.*
- - “knowledge-guided data augmentation” ([“icse2025-paper215.pdf”, p. 7](zotero://select/library/items/9ZBQY9SL)) ([pdf](zotero://open-pdf/library/items/WDJYB643?page=7&annotation=WXD7X8KU)) #5fb236
- - “knowledgeguided beam search” ([“icse2025-paper215.pdf”, p. 7](zotero://select/library/items/9ZBQY9SL)) ([pdf](zotero://open-pdf/library/items/WDJYB643?page=7&annotation=TY4AJT4Y)) #5fb236
- - “SantaCoder” ([“icse2025-paper215.pdf”, p. 7](zotero://select/library/items/9ZBQY9SL)) ([pdf](zotero://open-pdf/library/items/WDJYB643?page=7&annotation=E4UGCGKY)) #5fb236
- - “This may be because a larger model has a stronger fitting ability, which may be more likely to lead to overfitting after increasing the number of samples through data augmentation” ([“icse2025-paper215.pdf”, p. 7](zotero://select/library/items/9ZBQY9SL)) ([pdf](zotero://open-pdf/library/items/WDJYB643?page=7&annotation=RB5BMCG3)) #5fb236
- - “Therefore, the knowledge-guided data augmentation may be more suitable for smaller LLMs.” ([“icse2025-paper215.pdf”, p. 7](zotero://select/library/items/9ZBQY9SL)) ([pdf](zotero://open-pdf/library/items/WDJYB643?page=7&annotation=YCV6A9RJ)) #e56eee
- - “number of interface parameter types varies” ([“icse2025-paper215.pdf”, p. 7](zotero://select/library/items/9ZBQY9SL)) ([pdf](zotero://open-pdf/library/items/WDJYB643?page=7&annotation=RC665IDQ)) #5fb236
- - “reducing different types of untrustworthy issues” ([“icse2025-paper215.pdf”, p. 8](zotero://select/library/items/9ZBQY9SL)) ([pdf](zotero://open-pdf/library/items/WDJYB643?page=8&annotation=P7DABJ7P)) #5fb236
- - “This is because LLMs are guided to generate only realworld APIs whose calling conditions can be met through the vocabulary constraints.” ([“icse2025-paper215.pdf”, p. 8](zotero://select/library/items/9ZBQY9SL)) ([pdf](zotero://open-pdf/library/items/WDJYB643?page=8&annotation=C7AZHQAB)) #5fb236
- - “[11]” ([“icse2025-paper215.pdf”, p. 11](zotero://select/library/items/9ZBQY9SL)) #00b036
- ### [[Comments]]
- #.tabular
- Summary
- The paper presents an approach to recommending APIs to developers based on the textual description of the functional requirements and predefined interface parameter types. This framework aims to mitigate the untrustworthy issues (UTIs) common in LLMs, such as recommending fictitious APIs or those with unsatisfied calling conditions. The approach leverages API knowledge from the Java Development Kit (JDK) documentation and consists of knowledge-guided data augmentation and beam search to improve the trustworthiness of LLM-generated recommendations. The experimental results demonstrate significant trustworthiness and performance metrics improvements across various LLMs.
- Strengths
- The paper introduces a novel approach supporting the recommendation of APIs to conform to specific interface parameter types, addressing a practical problem in software development.
- Extensive experiments have been conducted across multiple LLMs.
- Weaknesses
- The implementation details of the approach are not sufficiently clear. Figures and descriptions could be improved for better understanding.
- The paper mentions hallucinations but does not provide a detailed discussion on how these are specifically addressed in the context of API recommendations.
- Detailed Comments
- Novelty: The paper presents an interesting approach to dealing with the problem of API recommendation. Typically, existing approaches can recommend APIs by taking as input the current development context without taking into account potential constraints related to the type and number of parameters of the wanted methods.
- Rigor: The proposed framework is properly designed to address the identified issues in LLMs. However, the clarity of implementation details could be improved, particularly how different components interact and support each other.
- Relevance: APIRIP is highly relevant to real-world software development scenarios where developers often face challenges in selecting appropriate APIs due to the vast number of available options and the complexity of interface parameter constraints. Thus, the proposed framework addresses a critical need in this domain.
- Verifiability & Transparency: The paper provides a comprehensive dataset and an API knowledge base, which are valuable for verifying the results and extending the research. However, the README file is not properly detailed in describing the different scripts and how to use them. Reviewers should see the content of the Python scripts and the provided data sets on their own.
- Presentation: The overall presentation of the paper is clear, but certain sections, especially those explaining the methodology and results, could benefit from additional clarity. Figures such as Fig. 3 and Fig. 8 should be revised for better comprehension, and the descriptions should be more detailed and precise.
- Detailed commnents:
- The proposed framework shares similarities with Retrieval-Augmented Generation (RAG) in leveraging external knowledge to guide LLM outputs. However, more explicit comparisons and discussions could enhance understanding.
- The paper briefly touches on hallucinations and their impact on UTIs. A more detailed discussion on detecting and mitigating these hallucinations would be beneficial.
- The relevance of dimensions such as API parameter types and return types should be explained in the context of their impact on recommendations.
- The reliance on method annotations as functional descriptions could be a threat to validity, as these annotations may not always be accurate or comprehensive.
- The concept of quality used in scoring API sequences should be defined more clearly. What do you mean by quality in this domain? How are you assessing the quality of recommendations?
- Fig. 3 is fairly clear, even though later in the paper, it needs to be clarified how the different components of Fig. 3 have been implemented. Different LLMs have been used even though it is unclear how Algorithm 1 and Algorithm 2 have been developed in synergy with them.
- Page 3: "vocabulary constraints to guarantee that LLMs can only generate real-world APIs whose calling conditions can be met." -> What does it mean. How can you ensure that calling conditions are met?
- Fig. 6: byte should be byte[] in the "interface parameter types" section, isn't it?
- Page 4: "we treat the unsatisfied parameter types of the augmented API sequence as interface parameter types to ensure that enhanced API sequences can conform to these interface parameter types." -> While such a generation can be beneficial in avoiding calling conditions that are not satisfied, this may contribute to the generation of fictitious APIs, isn't it?
- Fig. 7: Why FileFilter has been renamed as FilenameFilter?
- ### [[REVIEWS/Notes]]
- ### YELLOW CONCERNS
background-color:: yellow
- {{query (and [[ffd400]] [[icse2025-paper215]] )}}
collapsed:: true
- ### ❓️Questions
- {{query (and [[question]] [[icse2025-paper215]] )[[question]]}}
query-table:: true
query-properties:: [:block]