logseq/pages/hls__TOSEM-2023-0432_Proof_hi_1704400504261_0.md

file:: TOSEM-2023-0432_Proof_hi_1704400504261_0.pdf file-path:: ../assets/TOSEM-2023-0432_Proof_hi_1704400504261_0.pdf

• "APIDocBooster: An Extract-Then-Abstract Framework Leveraging Large Language Models for Augmenting API Documentation ls-type:: annotation hl-page:: 2 hl-color:: green id:: 65a552f8-8f15-4baf-87cf-ee9d147ffbf1 hl-stamp:: 1705333498845
• ACM Transactions on Software Engineering and Methodology ls-type:: annotation hl-page:: 2 hl-color:: green id:: 65a55303-227e-48e2-bdfd-ff0871860934
• API documents often fall short in comprehensiveness ls-type:: annotation hl-page:: 2 hl-color:: green id:: 65a55314-16ac-40c5-a6d5-f462afa1b8ce
• GPT-4 produces coherent and concise summaries, it lacks in informativeness and faithfulness ls-type:: annotation hl-page:: 2 hl-color:: green id:: 65a55347-3716-4d31-bf69-9c8939f45e13
• extractive-then-abstractive ls-type:: annotation hl-page:: 2 hl-color:: blue id:: 65a5534f-29df-4cff-a9ef-cc547259369b hl-stamp:: 1705333584901
• (1) Context-aware Sentence Section Classification (CSSC) ls-type:: annotation hl-page:: 2 hl-color:: green id:: 65a553aa-157a-4cc3-b531-6a901ed27828 hl-stamp:: 1705333676966
• UPdate SUMmarization (UPSUM) ls-type:: annotation hl-page:: 2 hl-color:: green id:: 65a553b0-11ef-44ce-8e76-1f0c4ae39ee1
• To enable automatic evaluation of APIDocBooster, we construct the first dataset for API documentation augmentation ls-type:: annotation hl-page:: 2 hl-color:: green id:: 65a553c5-897c-416d-826f-f3b702c566cf
• Our automatic evaluation results reveal that each stage in APIDocBooster outperforms its baselines by a large margin. Our human evaluation also demonstrates the superiority of APIDocBooster over GPT-4 ls-type:: annotation hl-page:: 2 hl-color:: green id:: 65a553f3-c4cf-4dc1-be96-609fa901858d
• API documentation is often the most trusted resource for programming. ls-type:: annotation hl-page:: 3 hl-color:: green id:: 65a554b2-0c0d-49fc-88fb-910ba5e78236
• any approaches have been proposed to augment API documentation by summarizing complementary information from external resources such as Stack Overflow. ls-type:: annotation hl-page:: 3 hl-color:: green id:: 65a554cf-0ffb-4cce-9ba0-4e03d1310765
• Augmenting API Documentation ls-type:: annotation hl-page:: 3 hl-color:: blue id:: 65aa84db-e2a9-4a0a-a823-8c951d31e8bf hl-stamp:: 1705673951774
• APIDocBooster ls-type:: annotation hl-page:: 3 hl-color:: blue id:: 65aa84dd-895b-4139-9a73-f18b41b663a9
• summarization ls-type:: annotation hl-page:: 3 hl-color:: green id:: 65aa84fd-e2f2-41f0-a082-14c7bc312f57
• summaries that accurately represent the source content without input length restrictions ls-type:: annotation hl-page:: 3 hl-color:: green id:: 65aa8505-59eb-47c1-a0f6-0f992c2c9149
• bstractive-based summarization method ls-type:: annotation hl-page:: 3 hl-color:: green id:: 65aa8510-fddc-4706-8ad0-e06d6960226b
• We introduce APIDocBooster, an extract-then-abstract framework that seamlessly fuses the advantages of both extractive (i.e., enabling faithful summaries without length limitation) and abstractive summarization(i.e., producing coherent and concise summaries) ls-type:: annotation hl-page:: 3 hl-color:: purple id:: 65aa8533-9ee5-477f-8f7a-096c2b45b9a7 hl-stamp:: 1705677474670
• APIDocBooster consists of two stages: (1) Context-aware Sentence Section Classification ( CSSC ) and (2) UPdate SUMmarization (UPSUM). CSSC classifies API-relevant information collected from multiple sources into API documentation sections. UPSUM first generates extractive summaries distinct from original API documentation and then generates abstractive summaries guided by extractive summaries through in-context learning ls-type:: annotation hl-page:: 3 hl-color:: green id:: 65aa879b-fdac-4741-81e4-d498b37b7315
• API documentation is a set of documents indexed by the API name, where each document provides information about a specific API [33 ]. ls-type:: annotation hl-page:: 4 hl-color:: green id:: 65aa89b5-3490-4717-8234-52d68932e68e
• technical documentation is the most trusted resource for programming ls-type:: annotation hl-page:: 4 hl-color:: blue id:: 65aa89bc-3315-442e-a483-5e7e4accc35e
• Existing API documents are often incomplete and not always equally readable [ 13, 33, 41, 44, 49]. ls-type:: annotation hl-page:: 4 hl-color:: green id:: 65aa89c9-4913-4dc0-ab17-46ef9e376e7d
• 60% of the participants have suffered from inadequate API documentation in the last three months ls-type:: annotation hl-page:: 4 hl-color:: green id:: 65aa89d6-5809-4d2d-8068-8996864e0a8b
• solutions to augment API documentation [ ls-type:: annotation hl-page:: 4 hl-color:: green id:: 65aa89de-820c-484e-9faa-e7499d4aacca
• update extractive summarization ls-type:: annotation hl-page:: 4 hl-color:: green id:: 65aa89e6-79df-4e82-b742-60e9074e3e4f
• abstractive summarization has garnered significant attention ls-type:: annotation hl-page:: 4 hl-color:: green id:: 65aa927a-2da9-49dd-9126-23f7e75d9b39
• Abstractive summarization generates new sentences, in contrast to extractive summarization which selects sentences from input. ls-type:: annotation hl-page:: 4 hl-color:: green id:: 65aa9289-84d9-4c4f-83b9-a143b338e35a
• However, its effectiveness on API documentation augmentation remains unknown ls-type:: annotation hl-page:: 4 hl-color:: blue id:: 65aa929d-35cb-4520-89dd-df6e884d30ef
• GPT-4 can generate coherent and concise summaries to augment API documentation ls-type:: annotation hl-page:: 4 hl-color:: green id:: 65aa92bb-6fae-4b79-a18c-585443411637
• drawbacks concerning informativeness and faithfulness ls-type:: annotation hl-page:: 4 hl-color:: purple id:: 65aa92c4-1e2c-4a4b-b7cb-01d2bdc00ed7
• input length limitation of GPT-4 leads to information loss due to the truncation of the prompt ls-type:: annotation hl-page:: 4 hl-color:: purple id:: 65aa92cf-1254-4b7c-902c-fd0c4e8775c3
• GPT-4 API may be a burden for individual developers generating documentation (e.g., each GPT-4 API call costs around $0.3 if the token number reaches the 8k limit ls-type:: annotation hl-page:: 4 hl-color:: purple id:: 65aa92eb-e333-4add-a6c6-92712f4ee4bb
• summaries that do not accurately reflect the original meaning of API-relevant resources ls-type:: annotation hl-page:: 4 hl-color:: purple id:: 65aa931d-c9a5-49e8-8dba-19c5efa6847f
• hallucination ls-type:: annotation hl-page:: 4 hl-color:: purple id:: 65aa9320-d75d-45fd-b56a-7c1adee40cb9
• assessing the faithfulness of such hallucinations may require substantial time or, in some cases, might not even be feasible ls-type:: annotation hl-page:: 4 hl-color:: blue id:: 65aa9334-25a8-4aca-8e00-e61423b795ae
• existence of hallucinations also hinders the adoption of GPT-4 in this task ls-type:: annotation hl-page:: 4 hl-color:: purple id:: 65aa9340-8b41-4d03-8ab0-cf705cfe3a74
• Besides, extractive summaries demonstrate a considerable degree of informativeness and relevance, since there is no limitation on input length ls-type:: annotation hl-page:: 5 hl-color:: green id:: 65aa9372-bd16-48f3-8162-a958f1ceb43e hl-stamp:: 1705677684890
• The success of extractive summaries in terms of informativeness and faithfulness inspires us to use extractive summarization to guide abstractive summarization, addressing the drawbacks of the latter. ls-type:: annotation hl-page:: 5 hl-color:: purple id:: 65aa9399-d50a-43aa-ac24-319efdebdb11
• We leverage an extractive-then-abstractive pipeline including two phases: 1) Extract Phase: extract insight sentences from external resources to form extractive summaries, and 2) Abstract Phase: ask GPT-4 to generate abstractive summaries guided by extractive summaries. ls-type:: annotation hl-page:: 5 hl-color:: purple id:: 65aa93cf-854a-4d0c-9d3d-6150dcf45874
• Extract Phase allows input without length limitations, ls-type:: annotation hl-page:: 5 hl-color:: yellow id:: 65aa93f9-c387-4b64-8513-7bcccf3aa3da
• he Abstract Phase ensures that abstractive summaries are aligned with extractive summaries, thereby enhancing faithfulness and facilitating data provenance. ls-type:: annotation hl-page:: 5 hl-color:: yellow id:: 65aa9490-1b18-4f22-88ab-757a93a34762
• API documentation augmentation requires generated summaries to be distinct from the original API documentation for readability [45]. However, existing approaches neglect to reduce redundancy between generated summaries and API documentation ls-type:: annotation hl-page:: 5 hl-color:: green id:: 65aa9754-05bf-4be0-a365-90070a5e32fe
• CSSC takes as input documents relevant to a specific API from multiple sources. CSSC identifies insight sentences and classifies them into suitable API documentation sections for structure awareness while considering the contextual dependency of sentences if necessary. ls-type:: annotation hl-page:: 5 hl-color:: green id:: 65ae6847-49d9-4c16-be65-dac6db622a6d
• extract-then-abstract pipeline, ls-type:: annotation hl-page:: 5 hl-color:: blue id:: 65ae6859-79ce-4ef5-90c4-b5aaeab82a70
• APISumBench ls-type:: annotation hl-page:: 5 hl-color:: blue id:: 65ae68a1-fd3d-4424-8750-7d6ad1cf84df
• ROUGE-1, ROUGE-2, and ROUGE- ls-type:: annotation hl-page:: 6 hl-color:: yellow id:: 65ae68c0-c2e0-438d-aa13-a2e3af5020e3
• We conduct the first empirical study on GPT-4 performing abstractive summarization for API documentation augmentation. ls-type:: annotation hl-page:: 6 hl-color:: green id:: 65ae6920-5f7d-4bdd-9b9e-cc2ac464ad62
• extractive-based summarization approaches. ls-type:: annotation hl-page:: 6 hl-color:: green id:: 65ae692b-9085-4274-af75-35e314ce384b
• We propose APIDocBooster, a two-stage approach with an extract-then-abstract framework to address these challenges. ls-type:: annotation hl-page:: 6 hl-color:: purple id:: 65ae6939-d56b-459a-a8db-7459ee83abb7
• [:span] ls-type:: annotation hl-page:: 7 hl-color:: green id:: 65ae696d-687c-45d7-8895-406903895f7a hl-type:: area hl-stamp:: 1705929068273
• inadequate API documentation. ls-type:: annotation hl-page:: 6 hl-color:: green id:: 65ae69b8-444c-40ed-bdd3-8519f46af506
• brief definition of the algorithm used in Tanh as well as the math formula. ls-type:: annotation hl-page:: 6 hl-color:: green id:: 65ae69c7-0ea1-4239-9cfc-9f89fe78fefb
• usage scenarios and range of the output for Tanh API, ls-type:: annotation hl-page:: 6 hl-color:: green id:: 65ae69f3-d8d0-4251-b9a3-484eed480f6b
• highlight an essential property of Tanh. ls-type:: annotation hl-page:: 6 hl-color:: yellow id:: 65ae6a02-61ce-4073-80b9-a8aab3e06777
• tutorial videos related to the target API often cover detailed operation steps of using the target API. ls-type:: annotation hl-page:: 6 hl-color:: green id:: 65ae6c7b-f3ff-4845-b907-5e237df7343c hl-stamp:: 1705929856901
• These observations inspire us to believe that the information contained in both YouTube videos and Stack Overflow posts has great potential to augment API documentation and complement each other ls-type:: annotation hl-page:: 6 hl-color:: blue id:: 65ae6c97-7b29-4f06-95ef-527ae81e3da6
• Presenting either of these two sentences individually proves insufficient in conveying comprehensive insights concerning the target API. ls-type:: annotation hl-page:: 7 hl-color:: green id:: 65ae6cde-fb8e-438d-bc84-6b38446fc531
• This motivates us to incorporate contextual information when performing API documentation augmentation ls-type:: annotation hl-page:: 7 hl-color:: blue id:: 65ae6cf5-57ab-42f1-82fd-5f27f9689916
• Task Formulation ls-type:: annotation hl-page:: 7 hl-color:: green id:: 65ae7951-6354-4455-a7f8-e19d72e90148
• The reason for this coarse categorization is to better fit the structure of real-world API documentation. ls-type:: annotation hl-page:: 7 hl-color:: green id:: 65ae79ef-7789-41c3-86f4-f2bcaa91ea9d
• augment API documentation by generating complementary summaries for each documentation section from multiple online sources ls-type:: annotation hl-page:: 7 hl-color:: blue id:: 65aeed8a-84ae-4721-ab11-bdb878b0001b
• As a initial attempt, we consider two sources, i.e., Stack Overflow and YouTube, as input sources ls-type:: annotation hl-page:: 7 hl-color:: blue id:: 65aeed92-4606-4c33-8a66-d0cdbc5e0336
• , Functionality, Parameter, and Notes ls-type:: annotation hl-page:: 7 hl-color:: green id:: 65aeedac-a438-48cb-a163-2ab81581a438
• API documentation augmentation task ls-type:: annotation hl-page:: 8 hl-color:: green id:: 65aeee01-7133-4466-bb61-db19075054bd
• We aim to generate extractive summaries (i.e., an optimal subset 𝑠𝑢𝑚 =𝐸𝑥𝑡𝑟𝑎𝑐𝑡 (𝑠 |𝑎𝑝𝑖, 𝑑𝑜𝑐), 𝑠𝑢𝑚 ⊆ s), and then abstractive summaries (i.e., 𝑠𝑢𝑚′= 𝐴𝑏𝑠𝑡𝑟𝑎𝑐𝑡 (𝑠𝑢𝑚|𝑎𝑝𝑖, 𝑑𝑜𝑐)) that satisfy the requirements of readability, relevance of content, and structure of API documentation. ls-type:: annotation hl-page:: 8 hl-color:: purple id:: 65aeee6e-4c01-4514-bbe7-6e07f86d3540
• We carry out an empirical study to understand the capacity and limitations of using GPT-4 to augment API documentation. ls-type:: annotation hl-page:: 8 hl-color:: blue id:: 65b53353-51f4-4e98-a213-a533458e4d31
• API-relevant information exists in external sources ls-type:: annotation hl-page:: 8 hl-color:: green id:: 65b533df-0086-456d-890e-14a4c323f4b4
• e randomly sample 100 APIs from each library, respectively. As discussed in Section 2.2, we consider Stack Overflow and YouTube as preliminary external sources. ls-type:: annotation hl-page:: 8 hl-color:: yellow id:: 65b533fb-538b-4ae9-8219-3c5b9eeed815
• We set the parameter temperature of GPT-4 as 0 to ensure GPT-4 produces a deterministic response. ls-type:: annotation hl-page:: 8 hl-color:: blue id:: 65b540ac-cc33-4f4d-8db8-e6015cb956af hl-stamp:: 1706377390776
• Zero-shot Setting ls-type:: annotation hl-page:: 8 hl-color:: green id:: 65b54362-3daa-430e-9609-94680bd22e48
• Summarization Setting ls-type:: annotation hl-page:: 8 hl-color:: green id:: 65b54367-239a-45dd-bb13-5a5b68e1f796
• read all augmented API documentation pages and rate each section on a 5-point Likert scale in terms of informativeness, relevance, readability, non-redundancy, and faithfulness. ls-type:: annotation hl-page:: 9 hl-color:: green id:: 65b54429-9992-4225-a903-00fe589a7035
• Participants are also required to highlight sentences that they think should not be in API documentation given the above criteria ls-type:: annotation hl-page:: 9 hl-color:: green id:: 65b5444f-607e-4678-97cc-764316f13091
• faithfulness is a measure of the degree to which information in summaries aligns with corresponding information in external resources. ls-type:: annotation hl-page:: 9 hl-color:: green id:: 65b54463-a95b-4fbb-84ec-e163051f7db2
• However, it is impossible to assess the faithfulness of 𝐺𝑃𝑇𝑧𝑒𝑟𝑜 as GPT-4 generates the summaries without external resources as reference ls-type:: annotation hl-page:: 9 hl-color:: purple id:: 65b54468-1a61-48ec-8f17-2277ae00d682
• APIDocBooster simplifies the tedious task of documentation maintenance [12 ] into an automatic pipeline: 1) automatically collecting API-related information from external sources; 2) automatically generating summaries for each section of the API documentation; and 3) integrating the generated summaries into the original API documentation ls-type:: annotation hl-page:: 10 hl-color:: green id:: 65b544e3-9d23-44cb-b219-be2d5522a526
• context identifier ls-type:: annotation hl-page:: 11 hl-color:: green id:: 65b56de3-0110-4ca8-9d22-5e409a4fa4e0
• entence section classifie ls-type:: annotation hl-page:: 11 hl-color:: green id:: 65b56de9-5e51-497a-b691-7ec119f6f05d
• we generate extractive summaries for each section by using our proposed update summarization algorithm. ls-type:: annotation hl-page:: 11 hl-color:: green id:: 65b56df8-bb7a-4cb8-bdad-80fb2b1f72f7
• RQ1: How effective is CSSC for sentence section classification? ls-type:: annotation hl-page:: 19 hl-color:: green id:: 65b56e38-2696-41a4-bc44-4c7bfdbf3bab
• RQ2: How effective is ExtUP for update extractive summarization? ls-type:: annotation hl-page:: 19 hl-color:: green id:: 65b56e3d-7686-454d-9653-e2fa0617bd92
• 4 EXPERIMENT SETTING ls-type:: annotation hl-page:: 16 hl-color:: yellow id:: 65b56e8b-ee70-46cc-96a8-f86e4f380c17
• 2.3 Empirical Study on GPT-4 ls-type:: annotation hl-page:: 8 hl-color:: yellow id:: 65b56eb1-cd47-44c1-b734-64d180583793