423 lines
17 KiB
Markdown
423 lines
17 KiB
Markdown
file:: [TOSEM-2023-0432_Proof_hi_1704400504261_0.pdf](../assets/TOSEM-2023-0432_Proof_hi_1704400504261_0.pdf)
|
|
file-path:: ../assets/TOSEM-2023-0432_Proof_hi_1704400504261_0.pdf
|
|
|
|
- "APIDocBooster: An Extract-Then-Abstract Framework Leveraging Large Language Models for Augmenting API Documentation
|
|
ls-type:: annotation
|
|
hl-page:: 2
|
|
hl-color:: green
|
|
id:: 65a552f8-8f15-4baf-87cf-ee9d147ffbf1
|
|
hl-stamp:: 1705333498845
|
|
- ACM Transactions on Software Engineering and Methodology
|
|
ls-type:: annotation
|
|
hl-page:: 2
|
|
hl-color:: green
|
|
id:: 65a55303-227e-48e2-bdfd-ff0871860934
|
|
- API documents often fall short in comprehensiveness
|
|
ls-type:: annotation
|
|
hl-page:: 2
|
|
hl-color:: green
|
|
id:: 65a55314-16ac-40c5-a6d5-f462afa1b8ce
|
|
- GPT-4 produces coherent and concise summaries, it lacks in informativeness and faithfulness
|
|
ls-type:: annotation
|
|
hl-page:: 2
|
|
hl-color:: green
|
|
id:: 65a55347-3716-4d31-bf69-9c8939f45e13
|
|
- extractive-then-abstractive
|
|
ls-type:: annotation
|
|
hl-page:: 2
|
|
hl-color:: blue
|
|
id:: 65a5534f-29df-4cff-a9ef-cc547259369b
|
|
hl-stamp:: 1705333584901
|
|
- (1) Context-aware Sentence Section Classification (CSSC)
|
|
ls-type:: annotation
|
|
hl-page:: 2
|
|
hl-color:: green
|
|
id:: 65a553aa-157a-4cc3-b531-6a901ed27828
|
|
hl-stamp:: 1705333676966
|
|
- UPdate SUMmarization (UPSUM)
|
|
ls-type:: annotation
|
|
hl-page:: 2
|
|
hl-color:: green
|
|
id:: 65a553b0-11ef-44ce-8e76-1f0c4ae39ee1
|
|
- To enable automatic evaluation of APIDocBooster, we construct the first dataset for API documentation augmentation
|
|
ls-type:: annotation
|
|
hl-page:: 2
|
|
hl-color:: green
|
|
id:: 65a553c5-897c-416d-826f-f3b702c566cf
|
|
- Our automatic evaluation results reveal that each stage in APIDocBooster outperforms its baselines by a large margin. Our human evaluation also demonstrates the superiority of APIDocBooster over GPT-4
|
|
ls-type:: annotation
|
|
hl-page:: 2
|
|
hl-color:: green
|
|
id:: 65a553f3-c4cf-4dc1-be96-609fa901858d
|
|
- API documentation is often the most trusted resource for programming.
|
|
ls-type:: annotation
|
|
hl-page:: 3
|
|
hl-color:: green
|
|
id:: 65a554b2-0c0d-49fc-88fb-910ba5e78236
|
|
- any approaches have been proposed to augment API documentation by summarizing complementary information from external resources such as Stack Overflow.
|
|
ls-type:: annotation
|
|
hl-page:: 3
|
|
hl-color:: green
|
|
id:: 65a554cf-0ffb-4cce-9ba0-4e03d1310765
|
|
- Augmenting API Documentation
|
|
ls-type:: annotation
|
|
hl-page:: 3
|
|
hl-color:: blue
|
|
id:: 65aa84db-e2a9-4a0a-a823-8c951d31e8bf
|
|
hl-stamp:: 1705673951774
|
|
- APIDocBooster
|
|
ls-type:: annotation
|
|
hl-page:: 3
|
|
hl-color:: blue
|
|
id:: 65aa84dd-895b-4139-9a73-f18b41b663a9
|
|
- summarization
|
|
ls-type:: annotation
|
|
hl-page:: 3
|
|
hl-color:: green
|
|
id:: 65aa84fd-e2f2-41f0-a082-14c7bc312f57
|
|
- summaries that accurately represent the source content without input length restrictions
|
|
ls-type:: annotation
|
|
hl-page:: 3
|
|
hl-color:: green
|
|
id:: 65aa8505-59eb-47c1-a0f6-0f992c2c9149
|
|
- bstractive-based summarization method
|
|
ls-type:: annotation
|
|
hl-page:: 3
|
|
hl-color:: green
|
|
id:: 65aa8510-fddc-4706-8ad0-e06d6960226b
|
|
- We introduce APIDocBooster, an extract-then-abstract framework that seamlessly fuses the advantages of both extractive (i.e., enabling faithful summaries without length limitation) and abstractive summarization(i.e., producing coherent and concise summaries)
|
|
ls-type:: annotation
|
|
hl-page:: 3
|
|
hl-color:: purple
|
|
id:: 65aa8533-9ee5-477f-8f7a-096c2b45b9a7
|
|
hl-stamp:: 1705677474670
|
|
- APIDocBooster consists of two stages: (1) Context-aware Sentence Section Classification ( CSSC ) and (2) UPdate SUMmarization (UPSUM). CSSC classifies API-relevant information collected from multiple sources into API documentation sections. UPSUM first generates extractive summaries distinct from original API documentation and then generates abstractive summaries guided by extractive summaries through in-context learning
|
|
ls-type:: annotation
|
|
hl-page:: 3
|
|
hl-color:: green
|
|
id:: 65aa879b-fdac-4741-81e4-d498b37b7315
|
|
- API documentation is a set of documents indexed by the API name, where each document provides information about a specific API [33 ].
|
|
ls-type:: annotation
|
|
hl-page:: 4
|
|
hl-color:: green
|
|
id:: 65aa89b5-3490-4717-8234-52d68932e68e
|
|
- technical documentation is the most trusted resource for programming
|
|
ls-type:: annotation
|
|
hl-page:: 4
|
|
hl-color:: blue
|
|
id:: 65aa89bc-3315-442e-a483-5e7e4accc35e
|
|
- Existing API documents are often incomplete and not always equally readable [ 13, 33, 41, 44, 49].
|
|
ls-type:: annotation
|
|
hl-page:: 4
|
|
hl-color:: green
|
|
id:: 65aa89c9-4913-4dc0-ab17-46ef9e376e7d
|
|
- 60% of the participants have suffered from inadequate API documentation in the last three months
|
|
ls-type:: annotation
|
|
hl-page:: 4
|
|
hl-color:: green
|
|
id:: 65aa89d6-5809-4d2d-8068-8996864e0a8b
|
|
- solutions to augment API documentation [
|
|
ls-type:: annotation
|
|
hl-page:: 4
|
|
hl-color:: green
|
|
id:: 65aa89de-820c-484e-9faa-e7499d4aacca
|
|
- update extractive summarization
|
|
ls-type:: annotation
|
|
hl-page:: 4
|
|
hl-color:: green
|
|
id:: 65aa89e6-79df-4e82-b742-60e9074e3e4f
|
|
- abstractive summarization has garnered significant attention
|
|
ls-type:: annotation
|
|
hl-page:: 4
|
|
hl-color:: green
|
|
id:: 65aa927a-2da9-49dd-9126-23f7e75d9b39
|
|
- Abstractive summarization generates new sentences, in contrast to extractive summarization which selects sentences from input.
|
|
ls-type:: annotation
|
|
hl-page:: 4
|
|
hl-color:: green
|
|
id:: 65aa9289-84d9-4c4f-83b9-a143b338e35a
|
|
- However, its effectiveness on API documentation augmentation remains unknown
|
|
ls-type:: annotation
|
|
hl-page:: 4
|
|
hl-color:: blue
|
|
id:: 65aa929d-35cb-4520-89dd-df6e884d30ef
|
|
- GPT-4 can generate coherent and concise summaries to augment API documentation
|
|
ls-type:: annotation
|
|
hl-page:: 4
|
|
hl-color:: green
|
|
id:: 65aa92bb-6fae-4b79-a18c-585443411637
|
|
- drawbacks concerning informativeness and faithfulness
|
|
ls-type:: annotation
|
|
hl-page:: 4
|
|
hl-color:: purple
|
|
id:: 65aa92c4-1e2c-4a4b-b7cb-01d2bdc00ed7
|
|
- input length limitation of GPT-4 leads to information loss due to the truncation of the prompt
|
|
ls-type:: annotation
|
|
hl-page:: 4
|
|
hl-color:: purple
|
|
id:: 65aa92cf-1254-4b7c-902c-fd0c4e8775c3
|
|
- GPT-4 API may be a burden for individual developers generating documentation (e.g., each GPT-4 API call costs around $0.3 if the token number reaches the 8k limit
|
|
ls-type:: annotation
|
|
hl-page:: 4
|
|
hl-color:: purple
|
|
id:: 65aa92eb-e333-4add-a6c6-92712f4ee4bb
|
|
- summaries that do not accurately reflect the original meaning of API-relevant resources
|
|
ls-type:: annotation
|
|
hl-page:: 4
|
|
hl-color:: purple
|
|
id:: 65aa931d-c9a5-49e8-8dba-19c5efa6847f
|
|
- hallucination
|
|
ls-type:: annotation
|
|
hl-page:: 4
|
|
hl-color:: purple
|
|
id:: 65aa9320-d75d-45fd-b56a-7c1adee40cb9
|
|
- assessing the faithfulness of such hallucinations may require substantial time or, in some cases, might not even be feasible
|
|
ls-type:: annotation
|
|
hl-page:: 4
|
|
hl-color:: blue
|
|
id:: 65aa9334-25a8-4aca-8e00-e61423b795ae
|
|
- existence of hallucinations also hinders the adoption of GPT-4 in this task
|
|
ls-type:: annotation
|
|
hl-page:: 4
|
|
hl-color:: purple
|
|
id:: 65aa9340-8b41-4d03-8ab0-cf705cfe3a74
|
|
- Besides, extractive summaries demonstrate a considerable degree of informativeness and relevance, since there is no limitation on input length
|
|
ls-type:: annotation
|
|
hl-page:: 5
|
|
hl-color:: green
|
|
id:: 65aa9372-bd16-48f3-8162-a958f1ceb43e
|
|
hl-stamp:: 1705677684890
|
|
- The success of extractive summaries in terms of informativeness and faithfulness inspires us to use extractive summarization to guide abstractive summarization, addressing the drawbacks of the latter.
|
|
ls-type:: annotation
|
|
hl-page:: 5
|
|
hl-color:: purple
|
|
id:: 65aa9399-d50a-43aa-ac24-319efdebdb11
|
|
- We leverage an extractive-then-abstractive pipeline including two phases: 1) Extract Phase: extract insight sentences from external resources to form extractive summaries, and 2) Abstract Phase: ask GPT-4 to generate abstractive summaries guided by extractive summaries.
|
|
ls-type:: annotation
|
|
hl-page:: 5
|
|
hl-color:: purple
|
|
id:: 65aa93cf-854a-4d0c-9d3d-6150dcf45874
|
|
- Extract Phase allows input without length limitations,
|
|
ls-type:: annotation
|
|
hl-page:: 5
|
|
hl-color:: yellow
|
|
id:: 65aa93f9-c387-4b64-8513-7bcccf3aa3da
|
|
- he Abstract Phase ensures that abstractive summaries are aligned with extractive summaries, thereby enhancing faithfulness and facilitating data provenance.
|
|
ls-type:: annotation
|
|
hl-page:: 5
|
|
hl-color:: yellow
|
|
id:: 65aa9490-1b18-4f22-88ab-757a93a34762
|
|
- API documentation augmentation requires generated summaries to be distinct from the original API documentation for readability [45]. However, existing approaches neglect to reduce redundancy between generated summaries and API documentation
|
|
ls-type:: annotation
|
|
hl-page:: 5
|
|
hl-color:: green
|
|
id:: 65aa9754-05bf-4be0-a365-90070a5e32fe
|
|
- CSSC takes as input documents relevant to a specific API from multiple sources. CSSC identifies insight sentences and classifies them into suitable API documentation sections for structure awareness while considering the contextual dependency of sentences if necessary.
|
|
ls-type:: annotation
|
|
hl-page:: 5
|
|
hl-color:: green
|
|
id:: 65ae6847-49d9-4c16-be65-dac6db622a6d
|
|
- extract-then-abstract pipeline,
|
|
ls-type:: annotation
|
|
hl-page:: 5
|
|
hl-color:: blue
|
|
id:: 65ae6859-79ce-4ef5-90c4-b5aaeab82a70
|
|
- APISumBench
|
|
ls-type:: annotation
|
|
hl-page:: 5
|
|
hl-color:: blue
|
|
id:: 65ae68a1-fd3d-4424-8750-7d6ad1cf84df
|
|
- ROUGE-1, ROUGE-2, and ROUGE-
|
|
ls-type:: annotation
|
|
hl-page:: 6
|
|
hl-color:: yellow
|
|
id:: 65ae68c0-c2e0-438d-aa13-a2e3af5020e3
|
|
- We conduct the first empirical study on GPT-4 performing abstractive summarization for API documentation augmentation.
|
|
ls-type:: annotation
|
|
hl-page:: 6
|
|
hl-color:: green
|
|
id:: 65ae6920-5f7d-4bdd-9b9e-cc2ac464ad62
|
|
- extractive-based summarization approaches.
|
|
ls-type:: annotation
|
|
hl-page:: 6
|
|
hl-color:: green
|
|
id:: 65ae692b-9085-4274-af75-35e314ce384b
|
|
- We propose APIDocBooster, a two-stage approach with an extract-then-abstract framework to address these challenges.
|
|
ls-type:: annotation
|
|
hl-page:: 6
|
|
hl-color:: purple
|
|
id:: 65ae6939-d56b-459a-a8db-7459ee83abb7
|
|
- [:span]
|
|
ls-type:: annotation
|
|
hl-page:: 7
|
|
hl-color:: green
|
|
id:: 65ae696d-687c-45d7-8895-406903895f7a
|
|
hl-type:: area
|
|
hl-stamp:: 1705929068273
|
|
- inadequate API documentation.
|
|
ls-type:: annotation
|
|
hl-page:: 6
|
|
hl-color:: green
|
|
id:: 65ae69b8-444c-40ed-bdd3-8519f46af506
|
|
- brief definition of the algorithm used in Tanh as well as the math formula.
|
|
ls-type:: annotation
|
|
hl-page:: 6
|
|
hl-color:: green
|
|
id:: 65ae69c7-0ea1-4239-9cfc-9f89fe78fefb
|
|
- usage scenarios and range of the output for Tanh API,
|
|
ls-type:: annotation
|
|
hl-page:: 6
|
|
hl-color:: green
|
|
id:: 65ae69f3-d8d0-4251-b9a3-484eed480f6b
|
|
- highlight an essential property of Tanh.
|
|
ls-type:: annotation
|
|
hl-page:: 6
|
|
hl-color:: yellow
|
|
id:: 65ae6a02-61ce-4073-80b9-a8aab3e06777
|
|
- tutorial videos related to the target API often cover detailed operation steps of using the target API.
|
|
ls-type:: annotation
|
|
hl-page:: 6
|
|
hl-color:: green
|
|
id:: 65ae6c7b-f3ff-4845-b907-5e237df7343c
|
|
hl-stamp:: 1705929856901
|
|
- These observations inspire us to believe that the information contained in both YouTube videos and Stack Overflow posts has great potential to augment API documentation and complement each other
|
|
ls-type:: annotation
|
|
hl-page:: 6
|
|
hl-color:: blue
|
|
id:: 65ae6c97-7b29-4f06-95ef-527ae81e3da6
|
|
- Presenting either of these two sentences individually proves insufficient in conveying comprehensive insights concerning the target API.
|
|
ls-type:: annotation
|
|
hl-page:: 7
|
|
hl-color:: green
|
|
id:: 65ae6cde-fb8e-438d-bc84-6b38446fc531
|
|
- This motivates us to incorporate contextual information when performing API documentation augmentation
|
|
ls-type:: annotation
|
|
hl-page:: 7
|
|
hl-color:: blue
|
|
id:: 65ae6cf5-57ab-42f1-82fd-5f27f9689916
|
|
- Task Formulation
|
|
ls-type:: annotation
|
|
hl-page:: 7
|
|
hl-color:: green
|
|
id:: 65ae7951-6354-4455-a7f8-e19d72e90148
|
|
- The reason for this coarse categorization is to better fit the structure of real-world API documentation.
|
|
ls-type:: annotation
|
|
hl-page:: 7
|
|
hl-color:: green
|
|
id:: 65ae79ef-7789-41c3-86f4-f2bcaa91ea9d
|
|
- augment API documentation by generating complementary summaries for each documentation section from multiple online sources
|
|
ls-type:: annotation
|
|
hl-page:: 7
|
|
hl-color:: blue
|
|
id:: 65aeed8a-84ae-4721-ab11-bdb878b0001b
|
|
- As a initial attempt, we consider two sources, i.e., Stack Overflow and YouTube, as input sources
|
|
ls-type:: annotation
|
|
hl-page:: 7
|
|
hl-color:: blue
|
|
id:: 65aeed92-4606-4c33-8a66-d0cdbc5e0336
|
|
- , Functionality, Parameter, and Notes
|
|
ls-type:: annotation
|
|
hl-page:: 7
|
|
hl-color:: green
|
|
id:: 65aeedac-a438-48cb-a163-2ab81581a438
|
|
- API documentation augmentation task
|
|
ls-type:: annotation
|
|
hl-page:: 8
|
|
hl-color:: green
|
|
id:: 65aeee01-7133-4466-bb61-db19075054bd
|
|
- We aim to generate extractive summaries (i.e., an optimal subset 𝑠𝑢𝑚 =𝐸𝑥𝑡𝑟𝑎𝑐𝑡 (𝑠 |𝑎𝑝𝑖, 𝑑𝑜𝑐), 𝑠𝑢𝑚 ⊆ s), and then abstractive summaries (i.e., 𝑠𝑢𝑚′= 𝐴𝑏𝑠𝑡𝑟𝑎𝑐𝑡 (𝑠𝑢𝑚|𝑎𝑝𝑖, 𝑑𝑜𝑐)) that satisfy the requirements of readability, relevance of content, and structure of API documentation.
|
|
ls-type:: annotation
|
|
hl-page:: 8
|
|
hl-color:: purple
|
|
id:: 65aeee6e-4c01-4514-bbe7-6e07f86d3540
|
|
- We carry out an empirical study to understand the capacity and limitations of using GPT-4 to augment API documentation.
|
|
ls-type:: annotation
|
|
hl-page:: 8
|
|
hl-color:: blue
|
|
id:: 65b53353-51f4-4e98-a213-a533458e4d31
|
|
- API-relevant information exists in external sources
|
|
ls-type:: annotation
|
|
hl-page:: 8
|
|
hl-color:: green
|
|
id:: 65b533df-0086-456d-890e-14a4c323f4b4
|
|
- e randomly sample 100 APIs from each library, respectively. As discussed in Section 2.2, we consider Stack Overflow and YouTube as preliminary external sources.
|
|
ls-type:: annotation
|
|
hl-page:: 8
|
|
hl-color:: yellow
|
|
id:: 65b533fb-538b-4ae9-8219-3c5b9eeed815
|
|
- We set the parameter temperature of GPT-4 as 0 to ensure GPT-4 produces a deterministic response.
|
|
ls-type:: annotation
|
|
hl-page:: 8
|
|
hl-color:: blue
|
|
id:: 65b540ac-cc33-4f4d-8db8-e6015cb956af
|
|
hl-stamp:: 1706377390776
|
|
- Zero-shot Setting
|
|
ls-type:: annotation
|
|
hl-page:: 8
|
|
hl-color:: green
|
|
id:: 65b54362-3daa-430e-9609-94680bd22e48
|
|
- Summarization Setting
|
|
ls-type:: annotation
|
|
hl-page:: 8
|
|
hl-color:: green
|
|
id:: 65b54367-239a-45dd-bb13-5a5b68e1f796
|
|
- read all augmented API documentation pages and rate each section on a 5-point Likert scale in terms of informativeness, relevance, readability, non-redundancy, and faithfulness.
|
|
ls-type:: annotation
|
|
hl-page:: 9
|
|
hl-color:: green
|
|
id:: 65b54429-9992-4225-a903-00fe589a7035
|
|
- Participants are also required to highlight sentences that they think should not be in API documentation given the above criteria
|
|
ls-type:: annotation
|
|
hl-page:: 9
|
|
hl-color:: green
|
|
id:: 65b5444f-607e-4678-97cc-764316f13091
|
|
- faithfulness is a measure of the degree to which information in summaries aligns with corresponding information in external resources.
|
|
ls-type:: annotation
|
|
hl-page:: 9
|
|
hl-color:: green
|
|
id:: 65b54463-a95b-4fbb-84ec-e163051f7db2
|
|
- However, it is impossible to assess the faithfulness of 𝐺𝑃𝑇𝑧𝑒𝑟𝑜 as GPT-4 generates the summaries without external resources as reference
|
|
ls-type:: annotation
|
|
hl-page:: 9
|
|
hl-color:: purple
|
|
id:: 65b54468-1a61-48ec-8f17-2277ae00d682
|
|
- APIDocBooster simplifies the tedious task of documentation maintenance [12 ] into an automatic pipeline: 1) automatically collecting API-related information from external sources; 2) automatically generating summaries for each section of the API documentation; and 3) integrating the generated summaries into the original API documentation
|
|
ls-type:: annotation
|
|
hl-page:: 10
|
|
hl-color:: green
|
|
id:: 65b544e3-9d23-44cb-b219-be2d5522a526
|
|
- context identifier
|
|
ls-type:: annotation
|
|
hl-page:: 11
|
|
hl-color:: green
|
|
id:: 65b56de3-0110-4ca8-9d22-5e409a4fa4e0
|
|
- entence section classifie
|
|
ls-type:: annotation
|
|
hl-page:: 11
|
|
hl-color:: green
|
|
id:: 65b56de9-5e51-497a-b691-7ec119f6f05d
|
|
- we generate extractive summaries for each section by using our proposed update summarization algorithm.
|
|
ls-type:: annotation
|
|
hl-page:: 11
|
|
hl-color:: green
|
|
id:: 65b56df8-bb7a-4cb8-bdad-80fb2b1f72f7
|
|
- RQ1: How effective is CSSC for sentence section classification?
|
|
ls-type:: annotation
|
|
hl-page:: 19
|
|
hl-color:: green
|
|
id:: 65b56e38-2696-41a4-bc44-4c7bfdbf3bab
|
|
- RQ2: How effective is ExtUP for update extractive summarization?
|
|
ls-type:: annotation
|
|
hl-page:: 19
|
|
hl-color:: green
|
|
id:: 65b56e3d-7686-454d-9653-e2fa0617bd92
|
|
- 4 EXPERIMENT SETTING
|
|
ls-type:: annotation
|
|
hl-page:: 16
|
|
hl-color:: yellow
|
|
id:: 65b56e8b-ee70-46cc-96a8-f86e4f380c17
|
|
- 2.3 Empirical Study on GPT-4
|
|
ls-type:: annotation
|
|
hl-page:: 8
|
|
hl-color:: yellow
|
|
id:: 65b56eb1-cd47-44c1-b734-64d180583793 |