246 lines
11 KiB
Markdown
246 lines
11 KiB
Markdown
file:: [mastropaolo-proposal_1674656747747_0.pdf](../assets/mastropaolo-proposal_1674656747747_0.pdf)
|
||
file-path:: ../assets/mastropaolo-proposal_1674656747747_0.pdf
|
||
|
||
- software tools that can assist developers with a wide range of activities, from reusing code to writing effective bug reports
|
||
ls-type:: annotation
|
||
hl-page:: 7
|
||
hl-color:: green
|
||
id:: 63d5068a-ae7b-451a-ac16-ee3a487e286e
|
||
- JSummarizer
|
||
ls-type:: annotation
|
||
hl-page:: 7
|
||
hl-color:: green
|
||
id:: 63d5069c-11d3-42b3-b38d-0d379454913e
|
||
- data-driven recommenders that can learn how to automate code-related tasks by “looking” at activities performed by real developers in open source project
|
||
ls-type:: annotation
|
||
hl-page:: 7
|
||
hl-color:: green
|
||
id:: 63d506b9-ca0f-4af1-a58c-0dce127890cb
|
||
- Automating Code-Related Tasks via Pre-trained Models of Code
|
||
ls-type:: annotation
|
||
hl-page:: 1
|
||
hl-color:: green
|
||
id:: 63d506f9-c019-4e65-8ea7-db459286745e
|
||
- Such a sheer amount of data unlocked the usage of deep learning (DL) models to support code-related tasks, such as automatic bug-fixing [9, 10 , 11 , 12], code summarization [ 13, 14 , 15 ] and code completion[ 16, 17 , 18 , 19, 20 , 21].
|
||
ls-type:: annotation
|
||
hl-page:: 7
|
||
hl-color:: green
|
||
id:: 63d50727-0b4e-4254-8c5f-cde4d6a19465
|
||
- Despite the large-scale training, the DL model was able to fix only ∼9% of bugs in Java methods as a real developer would do, posing questions about the possibility of integrating such an approach into recommender systems that can support developers during bug-fixing activities
|
||
ls-type:: annotation
|
||
hl-page:: 7
|
||
hl-color:: green
|
||
id:: 63d6ce4b-22e5-44b8-9fc1-5b63233105f8
|
||
- Thus, despite the progress in this field, the support that state-of-the-art techniques can offer to developers is still limited, pointing to the need for more researc
|
||
ls-type:: annotation
|
||
hl-page:: 8
|
||
hl-color:: purple
|
||
id:: 63d6ce9c-eaab-4902-aeee-634b601ee12d
|
||
hl-stamp:: 1675021983379
|
||
- hese models can then be fine-tuned in a supervised way to learn how to automate specific tasks (e.g., code summarization)
|
||
ls-type:: annotation
|
||
hl-page:: 8
|
||
hl-color:: purple
|
||
id:: 63d6cf76-f25a-4cd9-9dc1-566a0986fcaa
|
||
- oncrete examples of these applications are CodeBERT [ 24], a pre-trained model for programming language and natural language, and IntelliCode Compose[16], an advanced code completion technique exploiting a multi-layer generative pretrained Transformer for code (GPT-C
|
||
ls-type:: annotation
|
||
hl-page:: 8
|
||
hl-color:: purple
|
||
id:: 63d6d0c5-cca0-408a-86d7-afef395322c3
|
||
- T5 also allows multi-task learning, meaning that a single model can be trained to learn several tasks at the same time
|
||
ls-type:: annotation
|
||
hl-page:: 8
|
||
hl-color:: purple
|
||
id:: 63d6d13a-b8aa-4d1a-a174-05bc483f6c8f
|
||
- The idea of using the “pretrain-then-finetune” paradigm to support software engineering(SE) tasks has been firstly proposed by Robbes et al. [ 22], which suggested it as a way to overcome the limited size of training datasets available for specific tasks (e.g., sentiment analysis on software-related corpora such as Stack Overflow discussions
|
||
ls-type:: annotation
|
||
hl-page:: 8
|
||
hl-color:: blue
|
||
id:: 63d7c598-cde8-4f6d-8363-7a6307124e9e
|
||
hl-stamp:: 1675085210794
|
||
- Studying the Usage of Text-To-Text Transfer Transformer to Support Code-Related Tasks
|
||
ls-type:: annotation
|
||
hl-page:: 8
|
||
hl-color:: green
|
||
id:: 63d7cca6-842f-4792-80aa-3428b6ec17cc
|
||
hl-stamp:: 1675087021555
|
||
- we did not investigate the role played by pre-training the model and then fine-tuning it.
|
||
ls-type:: annotation
|
||
hl-page:: 8
|
||
hl-color:: yellow
|
||
id:: 63d7cce0-29be-4e63-bd6b-6e6364f87a1f
|
||
hl-stamp:: 1675087074878
|
||
- Using Transfer-Learning for Code-Related Tasks
|
||
ls-type:: annotation
|
||
hl-page:: 9
|
||
hl-color:: green
|
||
id:: 63d7cd04-93f5-4553-994a-58490e955448
|
||
hl-stamp:: 1675087111071
|
||
- pretraining phase
|
||
ls-type:: annotation
|
||
hl-page:: 9
|
||
hl-color:: yellow
|
||
id:: 63d7cd11-e1df-4a63-b0da-e4e7a799c6ff
|
||
hl-stamp:: 1675087159095
|
||
- (i) code snippet summarization
|
||
ls-type:: annotation
|
||
hl-page:: 9
|
||
hl-color:: blue
|
||
id:: 63d7cdba-9792-42ec-9bc0-5709b66b1c89
|
||
hl-stamp:: 1675087292358
|
||
- (ii) generation and injection of complete log statements.
|
||
ls-type:: annotation
|
||
hl-page:: 9
|
||
hl-color:: blue
|
||
id:: 63d7cdc0-9ef0-4b28-98cd-827fe7b34f01
|
||
- ∼10% correct prediction
|
||
ls-type:: annotation
|
||
hl-page:: 9
|
||
hl-color:: blue
|
||
id:: 63d7cde1-4383-4bcf-a4a9-8c76a2756241
|
||
- pretrained T5
|
||
ls-type:: annotation
|
||
hl-page:: 9
|
||
hl-color:: yellow
|
||
id:: 63d7cde9-fa31-45d3-9661-120c582348ed
|
||
hl-stamp:: 1675087346898
|
||
- we moved from the automated generation of complete summaries, to code comment comple
|
||
ls-type:: annotation
|
||
hl-page:: 9
|
||
hl-color:: yellow
|
||
id:: 63d7cefc-62c7-46a5-9f9b-04c2291a2aca
|
||
- An Empirical Study on Code Comment Completion
|
||
ls-type:: annotation
|
||
hl-page:: 9
|
||
hl-color:: yellow
|
||
id:: 63d7cf4a-4303-432b-b8e0-e495a4e73ba2
|
||
- 3 1.1 Expected Contributions and State of the Research In other words, we were not aware of the benefits (if any) brought by pre-training the model, since we did not compare a pre-trained T5 with a non pre-trained T5 (i.e., only fine-tuning). Second, we did not study the impact on performances of mixing tasks (i.e., multi-task) when specializing the model on the set of code-related tasks we aimed at supporting. For such reasons, we extended our ICSE 2021 work by investigating these two specific aspects. We reported the results of this second investigation in the following paper [2]: Using Transfer-Learning for Code-Related Tasks Antonio Mastropaolo , Nathan Cooper, David Nader-Palacio, Simone Scalabrino, Denys Poshyvanyk, Rocco Oliveto, Gabriele Bavota. In IEEE Transactions on Software Engineering (TSE2022), to appear. Our findings showed the boost in performance provided by the pretraining phase, while did not highlight a major role of the multi-task learning. Once established the effectiveness of the “pretrain-then-finetune” paradigm in dealing with code-related tasks, we decided to explicitly focus on two tasks which are characterized by a mix of code and natural language and for which support in the literature is lacking: (i) code snippet summarization (i.e., the ability to describe in natural language a snippet of code), and (ii) generation and injection of complete log statements. As for the former (i.e., code snippet summarization), the results of our ICSE 2021 paper [ 1 ] showed that even the pretrained T5 struggles in generating summaries for entire Java methods(∼10% correct predictions). Thus, as a first step, we studied whether by simplifying the problem it was possible to achieve better results. In particular, we moved from the automated generation of complete summaries, to code comment completion, in which the
|
||
ls-type:: annotation
|
||
hl-page:: 9
|
||
hl-color:: yellow
|
||
id:: 63d7cf6c-e160-4d69-8b5d-ef64f2e819f4
|
||
- (∼16% correct predictions)
|
||
ls-type:: annotation
|
||
hl-page:: 9
|
||
hl-color:: yellow
|
||
id:: 63d7cfbb-41c8-42c5-b6e3-88394be725a3
|
||
- A first challenge to overcome here is the building of the training datase
|
||
ls-type:: annotation
|
||
hl-page:: 9
|
||
hl-color:: yellow
|
||
id:: 63d7cfd1-3aca-4719-b991-ca8b266613a5
|
||
- <method, javadoc>
|
||
ls-type:: annotation
|
||
hl-page:: 9
|
||
hl-color:: yellow
|
||
id:: 63d7cfd8-3ec7-4c7e-8c5b-617d7f298916
|
||
- As of Today, our dataset counts over 11k pairs and will be soon made available to the research community
|
||
ls-type:: annotation
|
||
hl-page:: 9
|
||
hl-color:: green
|
||
id:: 63d7cff1-e728-48b0-ad6f-7cc2a2d0a4eb
|
||
hl-stamp:: 1675087859361
|
||
- LANCE (Log stAtemeNt reCommEnder
|
||
ls-type:: annotation
|
||
hl-page:: 10
|
||
hl-color:: green
|
||
id:: 63d7d2a3-4a2a-4680-85e3-170a319a42fb
|
||
- Using Deep Learning to Generate Complete Log Statements
|
||
ls-type:: annotation
|
||
hl-page:: 10
|
||
hl-color:: green
|
||
id:: 63d7d2bc-e0a4-4c8b-bb12-b7655d3a8e07
|
||
- LANCE in order to decide a proper number of statements which may be needed (0 to n)
|
||
ls-type:: annotation
|
||
hl-page:: 10
|
||
hl-color:: green
|
||
id:: 63d7d30a-2499-4c9b-97fa-9e6177b83af1
|
||
- ata-driven techniques” can support the task of automated variable renaming in the context of Java methods
|
||
ls-type:: annotation
|
||
hl-page:: 10
|
||
hl-color:: green
|
||
id:: 63d7d31a-073a-4071-90ec-c44124e40df4
|
||
- extractive
|
||
ls-type:: annotation
|
||
hl-page:: 12
|
||
hl-color:: green
|
||
id:: 63d7d524-b64f-4660-b5e4-45850686bc81
|
||
- abstractive
|
||
ls-type:: annotation
|
||
hl-page:: 12
|
||
hl-color:: green
|
||
id:: 63d7d526-e4b8-4e8f-92ff-028494d0f906
|
||
- (i) techniques aimed at documenting code snippets
|
||
ls-type:: annotation
|
||
hl-page:: 12
|
||
hl-color:: green
|
||
id:: 63d7d59c-004b-4a82-9256-358c87cf4918
|
||
- (ii) DL-based approaches
|
||
ls-type:: annotation
|
||
hl-page:: 12
|
||
hl-color:: green
|
||
id:: 63d7d59f-f962-4765-895c-cbce7fa2a707
|
||
- Code-Related Tasks
|
||
ls-type:: annotation
|
||
hl-page:: 15
|
||
hl-color:: green
|
||
id:: 63d7d936-6d72-4d36-adfb-7a6961f16c66
|
||
- bi-modal data (i.e., involving both source code and technical natural language)
|
||
ls-type:: annotation
|
||
hl-page:: 15
|
||
hl-color:: green
|
||
id:: 63da5551-7fa6-42a9-a66a-bee8cb8cf16e
|
||
- e trained model is fine-tuned on smaller and specialized datasets, each one aimed at supporting a specific task
|
||
ls-type:: annotation
|
||
hl-page:: 15
|
||
hl-color:: green
|
||
id:: 63da582e-7e3b-4874-816f-878254c27187
|
||
- Automatic bug-fixing
|
||
ls-type:: annotation
|
||
hl-page:: 15
|
||
hl-color:: purple
|
||
id:: 63da5c2a-9a57-497f-b3f3-67ab86a35147
|
||
- he abstraction process includes a mechanism to transform back the abstracted code into its raw format.
|
||
ls-type:: annotation
|
||
hl-page:: 15
|
||
hl-color:: green
|
||
id:: 63da5cde-1ecc-475e-8410-064134269db1
|
||
- The choice of the four tasks subject of our study (i.e., bug-fixing, mutants injection, asserts generation, and code summarization) is dictated by the will of experimenting with tasks that use, represent, and manipulate code in different ways. In particular, we include in our study tasks aimed at (i) transforming the input code with the goal of changing its behavior (bug-fixing and mutants injection); (ii) “comprehending code” to verify its behavior(asserts generation); and (iii) “comprehending code” to summarize it in natural language(code summarization).
|
||
ls-type:: annotation
|
||
hl-page:: 16
|
||
hl-color:: green
|
||
id:: 63da5db0-1788-49d0-981f-4f4a160c0096
|
||
- Once the T5 model has been fine-tuned on all these tasks, we run it on the same test sets used in the four referenced works [ 9, 26 , 27 , 14] comparing the achieved results to those reported in the original work. Ta
|
||
ls-type:: annotation
|
||
hl-page:: 16
|
||
hl-color:: yellow
|
||
id:: 63da5def-890e-49e0-ad12-5d30f45b5a21
|
||
- [:span]
|
||
ls-type:: annotation
|
||
hl-page:: 16
|
||
hl-color:: green
|
||
id:: 63da5e2b-3eae-45d1-9aa9-2b6f752eb86a
|
||
hl-type:: area
|
||
hl-stamp:: 1675255337534
|
||
- substantial
|
||
ls-type:: annotation
|
||
hl-page:: 16
|
||
hl-color:: yellow
|
||
id:: 63da5ea9-868e-4f7f-9d5d-83d82ee015e0
|
||
- 11% vs 3% of correct predictions
|
||
ls-type:: annotation
|
||
hl-page:: 16
|
||
hl-color:: green
|
||
id:: 63da5eae-172a-4698-943f-d42195066d80
|
||
- e fine-tuned a single pre-trained T5 model in a multi-task setting on all four tasks, showing that it is able to achieve better results as compared to the four referenced baselines
|
||
ls-type:: annotation
|
||
hl-page:: 17
|
||
hl-color:: green
|
||
id:: 63da5ec6-500f-44cd-a7a0-4c9663074e28
|
||
- [:span]
|
||
ls-type:: annotation
|
||
hl-page:: 17
|
||
hl-color:: green
|
||
id:: 63da605b-9f06-4373-8aab-ffd2ea7e3a9f
|
||
hl-type:: area
|
||
hl-stamp:: 1675255897546 |