283 lines
17 KiB
Markdown
283 lines
17 KiB
Markdown
file:: [TOSEM-2023-0391_Proof_hi_1705078879964_0.pdf](../assets/TOSEM-2023-0391_Proof_hi_1705078879964_0.pdf)
|
||
file-path:: ../assets/TOSEM-2023-0391_Proof_hi_1705078879964_0.pdf
|
||
|
||
- Special Section: AI and SE
|
||
ls-type:: annotation
|
||
hl-page:: 1
|
||
hl-color:: green
|
||
id:: 65ca8920-7d42-4b90-8086-fc059671e8aa
|
||
- Learning to edit code automatically is becoming more and more feasible
|
||
ls-type:: annotation
|
||
hl-page:: 3
|
||
hl-color:: green
|
||
id:: 65cb3db9-c868-40fe-b3f9-bbc12ba7a797
|
||
- provide a natural language input of the requirement for the code change
|
||
ls-type:: annotation
|
||
hl-page:: 3
|
||
hl-color:: green
|
||
id:: 65cb3de1-4649-442b-a1ee-9ad2cfe079ec
|
||
- In this work, we propose to address the challenge of generating code changes without precise location information.
|
||
ls-type:: annotation
|
||
hl-page:: 3
|
||
hl-color:: blue
|
||
id:: 65cb3e06-c374-44a6-99f2-6df976ec202e
|
||
- a significant portion of code editing across various projects is repetitive or similar in practice, which results in decreased efficiency in software development
|
||
ls-type:: annotation
|
||
hl-page:: 4
|
||
hl-color:: green
|
||
id:: 65cb3e8c-2326-4b51-9609-e0b8a1ffed18
|
||
- researchers are motivated to devise automated approaches that can facilitate code editing by learning from historical examples
|
||
ls-type:: annotation
|
||
hl-page:: 4
|
||
hl-color:: blue
|
||
id:: 65cb3e92-a442-4361-8da8-557a5dcd9018
|
||
- Neural Machine Translation (NMT)
|
||
ls-type:: annotation
|
||
hl-page:: 4
|
||
hl-color:: green
|
||
id:: 65cb3e99-45bd-4518-a51e-faf0c9ba5fab
|
||
- NMT is a type of machine learning approach that uses the sequence-to-sequence [46] architecture to predict the target sequence based on the source sequence of words or tokens, which is commonly used to translate sentences from one language to another, or to generate answers from questions
|
||
ls-type:: annotation
|
||
hl-page:: 4
|
||
hl-color:: yellow
|
||
id:: 65cb3eb9-15d9-46a3-98b2-0e9388cabf4d
|
||
- NMT models the entire context of a sentence or even a paragraph to produce a nuanced translation
|
||
ls-type:: annotation
|
||
hl-page:: 4
|
||
hl-color:: green
|
||
id:: 65cb3ed8-b102-4e38-946d-c6f776cd90da
|
||
- The findings indicate a considerable decline in performance when the edit location remains unknown to the NMT model.
|
||
ls-type:: annotation
|
||
hl-page:: 4
|
||
hl-color:: green
|
||
id:: 65cb3f01-9da4-4b0f-85b8-c9be6fa46103
|
||
- ntegrated approach to enable precise localization and edition of source code without the knowledge of exact edit locations
|
||
ls-type:: annotation
|
||
hl-page:: 4
|
||
hl-color:: purple
|
||
id:: 65cb3f19-70ec-4b79-986c-7bd0f3cd1352
|
||
- jLED (jointly Localize and EDit), a novel supervised learning approach designed to enable the practical application of code editing without the need for edit location. jLED leverages large-scale language models to uniformly localize and edit source code.
|
||
ls-type:: annotation
|
||
hl-page:: 4
|
||
hl-color:: blue
|
||
id:: 65cb3f49-6431-4bc3-9bb0-e8e6109a25aa
|
||
hl-stamp:: 1707818827047
|
||
- comprehensive experiments to evaluate the performance changes by employing different modalities for sequence-to-sequence editing baselines.
|
||
ls-type:: annotation
|
||
hl-page:: 4
|
||
hl-color:: blue
|
||
id:: 65cb3f5d-9e7f-43ea-8531-8022900edad8
|
||
- 77,044 edited code samples from two famous github projects – Linux and Wireshark, we extensively evaluate the effectiveness of jLED to localize and edit source code.
|
||
ls-type:: annotation
|
||
hl-page:: 4
|
||
hl-color:: yellow
|
||
id:: 65cb3f6a-d458-45e3-baf7-1bc131f9a655
|
||
- Availability
|
||
ls-type:: annotation
|
||
hl-page:: 5
|
||
hl-color:: yellow
|
||
id:: 65cb3fc4-5d62-4583-94d0-39ee905acafc
|
||
- Neural Machine Translation (NMT)
|
||
ls-type:: annotation
|
||
hl-page:: 5
|
||
hl-color:: green
|
||
id:: 65d4f470-8f0f-42bd-a511-a88c3582f528
|
||
- NMT models are capable of learning and generating translations in an end-to-end manner, thereby overcoming the limitations of traditional statistical machine translation methods.
|
||
ls-type:: annotation
|
||
hl-page:: 5
|
||
hl-color:: green
|
||
id:: 65d4f481-6651-49dc-a325-531b4fdf796e
|
||
- NMT comprises two fundamental components: the encoder and the decoder, which work synergistically to facilitate the translation process
|
||
ls-type:: annotation
|
||
hl-page:: 5
|
||
hl-color:: green
|
||
id:: 65d4f4a2-e00e-46a6-b6ad-6a44f077a21d
|
||
- vector representation that encapsulates the underlying semantic meaning of the source text
|
||
ls-type:: annotation
|
||
hl-page:: 5
|
||
hl-color:: green
|
||
id:: 65d4f4c4-1fdd-49fb-9a07-640a4f1ff2f0
|
||
- subsequent translation process to leverage the encoded information effectively.
|
||
ls-type:: annotation
|
||
hl-page:: 5
|
||
hl-color:: green
|
||
id:: 65d4f4d2-c72c-49ec-91ea-aae18827afe6
|
||
- These applications capitalize on NMT’s ability to comprehend and generate intricate patterns, making it a valuable tool for SE-related tasks
|
||
ls-type:: annotation
|
||
hl-page:: 5
|
||
hl-color:: blue
|
||
id:: 65d4f5f2-cf09-40f5-8205-aac52b095343
|
||
- While RNNs showcased commendable performance, they encountered challenges in parallelization and capturing longrange dependencies adequatel
|
||
ls-type:: annotation
|
||
hl-page:: 5
|
||
hl-color:: green
|
||
id:: 65d6037e-1661-48cb-8ae2-90c3d40b386c
|
||
- self-attention mechanism, enabling selective attention to different parts of the input sequence through adaptive weighting.
|
||
ls-type:: annotation
|
||
hl-page:: 5
|
||
hl-color:: green
|
||
id:: 65d6040a-879c-444b-9921-14fd663541f9
|
||
- transfer learning involves the creation of task-agnostic representations of source code, which can be leveraged and repurposed across different tasks
|
||
ls-type:: annotation
|
||
hl-page:: 6
|
||
hl-color:: green
|
||
id:: 65d604e5-16a8-48de-a5e8-8f32cff830bd
|
||
- CodeBERT [11] focuses on learning continuous representations of code snippets, enabling a deeper understanding of their structural and contextual aspects
|
||
ls-type:: annotation
|
||
hl-page:: 6
|
||
hl-color:: green
|
||
id:: 65d605db-64a6-42a7-9815-260a14cf0cee
|
||
- GraphCodeBERT [14] incorporates graph neural networks to capture code dependencies and interactions, facilitating higher-level reasoning and tasks such as code completion and refactoring
|
||
ls-type:: annotation
|
||
hl-page:: 6
|
||
hl-color:: green
|
||
id:: 65d60619-e338-4c88-b239-5dc3ce62f46a
|
||
- CodeGPT
|
||
ls-type:: annotation
|
||
hl-page:: 6
|
||
hl-color:: green
|
||
id:: 65d6061e-2193-4257-a70a-557922ac7fd9
|
||
- pre-processing
|
||
ls-type:: annotation
|
||
hl-page:: 6
|
||
hl-color:: green
|
||
id:: 65d606fb-41d6-4287-a0b6-1570ec1f7a93
|
||
- model architecture
|
||
ls-type:: annotation
|
||
hl-page:: 6
|
||
hl-color:: green
|
||
id:: 65d606ff-b0df-4e7e-9067-5a6a6e87e3d9
|
||
- optimization objectiv
|
||
ls-type:: annotation
|
||
hl-page:: 6
|
||
hl-color:: green
|
||
id:: 65d60705-196f-4b8e-b720-8f58b4f3ceb5
|
||
- Input Modalities & Pre-processing
|
||
ls-type:: annotation
|
||
hl-page:: 6
|
||
hl-color:: green
|
||
id:: 65d60763-125b-4171-b6a6-8c285372d166
|
||
- We use G and C to denote the natural language guidance and the code context respectivel
|
||
ls-type:: annotation
|
||
hl-page:: 6
|
||
hl-color:: blue
|
||
id:: 65d60801-34c0-4c83-b643-04f2191bd783
|
||
- the model can only take natural language guidance and code contex
|
||
ls-type:: annotation
|
||
hl-page:: 6
|
||
hl-color:: purple
|
||
id:: 65d60824-73e2-4f62-ae64-ddbdb8c4c7cf
|
||
- encoder pretrained code models, decoder pre-trained code models, and encoder-decoder pre-trained code models
|
||
ls-type:: annotation
|
||
hl-page:: 7
|
||
hl-color:: blue
|
||
id:: 65d608c4-2e28-4293-9bcf-996655afbbda
|
||
- The basic component of the large code models is the encoder-decoder architecture, a powerful sequence-to-sequence deep learning architecture, which has been widely used in text-to-text task, text-to-code task, code-to-text, and code-to-code (our task) tasks
|
||
ls-type:: annotation
|
||
hl-page:: 7
|
||
hl-color:: purple
|
||
id:: 65d8b0a3-0a33-40e8-8eb5-96d92e1027ed
|
||
- the semantic feature space to obtain the representation of the input sequence.
|
||
ls-type:: annotation
|
||
hl-page:: 7
|
||
hl-color:: purple
|
||
id:: 65d8b0d5-243b-446a-b9e5-8ee847c432c8
|
||
- serving as the optimization target to minimize the distance (loss score) between the generated edits (sequence of output subtokens) and the ground truth edits.
|
||
ls-type:: annotation
|
||
hl-page:: 10
|
||
hl-color:: green
|
||
id:: 65d8b20f-f8e1-49ef-95e1-1f33cea587ca
|
||
- trained model will be utilized to predict edits based on the pre-processed and tokenized subtoken sequence
|
||
ls-type:: annotation
|
||
hl-page:: 10
|
||
hl-color:: green
|
||
id:: 65d8b23f-804a-48fb-b0ca-88820a6f0950
|
||
- These predicted edits will be used to calculate evaluation metrics and generate the post-editing code context.
|
||
ls-type:: annotation
|
||
hl-page:: 10
|
||
hl-color:: green
|
||
id:: 65d8b258-83bf-42b6-a039-c6b55acf639a
|
||
- before edited source code contex
|
||
ls-type:: annotation
|
||
hl-page:: 11
|
||
hl-color:: green
|
||
id:: 65da52cc-e605-4af1-853b-3d0a32dcd76e
|
||
- natural language description
|
||
ls-type:: annotation
|
||
hl-page:: 11
|
||
hl-color:: green
|
||
id:: 65da52d0-6904-496f-bbaa-02142388042f
|
||
- line-level edits location (line number)
|
||
ls-type:: annotation
|
||
hl-page:: 11
|
||
hl-color:: green
|
||
id:: 65da52dc-cd0e-47b7-a49c-f8f2843593a8
|
||
- the ground-truth edits
|
||
ls-type:: annotation
|
||
hl-page:: 11
|
||
hl-color:: green
|
||
id:: 65da52df-85c6-48e7-84f4-06c0a3689106
|
||
- r training, evaluation, and testing,
|
||
ls-type:: annotation
|
||
hl-page:: 11
|
||
hl-color:: green
|
||
id:: 65da54f4-57b8-471a-aeb4-e58f7db25edb
|
||
- For each sample in the dataset, it contains (1) a source code snippet with several lines as the code context, (2) a natural language description of the editing purpose as the natural language guidance, (3) a line number as the editing location, and (4) the ground-truth edits.
|
||
ls-type:: annotation
|
||
hl-page:: 11
|
||
hl-color:: purple
|
||
id:: 65da5529-b096-4969-934e-d6b27251ef93
|
||
hl-stamp:: 1708807467759
|
||
- Table 1. Statistic of the dataset
|
||
ls-type:: annotation
|
||
hl-page:: 11
|
||
hl-color:: red
|
||
id:: 65da554f-f96c-49f4-8edf-b82ac14acf87
|
||
hl-stamp:: 1708807504618
|
||
- we follow the pre-processing method described in Section 3.1 to pre-process each sample in the dataset by concatenating the source code context C and the associated natural language guidance G as an input data sample
|
||
ls-type:: annotation
|
||
hl-page:: 11
|
||
hl-color:: green
|
||
id:: 65da5592-4d36-4097-b5dd-04ce3755154b
|
||
hl-stamp:: 1708807573240
|
||
- [C <s> G] where <s> denotes the special token for splitting different modalities, and apply the tokenizer to tokenize each input data sample to generate the input subtoken sequence 𝑿.
|
||
ls-type:: annotation
|
||
hl-page:: 11
|
||
hl-color:: yellow
|
||
id:: 65da55ba-082a-4d48-9e25-011b08c7c432
|
||
hl-stamp:: 1708807612561
|
||
- CodeBERT [11], GraphCodeBERT [14], CodeGPT [29], PLBART [1], and CodeT5 [59]
|
||
ls-type:: annotation
|
||
hl-page:: 14
|
||
hl-color:: green
|
||
id:: 65dc6502-513e-49b5-b0bf-e1380f7efbfd
|
||
- the editing baseline and localization baseline respectively.
|
||
ls-type:: annotation
|
||
hl-page:: 14
|
||
hl-color:: green
|
||
id:: 65dc66e4-ad4d-44ea-a589-c57aba355416
|
||
- wo-stage localizationediting pipeline?
|
||
ls-type:: annotation
|
||
hl-page:: 16
|
||
hl-color:: green
|
||
id:: 65dc67cb-744e-40e4-a3a1-fca4127507b1
|
||
- our joint training pipeline does not totally rely on the predicted localization results to generate the edits, so that the models could pay some attention to the correct code lines, even they are not with the highest probability scores. Based these experimental results, we conclude that our joint training pipeline is a better choice compared to the two-stage method.
|
||
ls-type:: annotation
|
||
hl-page:: 18
|
||
hl-color:: green
|
||
id:: 65dc67f5-c15d-42ee-a136-07c30968f5c0
|
||
- 6.2 Limitations Our approach mainly focuses on the single line source code editing. However, when using the binary cross-entropy loss to replace the current standard cross-entropy loss, the localization parts of our approach can also handle multi-line source code editing by setting a threshold to filter multiple results as the predicted lines
|
||
ls-type:: annotation
|
||
hl-page:: 18
|
||
hl-color:: yellow
|
||
id:: 65dc684d-d921-415e-8efc-6ff6c7d64509
|
||
- In this paper, we investigate the performance of existing standard sequence-to-sequence multimodal learning for code editing under a more practical situation, in which the precise line-level location information is unknown or unavailable. Through comprehensive experiments, we have confirmed that the models are not able to generate precise edits after training without exact line-level location information. To tackle this challenge, we proposed jLED (jointly Localize and EDit), a training pipeline to jointly learn to localize the edit buggy code simultaneously, which enables models to learn additional knowledge and abilities regarding to the line-level localization while learning to edit. We conduct experiments using our proposed jLED, and the experimental results show that our approach not only generates more precise edits, but also predicts more accurate line-level locations than the considered baselines. Moreover, to evaluate the effectiveness of our joint learning pipeline against other localization and editing alternatives, we construct a two-stage localization-editing pipeline, in which a localization model and a editing model are trained separately. Experimental results demonstrate the superiority of our jLED over this two-stage manner
|
||
ls-type:: annotation
|
||
hl-page:: 20
|
||
hl-color:: yellow
|
||
id:: 65dcb1eb-d94a-4ddc-848f-d544d17db4ff
|
||
- https://anonymous.4open. science/r/Code_Edit_Joint_Learning-1F8D. The remainder of this paper is presented as follows. Section 2 introduces the background of this work. In Section 3, we present our methodology with detailed explanations. Section 4 and 5 cover the experimental design and results. We provide discussions and related work in Section 6 and 7. Section 8 concludes this work.2 BACKGROUND2.1 Neural Machine Translation Neural Machine Translation (NMT) has emerged as a promising approach in the field of machine translation, exhibiting well performance in automating language translation tasks [6 ]. By leveraging deep neural networks, NMT models are capable of learning and generating translations in an end-to-end manner, thereby overcoming the limitations of traditional statistical machine translation methods. At its core, NMT comprises two fundamental components: the encoder and the decoder, which work synergistically to facilitate the translation process. The encoder component plays a crucial role in comprehending and processing the input sentence, utilizing sophisticated neural architectures to generate a vector representation that encapsulates the underlying semantic meaning of the source text. This vector representation serves as a rich and comprehensive representation of the source sentence, enabling the subsequent translation process to leverage the encoded information effectively. The decoder component, on the other hand, capitalizes on the encoded input representation to sequentially generate the target sentence through a process of logical reasoning. In recent years, the field of Software Engineering (SE) has witnessed a broad range of applications for NMT. Notably, NMT has found utility in areas such as Automatic Program Repair [20, 30], Program Synthesis [65], and Code Edit Generation [55, 56]. These applications capitalize on NMT’s ability to comprehend and generate intricate patterns, making it a valuable tool for SE-related tasks. The integration of NMT in software engineering research highlights its potential to enhance various aspects of software development, offering new avenues for improving program understanding, code generation, and automated repair.2.2 Transformer Model for Sequence Processing Transformer model [ 58] has emerged as a highly influential and prominent model for sequence processing tasks within the field of natural language processing (NLP), leading to numerous state-of-the-art achievements [3 , 61]. Prior to the advent of the Transformer, recurrent neural networks (RNNs) and their variants, such as long short-term memory (LSTM) and gated recurrent units (GRUs), were conventionally utilized for sequence processing tasks. While RNNs showcased commendable performance, they encountered challenges in parallelization and capturing longrange dependencies adequately. To overcome these limitations, the Transformer model introduced a novel self-attention mechanism, enabling selective attention to different parts of the input sequence through adaptive weighting. This mechanism played a pivotal role in capturing dependencies between distinct positions within the sequence, facilitating parallel processing of the input. During the token representation learning phase, the Transformer model learned to attend to all input tokens, transforming the sequence into a comprehensive graph where each token represented a node. The edge weights in this graph denoted the attention weights between tokens, which were learned based on the specific task at hand. Additionally, positional encoding was incorporated J. ACM, Vol. 1, No. 1, Article . Publication date: October 2023. Page 4 of 22Transactions on Software Engineering and Methodology1234567891011121314151617181920212223242526272829303132333435363738394041424344454647484950515253545556
|
||
ls-type:: annotation
|
||
hl-page:: 5
|
||
hl-color:: yellow
|
||
id:: 65dcb343-08d6-43a9-a0f5-ba9dfa3cdc86 |