logseq/pages/hls__TOSEM-2023-0391_Proof_hi_1705078879964_0.md

file:: TOSEM-2023-0391_Proof_hi_1705078879964_0.pdf file-path:: ../assets/TOSEM-2023-0391_Proof_hi_1705078879964_0.pdf

• Special Section: AI and SE ls-type:: annotation hl-page:: 1 hl-color:: green id:: 65ca8920-7d42-4b90-8086-fc059671e8aa
• Learning to edit code automatically is becoming more and more feasible ls-type:: annotation hl-page:: 3 hl-color:: green id:: 65cb3db9-c868-40fe-b3f9-bbc12ba7a797
• provide a natural language input of the requirement for the code change ls-type:: annotation hl-page:: 3 hl-color:: green id:: 65cb3de1-4649-442b-a1ee-9ad2cfe079ec
• In this work, we propose to address the challenge of generating code changes without precise location information. ls-type:: annotation hl-page:: 3 hl-color:: blue id:: 65cb3e06-c374-44a6-99f2-6df976ec202e
• a significant portion of code editing across various projects is repetitive or similar in practice, which results in decreased efficiency in software development ls-type:: annotation hl-page:: 4 hl-color:: green id:: 65cb3e8c-2326-4b51-9609-e0b8a1ffed18
• researchers are motivated to devise automated approaches that can facilitate code editing by learning from historical examples ls-type:: annotation hl-page:: 4 hl-color:: blue id:: 65cb3e92-a442-4361-8da8-557a5dcd9018
• Neural Machine Translation (NMT) ls-type:: annotation hl-page:: 4 hl-color:: green id:: 65cb3e99-45bd-4518-a51e-faf0c9ba5fab
• NMT is a type of machine learning approach that uses the sequence-to-sequence [46] architecture to predict the target sequence based on the source sequence of words or tokens, which is commonly used to translate sentences from one language to another, or to generate answers from questions ls-type:: annotation hl-page:: 4 hl-color:: yellow id:: 65cb3eb9-15d9-46a3-98b2-0e9388cabf4d
• NMT models the entire context of a sentence or even a paragraph to produce a nuanced translation ls-type:: annotation hl-page:: 4 hl-color:: green id:: 65cb3ed8-b102-4e38-946d-c6f776cd90da
• The findings indicate a considerable decline in performance when the edit location remains unknown to the NMT model. ls-type:: annotation hl-page:: 4 hl-color:: green id:: 65cb3f01-9da4-4b0f-85b8-c9be6fa46103
• ntegrated approach to enable precise localization and edition of source code without the knowledge of exact edit locations ls-type:: annotation hl-page:: 4 hl-color:: purple id:: 65cb3f19-70ec-4b79-986c-7bd0f3cd1352
• jLED (jointly Localize and EDit), a novel supervised learning approach designed to enable the practical application of code editing without the need for edit location. jLED leverages large-scale language models to uniformly localize and edit source code. ls-type:: annotation hl-page:: 4 hl-color:: blue id:: 65cb3f49-6431-4bc3-9bb0-e8e6109a25aa hl-stamp:: 1707818827047
• comprehensive experiments to evaluate the performance changes by employing different modalities for sequence-to-sequence editing baselines. ls-type:: annotation hl-page:: 4 hl-color:: blue id:: 65cb3f5d-9e7f-43ea-8531-8022900edad8
• 77,044 edited code samples from two famous github projects – Linux and Wireshark, we extensively evaluate the effectiveness of jLED to localize and edit source code. ls-type:: annotation hl-page:: 4 hl-color:: yellow id:: 65cb3f6a-d458-45e3-baf7-1bc131f9a655
• Availability ls-type:: annotation hl-page:: 5 hl-color:: yellow id:: 65cb3fc4-5d62-4583-94d0-39ee905acafc
• Neural Machine Translation (NMT) ls-type:: annotation hl-page:: 5 hl-color:: green id:: 65d4f470-8f0f-42bd-a511-a88c3582f528
• NMT models are capable of learning and generating translations in an end-to-end manner, thereby overcoming the limitations of traditional statistical machine translation methods. ls-type:: annotation hl-page:: 5 hl-color:: green id:: 65d4f481-6651-49dc-a325-531b4fdf796e
• NMT comprises two fundamental components: the encoder and the decoder, which work synergistically to facilitate the translation process ls-type:: annotation hl-page:: 5 hl-color:: green id:: 65d4f4a2-e00e-46a6-b6ad-6a44f077a21d
• vector representation that encapsulates the underlying semantic meaning of the source text ls-type:: annotation hl-page:: 5 hl-color:: green id:: 65d4f4c4-1fdd-49fb-9a07-640a4f1ff2f0
• subsequent translation process to leverage the encoded information effectively. ls-type:: annotation hl-page:: 5 hl-color:: green id:: 65d4f4d2-c72c-49ec-91ea-aae18827afe6
• These applications capitalize on NMT’s ability to comprehend and generate intricate patterns, making it a valuable tool for SE-related tasks ls-type:: annotation hl-page:: 5 hl-color:: blue id:: 65d4f5f2-cf09-40f5-8205-aac52b095343
• While RNNs showcased commendable performance, they encountered challenges in parallelization and capturing longrange dependencies adequatel ls-type:: annotation hl-page:: 5 hl-color:: green id:: 65d6037e-1661-48cb-8ae2-90c3d40b386c
• self-attention mechanism, enabling selective attention to different parts of the input sequence through adaptive weighting. ls-type:: annotation hl-page:: 5 hl-color:: green id:: 65d6040a-879c-444b-9921-14fd663541f9
• transfer learning involves the creation of task-agnostic representations of source code, which can be leveraged and repurposed across different tasks ls-type:: annotation hl-page:: 6 hl-color:: green id:: 65d604e5-16a8-48de-a5e8-8f32cff830bd
• CodeBERT [11] focuses on learning continuous representations of code snippets, enabling a deeper understanding of their structural and contextual aspects ls-type:: annotation hl-page:: 6 hl-color:: green id:: 65d605db-64a6-42a7-9815-260a14cf0cee
• GraphCodeBERT [14] incorporates graph neural networks to capture code dependencies and interactions, facilitating higher-level reasoning and tasks such as code completion and refactoring ls-type:: annotation hl-page:: 6 hl-color:: green id:: 65d60619-e338-4c88-b239-5dc3ce62f46a
• CodeGPT ls-type:: annotation hl-page:: 6 hl-color:: green id:: 65d6061e-2193-4257-a70a-557922ac7fd9
• pre-processing ls-type:: annotation hl-page:: 6 hl-color:: green id:: 65d606fb-41d6-4287-a0b6-1570ec1f7a93
• model architecture ls-type:: annotation hl-page:: 6 hl-color:: green id:: 65d606ff-b0df-4e7e-9067-5a6a6e87e3d9
• optimization objectiv ls-type:: annotation hl-page:: 6 hl-color:: green id:: 65d60705-196f-4b8e-b720-8f58b4f3ceb5
• Input Modalities & Pre-processing ls-type:: annotation hl-page:: 6 hl-color:: green id:: 65d60763-125b-4171-b6a6-8c285372d166
• We use G and C to denote the natural language guidance and the code context respectivel ls-type:: annotation hl-page:: 6 hl-color:: blue id:: 65d60801-34c0-4c83-b643-04f2191bd783
• the model can only take natural language guidance and code contex ls-type:: annotation hl-page:: 6 hl-color:: purple id:: 65d60824-73e2-4f62-ae64-ddbdb8c4c7cf
• encoder pretrained code models, decoder pre-trained code models, and encoder-decoder pre-trained code models ls-type:: annotation hl-page:: 7 hl-color:: blue id:: 65d608c4-2e28-4293-9bcf-996655afbbda
• The basic component of the large code models is the encoder-decoder architecture, a powerful sequence-to-sequence deep learning architecture, which has been widely used in text-to-text task, text-to-code task, code-to-text, and code-to-code (our task) tasks ls-type:: annotation hl-page:: 7 hl-color:: purple id:: 65d8b0a3-0a33-40e8-8eb5-96d92e1027ed
• the semantic feature space to obtain the representation of the input sequence. ls-type:: annotation hl-page:: 7 hl-color:: purple id:: 65d8b0d5-243b-446a-b9e5-8ee847c432c8
• serving as the optimization target to minimize the distance (loss score) between the generated edits (sequence of output subtokens) and the ground truth edits. ls-type:: annotation hl-page:: 10 hl-color:: green id:: 65d8b20f-f8e1-49ef-95e1-1f33cea587ca
• trained model will be utilized to predict edits based on the pre-processed and tokenized subtoken sequence ls-type:: annotation hl-page:: 10 hl-color:: green id:: 65d8b23f-804a-48fb-b0ca-88820a6f0950
• These predicted edits will be used to calculate evaluation metrics and generate the post-editing code context. ls-type:: annotation hl-page:: 10 hl-color:: green id:: 65d8b258-83bf-42b6-a039-c6b55acf639a
• before edited source code contex ls-type:: annotation hl-page:: 11 hl-color:: green id:: 65da52cc-e605-4af1-853b-3d0a32dcd76e
• natural language description ls-type:: annotation hl-page:: 11 hl-color:: green id:: 65da52d0-6904-496f-bbaa-02142388042f
• line-level edits location (line number) ls-type:: annotation hl-page:: 11 hl-color:: green id:: 65da52dc-cd0e-47b7-a49c-f8f2843593a8
• the ground-truth edits ls-type:: annotation hl-page:: 11 hl-color:: green id:: 65da52df-85c6-48e7-84f4-06c0a3689106
• r training, evaluation, and testing, ls-type:: annotation hl-page:: 11 hl-color:: green id:: 65da54f4-57b8-471a-aeb4-e58f7db25edb
• For each sample in the dataset, it contains (1) a source code snippet with several lines as the code context, (2) a natural language description of the editing purpose as the natural language guidance, (3) a line number as the editing location, and (4) the ground-truth edits. ls-type:: annotation hl-page:: 11 hl-color:: purple id:: 65da5529-b096-4969-934e-d6b27251ef93 hl-stamp:: 1708807467759
• Table 1. Statistic of the dataset ls-type:: annotation hl-page:: 11 hl-color:: red id:: 65da554f-f96c-49f4-8edf-b82ac14acf87 hl-stamp:: 1708807504618
• we follow the pre-processing method described in Section 3.1 to pre-process each sample in the dataset by concatenating the source code context C and the associated natural language guidance G as an input data sample ls-type:: annotation hl-page:: 11 hl-color:: green id:: 65da5592-4d36-4097-b5dd-04ce3755154b hl-stamp:: 1708807573240
• [C G] where denotes the special token for splitting different modalities, and apply the tokenizer to tokenize each input data sample to generate the input subtoken sequence 𝑿. ls-type:: annotation hl-page:: 11 hl-color:: yellow id:: 65da55ba-082a-4d48-9e25-011b08c7c432 hl-stamp:: 1708807612561
• CodeBERT [11], GraphCodeBERT [14], CodeGPT [29], PLBART [1], and CodeT5 [59] ls-type:: annotation hl-page:: 14 hl-color:: green id:: 65dc6502-513e-49b5-b0bf-e1380f7efbfd
• the editing baseline and localization baseline respectively. ls-type:: annotation hl-page:: 14 hl-color:: green id:: 65dc66e4-ad4d-44ea-a589-c57aba355416
• wo-stage localizationediting pipeline? ls-type:: annotation hl-page:: 16 hl-color:: green id:: 65dc67cb-744e-40e4-a3a1-fca4127507b1
• our joint training pipeline does not totally rely on the predicted localization results to generate the edits, so that the models could pay some attention to the correct code lines, even they are not with the highest probability scores. Based these experimental results, we conclude that our joint training pipeline is a better choice compared to the two-stage method. ls-type:: annotation hl-page:: 18 hl-color:: green id:: 65dc67f5-c15d-42ee-a136-07c30968f5c0
• 6.2 Limitations Our approach mainly focuses on the single line source code editing. However, when using the binary cross-entropy loss to replace the current standard cross-entropy loss, the localization parts of our approach can also handle multi-line source code editing by setting a threshold to filter multiple results as the predicted lines ls-type:: annotation hl-page:: 18 hl-color:: yellow id:: 65dc684d-d921-415e-8efc-6ff6c7d64509
• In this paper, we investigate the performance of existing standard sequence-to-sequence multimodal learning for code editing under a more practical situation, in which the precise line-level location information is unknown or unavailable. Through comprehensive experiments, we have confirmed that the models are not able to generate precise edits after training without exact line-level location information. To tackle this challenge, we proposed jLED (jointly Localize and EDit), a training pipeline to jointly learn to localize the edit buggy code simultaneously, which enables models to learn additional knowledge and abilities regarding to the line-level localization while learning to edit. We conduct experiments using our proposed jLED, and the experimental results show that our approach not only generates more precise edits, but also predicts more accurate line-level locations than the considered baselines. Moreover, to evaluate the effectiveness of our joint learning pipeline against other localization and editing alternatives, we construct a two-stage localization-editing pipeline, in which a localization model and a editing model are trained separately. Experimental results demonstrate the superiority of our jLED over this two-stage manner ls-type:: annotation hl-page:: 20 hl-color:: yellow id:: 65dcb1eb-d94a-4ddc-848f-d544d17db4ff
• https://anonymous.4open. science/r/Code_Edit_Joint_Learning-1F8D. The remainder of this paper is presented as follows. Section 2 introduces the background of this work. In Section 3, we present our methodology with detailed explanations. Section 4 and 5 cover the experimental design and results. We provide discussions and related work in Section 6 and 7. Section 8 concludes this work.2 BACKGROUND2.1 Neural Machine Translation Neural Machine Translation (NMT) has emerged as a promising approach in the field of machine translation, exhibiting well performance in automating language translation tasks [6 ]. By leveraging deep neural networks, NMT models are capable of learning and generating translations in an end-to-end manner, thereby overcoming the limitations of traditional statistical machine translation methods. At its core, NMT comprises two fundamental components: the encoder and the decoder, which work synergistically to facilitate the translation process. The encoder component plays a crucial role in comprehending and processing the input sentence, utilizing sophisticated neural architectures to generate a vector representation that encapsulates the underlying semantic meaning of the source text. This vector representation serves as a rich and comprehensive representation of the source sentence, enabling the subsequent translation process to leverage the encoded information effectively. The decoder component, on the other hand, capitalizes on the encoded input representation to sequentially generate the target sentence through a process of logical reasoning. In recent years, the field of Software Engineering (SE) has witnessed a broad range of applications for NMT. Notably, NMT has found utility in areas such as Automatic Program Repair [20, 30], Program Synthesis [65], and Code Edit Generation [55, 56]. These applications capitalize on NMT’s ability to comprehend and generate intricate patterns, making it a valuable tool for SE-related tasks. The integration of NMT in software engineering research highlights its potential to enhance various aspects of software development, offering new avenues for improving program understanding, code generation, and automated repair.2.2 Transformer Model for Sequence Processing Transformer model [ 58] has emerged as a highly influential and prominent model for sequence processing tasks within the field of natural language processing (NLP), leading to numerous state-of-the-art achievements [3 , 61]. Prior to the advent of the Transformer, recurrent neural networks (RNNs) and their variants, such as long short-term memory (LSTM) and gated recurrent units (GRUs), were conventionally utilized for sequence processing tasks. While RNNs showcased commendable performance, they encountered challenges in parallelization and capturing longrange dependencies adequately. To overcome these limitations, the Transformer model introduced a novel self-attention mechanism, enabling selective attention to different parts of the input sequence through adaptive weighting. This mechanism played a pivotal role in capturing dependencies between distinct positions within the sequence, facilitating parallel processing of the input. During the token representation learning phase, the Transformer model learned to attend to all input tokens, transforming the sequence into a comprehensive graph where each token represented a node. The edge weights in this graph denoted the attention weights between tokens, which were learned based on the specific task at hand. Additionally, positional encoding was incorporated J. ACM, Vol. 1, No. 1, Article . Publication date: October 2023. Page 4 of 22Transactions on Software Engineering and Methodology1234567891011121314151617181920212223242526272829303132333435363738394041424344454647484950515253545556 ls-type:: annotation hl-page:: 5 hl-color:: yellow id:: 65dcb343-08d6-43a9-a0f5-ba9dfa3cdc86