logseq/pages/hls__TOSEM-2023-0288_Proof_hi_1693841825871_0.md

file:: TOSEM-2023-0288_Proof_hi_1693841825871_0.pdf file-path:: ../assets/TOSEM-2023-0288_Proof_hi_1693841825871_0.pdf

• comment updating dataset ls-type:: annotation hl-page:: 2 hl-color:: yellow id:: 65198f61-6aab-4f71-84b8-04a3c5a51746
• High-quality data is a key factor in deep learning-based approaches ls-type:: annotation hl-page:: 2 hl-color:: green id:: 6519908b-670b-4c0f-bfa7-6cf084367f32
• Comment updating ls-type:: annotation hl-page:: 3 hl-color:: blue id:: 65199206-3022-4f58-9825-618498bc74da
• This task aims to automatically update the corresponding comments based on code changes made by developer ls-type:: annotation hl-page:: 3 hl-color:: purple id:: 65199218-a454-4b39-997a-031ea97f4c0b
• comment updating can take into account more information, such as code changes and old comment ls-type:: annotation hl-page:: 3 hl-color:: purple id:: 65199327-ba6b-4525-939a-dcc07b4652b5
• if comments are not updated in a timely manner, they may mislead future development activities. ls-type:: annotation hl-page:: 3 hl-color:: purple id:: 65199337-7123-4dc5-a6c8-23f5536673e2
• if obsolete TODO comments are not removed, they may introduce new bugs in future development [ 6]. For the task of comment updating, the data we expect is to be able to meet the real editing intention of developers and reflect the changes in code functionality ls-type:: annotation hl-page:: 3 hl-color:: green id:: 6519934c-807c-464b-a598-c43531e6dcea
• Besides, comment updating datasets are often crawled from online repositories, such as Github, by extracting commit history versions ls-type:: annotation hl-page:: 3 hl-color:: green id:: 651993fb-82cc-4432-a5eb-2bdc37dee2c0
• a noisy data sample in the comment updating task. ls-type:: annotation hl-page:: 3 hl-color:: green id:: 65199415-7d85-4913-bddf-e36fe63a8b31
• From a code change perspective, the comment modification does not reflect the changes in the code. ls-type:: annotation hl-page:: 3 hl-color:: green id:: 6522dbea-ac76-4447-bef2-71012907b112
• If the dataset contains a large number of such noisy data, the results generated by a model trained with the dataset would also be misleading ls-type:: annotation hl-page:: 3 hl-color:: blue id:: 6522dc22-f58c-401d-b6a7-3076fff17688
• Figure 2 shows an overview of our research procedur ls-type:: annotation hl-page:: 4 hl-color:: green id:: 6522e59e-31d8-4013-9b60-e47ff0ac8d58
• weak correlation between old and new comments, ls-type:: annotation hl-page:: 4 hl-color:: blue id:: 6522eb18-02cb-4003-a0d7-8581843d20f3
• weak correlation between code changes and comment changes ls-type:: annotation hl-page:: 4 hl-color:: blue id:: 6522eb21-d417-453a-9308-c169d0e02032
• CupCleaner ls-type:: annotation hl-page:: 4 hl-color:: purple id:: 6522eb2e-2901-4762-997a-a85be116406f
• The first step is to design a criterion to calculate the score of all data and map them to a distribution ls-type:: annotation hl-page:: 4 hl-color:: green id:: 6522eb52-eb79-4a98-b1ce-1320f4a48540
• we design an anchor point search technique to adapt to different datasets. We are able to adaptively trim the tail of the score distribution, i.e., filter out the noisy data. ls-type:: annotation hl-page:: 4 hl-color:: blue id:: 6522eba7-0d34-41cf-a392-8053d2061aac
• mixing the noise data and high-quality data identified by CupCleaner ls-type:: annotation hl-page:: 4 hl-color:: green id:: 6522ec49-663d-43d6-ab74-c0528507ae0c
• The experimental results show that cleaning the training and validation sets with our approach can effectively improve the performance of the models on the same test set ls-type:: annotation hl-page:: 4 hl-color:: purple id:: 65230fcc-e64a-4439-90ba-6980093950db
• the prefromance of the model trained on the cleaned dataset still improves on all performance metrics. ls-type:: annotation hl-page:: 4 hl-color:: green id:: 65230fe4-59f5-4925-8f7e-5c2b5283cd29 hl-stamp:: 1697104112774
• . We find that the evaluation performance on the cleaned test set is significantly better than that on the pure noise test set, regardless of whether the training set is cleaned or not. ls-type:: annotation hl-page:: 4 hl-color:: yellow id:: 65231058-389d-460a-8848-2d7ae2c7bc02
• 30% of the data in the training set ls-type:: annotation hl-page:: 4 hl-color:: purple id:: 6527c0ec-e2b1-4e07-a604-08cd13e590d4
• we analyze the types of noises in this sectio ls-type:: annotation hl-page:: 5 hl-color:: blue id:: 6527c209-af02-417d-ace0-63b94c12fb12
• we define “noise” in the dataset as data samples where the target output cannot be generated based on the existing input ls-type:: annotation hl-page:: 5 hl-color:: purple id:: 6527c239-2278-4e06-bbbf-77bc08c30ddf
• there is a significant semantic gap between the input and the output after an update ls-type:: annotation hl-page:: 5 hl-color:: purple id:: 6527c25b-b318-43a5-94eb-a2de19e37990
• Below we discuss specific situations of these two types of weak correlations ls-type:: annotation hl-page:: 5 hl-color:: yellow id:: 6527c3eb-297a-44c0-99c2-34cdb14e8a9d
• invalid comments can be found among them ls-type:: annotation hl-page:: 5 hl-color:: green id:: 6527c42a-9e06-422f-89f9-42008f798773
• comments that consist of purely invalid characters and comments that only contain a single unrelated word do not fit the scenario of real updates ls-type:: annotation hl-page:: 5 hl-color:: green id:: 6527c63f-b3fb-4384-9a8f-db163ca17ea3
• new code has been deprecated ls-type:: annotation hl-page:: 6 hl-color:: blue id:: 6527c660-e045-4759-9a79-a2d6c9bee964
• correlation between the code changes and the corresponding comment updating may also be weak. ls-type:: annotation hl-page:: 6 hl-color:: green id:: 6527c67f-a00e-4fda-a819-e76337de3bec
• if the code is significantly modified but the comments barely change, it belongs to this type of noise ls-type:: annotation hl-page:: 6 hl-color:: blue id:: 6527c68a-1ca0-4ee1-9a25-a8ca3da895c9
• Similarly, if the code barely changes while the comments need to be significantly modified, this can also mislead the model. ls-type:: annotation hl-page:: 6 hl-color:: blue id:: 6527c6c6-b750-4172-85cd-908563a8440c
• oise can exist in data where both comments and code undergo changes but their changes are not related to each other. ls-type:: annotation hl-page:: 6 hl-color:: blue id:: 6527c6e3-a425-4038-b5d0-85f2b49eac3d
• In summary, weak correlations between comments and code, as well as weak correlations between comment changes and code changes, are typical reasons for data noise in datasets for comment updating. These noisy data can mislead the model’s generation process, thus reducing the overall performance. Therefore, we may need a criteria to evaluate the quality of individual data samples and filter out noises based on their data quality scores ls-type:: annotation hl-page:: 6 hl-color:: purple id:: 6527c751-fdb2-4348-8cb2-77a8bb9ce1b2
• anchor to filter out the noising data ls-type:: annotation hl-page:: 6 hl-color:: yellow id:: 6527c78e-33c1-4cce-ae58-79cf5b88a983
• we designed a criterion to calculate the quality scores for each data samp ls-type:: annotation hl-page:: 6 hl-color:: blue id:: 6527ea08-470e-48cd-a341-246bfa44dbf5
• internal semantic similarity ls-type:: annotation hl-page:: 6 hl-color:: purple id:: 6527ea11-4785-4a7b-acc7-ed57e72e64bb
• Weak correlations existing between comments and code (Type I) ls-type:: annotation hl-page:: 6 hl-color:: green id:: 6527ea21-7aca-4a90-9625-8689f2281167 hl-stamp:: 1697115132593
• r internal semantic similarity ls-type:: annotation hl-page:: 6 hl-color:: yellow id:: 6527ea2b-87d4-411c-9f87-6ea17e5dff20
• . It is worth noting that the semantic representation mentioned above is at the token level. ls-type:: annotation hl-page:: 6 hl-color:: blue id:: 6527eb2c-3d1e-496a-95fc-f0bd677b27e4
• We choose the state-of-the-art model pre-trained on code-related datasets, GraphCodeBert [ 8 ], to calculate the semantics of comments or cod ls-type:: annotation hl-page:: 6 hl-color:: green id:: 6527eb30-56b8-4463-8dca-b866af730ee3
• cosine similarity of semantic embeddings to describe the correlation between old and new comments, and between old and new code, respectively ls-type:: annotation hl-page:: 6 hl-color:: purple id:: 6527eb4f-5388-42aa-9b20-6252653d43f0
• Sentence-BERT helps in capturing natural language semantic information to enhance the overall performance and robustness of the evaluation. Finally, we compute the evaluation score 𝑆1 for identifying noise in type I ls-type:: annotation hl-page:: 7 hl-color:: purple id:: 6527ebde-d308-4bfa-b756-f1357554b3e2
• Weak correlation existing between comment changes and code changes (Type II) ls-type:: annotation hl-page:: 7 hl-color:: green id:: 6527ec07-8c4d-491c-bd5d-abed4cf3c0cd
• nconsistencies between comment changes and code changes ls-type:: annotation hl-page:: 7 hl-color:: blue id:: 6527ec5d-8a25-4040-9011-35f017bda998
• comment updating task ls-type:: annotation hl-page:: 9 hl-color:: blue id:: 6527ed40-630d-44d2-86be-06711677011d
• pdated comment ls-type:: annotation hl-page:: 9 hl-color:: blue id:: 6527ed50-1f6f-4ad9-be71-de06a3c61a27 hl-stamp:: 1697122949958
• metrics for evaluating the effectiveness ls-type:: annotation hl-page:: 9 hl-color:: blue id:: 65280a83-cf57-4801-b278-342075aba5b8
• baselines ls-type:: annotation hl-page:: 9 hl-color:: blue id:: 65280a8a-682c-4fe4-a495-96eb196316c4
• experiment settings ls-type:: annotation hl-page:: 9 hl-color:: blue id:: 65280a8f-37f8-4c01-b348-8596dc431019
• Whether ls-type:: annotation hl-page:: 9 hl-color:: red id:: 65280a9e-80e8-4c23-82b7-0b3ab47c7501
• changes in the number of data ls-type:: annotation hl-page:: 9 hl-color:: green id:: 65280b47-e7d9-4af7-84b0-3050101288de
• xamples that CupCleaner identifies as noisy and high-quality data, ls-type:: annotation hl-page:: 9 hl-color:: blue id:: 65280bb0-8987-4b66-9bd8-f3d4821482a2
• human evaluation on the noisy and high-quality data identified by CupCleane ls-type:: annotation hl-page:: 9 hl-color:: blue id:: 65280bb8-4b66-40e2-a189-ca3d373cdd49
• Q2: How effective is the training data cleaned by CupCleaner? ls-type:: annotation hl-page:: 9 hl-color:: green id:: 65280bc1-dd33-4b4c-b521-b847608eeb33
• CupCleaner to clean only the training and validation sets ls-type:: annotation hl-page:: 10 hl-color:: green id:: 65280bf7-8a02-4e7f-bb1e-dce324699404
• comparing the performance of the models trained on the cleaned dataset with the models trained on the original dataset ls-type:: annotation hl-page:: 10 hl-color:: blue id:: 65280c01-34a8-48c7-9f34-86e59c418846
• RQ3: How effective is the use of CupCleaner-cleaned data for evaluating models? ls-type:: annotation hl-page:: 10 hl-color:: yellow id:: 65280c1c-24d1-4a81-bc17-dbec3d6ab178 hl-stamp:: 1697126043388
• different versions of the test set using the best-performing model from the previous research question. ls-type:: annotation hl-page:: 10 hl-color:: green id:: 65281639-713e-4ff7-921e-6fb204174b3b
• RQ4: How efficient is CupCleaner? ls-type:: annotation hl-page:: 10 hl-color:: green id:: 6528166d-bbb6-4570-9178-041617f45ebb hl-stamp:: 1697126045961
• we calculate the time consumption for cleaning the entire dataset, including the time required to compute scores and filter out the data ls-type:: annotation hl-page:: 10 hl-color:: green id:: 65281694-0f23-4f97-bb44-691e9c91a6e7
• able 1 describes the basic statistics of these three datasets. ls-type:: annotation hl-page:: 10 hl-color:: green id:: 65281b6d-8201-421e-be12-794ebdc336a8
• Panthaplackel et al., 2020 dataset collects 7.2k data samples from the commit history of open-source Java projects on Github ls-type:: annotation hl-page:: 10 hl-color:: green id:: 65281bb4-a458-45d0-b6a1-afd7d6980e06
• Liu et al., 2020 dataset is constructed from 1,063K method-doc co-change instances from Github, resulting in 104k data samples ls-type:: annotation hl-page:: 10 hl-color:: green id:: 65281bbd-0377-41e6-85cc-b25f16325046
• Panthaplackel et al., 2021 dataset is originally designed for comment consistency detection, but some of the data can also be used for the comment updating task ls-type:: annotation hl-page:: 10 hl-color:: green id:: 65281bc7-fd1d-43f4-ac76-4d9be7d8299f
• We extract the data with comment changes from this dataset to construct a new comment updating dataset, and discard the data with no changes in the comments. ls-type:: annotation hl-page:: 10 hl-color:: yellow id:: 65281c21-7981-447a-9611-a880f5781076
• However, due to a lack of exploration into the characteristics of comment updating, the constructed datasets still contain a significant amount of noisy data ls-type:: annotation hl-page:: 11 hl-color:: yellow id:: 65281c6a-e936-4cd6-aac0-d7fc79cf8632
• oken level ls-type:: annotation hl-page:: 11 hl-color:: green id:: 65281cb8-75c4-4555-b8d1-aae95c63a161
• e sentence level ls-type:: annotation hl-page:: 11 hl-color:: green id:: 65281cbc-9398-4989-bbf8-bf86684392ae
• evaluation metrics in the field of text generation ls-type:: annotation hl-page:: 11 hl-color:: green id:: 65281cc5-b0a4-4d47-b42f-bd0d44565f40
• align ls-type:: annotation hl-page:: 11 hl-color:: yellow id:: 65281d03-bc13-4652-aaab-197042fe3e1e
• valuation metric based on edit distance, originally designed to assess the overall quality of text simplification. ls-type:: annotation hl-page:: 11 hl-color:: green id:: 65281d43-dd7a-4892-b487-4dcc5d352db9
• GLEU is closer to human-level judgment than BLEU and is more suitable for tasks involving edits ls-type:: annotation hl-page:: 11 hl-color:: green id:: 65281d78-6b37-462f-b679-f28724513791
• PLBART ls-type:: annotation hl-page:: 12 hl-color:: purple id:: 65281df5-34e7-4335-80e7-73c907dc3802
• he experimental results show that UniXcoder achieves improvements on multiple code-related tasks. ls-type:: annotation hl-page:: 12 hl-color:: purple id:: 65281e15-0e25-43ed-bded-06f26fb91853
• The experimental results demonstrate that PLBART outperforms CodeGPT[ 18 ] and CodeBERT[ 5 ] in multiple code understanding and code generation tasks ls-type:: annotation hl-page:: 12 hl-color:: purple id:: 65281e1e-0235-4784-810a-60d8435d3d0a
• 188GB RAM ls-type:: annotation hl-page:: 12 hl-color:: purple id:: 65281e56-7510-44fc-9b22-53b9e89d5c9d
• NVIDIA TITAN RTX GPU with 24GB of memory. ls-type:: annotation hl-page:: 12 hl-color:: purple id:: 65281e5d-e30b-4338-abfe-6c0ed7002216
• 20 ls-type:: annotation hl-page:: 12 hl-color:: purple id:: 65281e63-04fc-48df-934a-b1b687e0951b
• terminate the training process if the experimental performance on the validation set does not improve within three consecutive epochs. F ls-type:: annotation hl-page:: 12 hl-color:: purple id:: 65281e6f-e9ea-4ab6-9352-51c962ce0602
• The red dashed line in the second step of Figure 2 (Sec.3) corresponds approximately to0.65, and the green solid line corresponds to 0.8 in the score distributio ls-type:: annotation hl-page:: 13 hl-color:: yellow id:: 65282097-c213-4409-93a4-fcec00a651ad
• The final scores ls-type:: annotation hl-page:: 13 hl-color:: yellow id:: 65282108-3ce6-4d81-9758-39974f55389a
• As shown in Table 2, we filter out approximately 33%, 12%, and 18% of the data in the three datasets, respectively. ls-type:: annotation hl-page:: 13 hl-color:: yellow id:: 65282238-75a8-43c9-87b5-01a0f937dfcb
• “Would you accept this data sample into the comment updating dataset?" ls-type:: annotation hl-page:: 14 hl-color:: green id:: 652822d1-4eea-4c87-bc4b-dc3d3c9c1fd4
• The label for each data sample ranges from 1 to 5 ls-type:: annotation hl-page:: 14 hl-color:: green id:: 652822fe-ebd6-4ef4-8eea-41a38feec147
• This indicates that the majority of the data filtered out by CupCleaner belong to noisy data, while the data retained is predominantly in line with the intent of the comment updating task. ls-type:: annotation hl-page:: 14 hl-color:: blue id:: 6528232d-b8f4-4993-983a-d3aa4f8e0374
• This indicates that CupCleaner can distinguish noisy data from high-quality data in comment updating datasets ls-type:: annotation hl-page:: 15 hl-color:: green id:: 6528234b-e57e-40ec-a50c-ce2a23319b41
• original, random, and cleaned ls-type:: annotation hl-page:: 16 hl-color:: green id:: 65282382-46ef-49d3-b737-ec22905d30e2
• subset of the original data with the same size as the cleaned data ls-type:: annotation hl-page:: 16 hl-color:: green id:: 652823d4-f34c-4535-a478-5f27586bbc15
• the original test set remained unchanged ls-type:: annotation hl-page:: 16 hl-color:: green id:: 652823ea-211b-4cb8-9460-d90a7e31443a hl-stamp:: 1697129696172
• we process only the training and validation sets ls-type:: annotation hl-page:: 16 hl-color:: yellow id:: 652823f0-2a17-43c6-b49b-52874a56768e hl-stamp:: 1697129491533
• CupCleaner shows its effectiveness regardless of whether the model performs well on the original dataset or whether the original data is heavily contaminated with noises. ls-type:: annotation hl-page:: 16 hl-color:: green id:: 652824ee-8b64-401d-870b-a98d079b402a
• The data with scores higher than the anchor point are assigned to the cleaned version of the test set, while the rest are assigned to the noisy version of the test set. ls-type:: annotation hl-page:: 16 hl-color:: green id:: 65282518-469b-4d09-8c6b-4d691d13b2b9
• This indicates that the cleaned test set may better reflect the model’s real performance, and also shows that CupCleaner may help in constructing datasets for accurate measurement of model abilities ls-type:: annotation hl-page:: 17 hl-color:: green id:: 6528254b-bb05-469d-88b5-fedfd7543dba
• his indicates that our data cleaning approach can produce cleaned data within an acceptable time ls-type:: annotation hl-page:: 18 hl-color:: green id:: 65282593-ddec-49be-a400-bc3a4fbbae40
• esults of human evaluation also demonstrate that our cleaned data aligns better with the scenarios and intentions of the comment updating tasks ls-type:: annotation hl-page:: 18 hl-color:: green id:: 652825e3-00c1-4b1c-81ad-ffc1424c4000
• CupCleaner, a data cleaning approach for comment updating datasets. The data cleaning strategy of CupCleaner mainly consists of two steps. First, we design a criterion to calculate the quality scores for all data samples, and then we remove the tail of the score distribution. ls-type:: annotation hl-page:: 20 hl-color:: purple id:: 65282606-3f49-4d98-bd46-501d37963ac4
• scoring process, we consider the correlations within the comments or code, as well as the correlation between code changes and comment changes ls-type:: annotation hl-page:: 20 hl-color:: yellow id:: 65282611-a2df-42a5-bbea-d654edd9fda3
• CupCleaner can effectively filter out noisy data and improve the performance of the models without changing the test set ls-type:: annotation hl-page:: 20 hl-color:: purple id:: 6528266e-744e-4067-a4a9-06da1fec5d40
• expand our data cleaning approach ls-type:: annotation hl-page:: 20 hl-color:: yellow id:: 652826af-5114-42ef-9ad1-b858e9db5237 hl-stamp:: 1697130161212