logseq/pages/TOSEM-2023-0288.md

type:: REVIEWS tags:: year:: 2023 venue:: TOSEM full-title:: CupCleaner: A Data Cleaning Approach for Comment Updating date-start:: 04-09-2023 - 17:35 date-submitted:: external-links:: https://mc.manuscriptcentral.com/tosem?URL_MASK=9c111b1449d4484d9d197673bde0a450 status:: DONE deadline-submission:: 02-10-2023 file::

- [[Highlights]]
	- ((65198f61-6aab-4f71-84b8-04a3c5a51746))
		- Whare are they?
		- #+BEGIN_IMPORTANT
		  ((65282606-3f49-4d98-bd46-501d37963ac4))
		  ((6528266e-744e-4067-a4a9-06da1fec5d40))
		  ((65199218-a454-4b39-997a-031ea97f4c0b))
		  ((6527c751-fdb2-4348-8cb2-77a8bb9ce1b2))
		  #+END_IMPORTANT
	- ((65230fe4-59f5-4925-8f7e-5c2b5283cd29))
		- This seems to be quite obvious, isn't it?
			- ((65231058-389d-460a-8848-2d7ae2c7bc02))
	- ((6527c239-2278-4e06-bbbf-77bc08c30ddf))
		- ((6527c25b-b318-43a5-94eb-a2de19e37990))
	- There are two kinds of weak correlations: **The presentation of these two kinds of weak correlations need to be better presented.**
		- The fist one is withing the OLD and NEW comments or code
			- ((6527c63f-b3fb-4384-9a8f-db163ca17ea3))
		- The second one is between CODE CHANGES and COMMENT CHANGES
			- ((6527c68a-1ca0-4ee1-9a25-a8ca3da895c9))
			- ((6527c6c6-b750-4172-85cd-908563a8440c))
			- ((6527c6e3-a425-4038-b5d0-85f2b49eac3d))
	- ((6527c78e-33c1-4cce-ae58-79cf5b88a983))
		- What does it mean? Can you give an illustrative example of anchor in this context?
	- ((6527ea2b-87d4-411c-9f87-6ea17e5dff20))
		- What does it mean?
			- ((6527eb30-56b8-4463-8dca-b866af730ee3))
			- ((6527eb2c-3d1e-496a-95fc-f0bd677b27e4))
	- ((6527ed50-1f6f-4ad9-be71-de06a3c61a27))
		- Updated comment*s*
	- ((65280a9e-80e8-4c23-82b7-0b3ab47c7501))
		- I don't like starting the research question in this way
	- ((65280c1c-24d1-4a81-bc17-dbec3d6ab178))
		- This is included in RQ2, isn't it?
	- ((65281c21-7981-447a-9611-a880f5781076))
		- Is this done for each of the three datasets?
	- ((65281c6a-e936-4cd6-aac0-d7fc79cf8632))
		- Check if this represents a bias for the study.
	- ((65281d03-bc13-4652-aaab-197042fe3e1e))
		- What's kind of alignment?
	- ((65282097-c213-4409-93a4-fcec00a651ad))
		- These numbers require explanation.
	-
	- ((65282108-3ce6-4d81-9758-39974f55389a))
		- Can you explicitely discuss what the scores represent? What's the distributions are measuring/representing?
		- For inexpert readers, it can be helpful to give introductory sentences about anchor search, and anchor text distribution
	-
	- ((65282238-75a8-43c9-87b5-01a0f937dfcb))
		- It is not clear how authors got such filtering figures
	- ((652823ea-211b-4cb8-9460-d90a7e31443a))
		- Can you give you some examples?
	-
	- ((65282611-a2df-42a5-bbea-d654edd9fda3))
		- This process needs to be better described by means of explanatory and illustrative case, by showing even some corner cases so that all the details about the cleaning process can be given.
	- ((652826af-5114-42ef-9ad1-b858e9db5237))
		- What kinds of improvement do you foresee?
		-
- [[Comments]]
	- **Summary**: The article presents CupCleaner, a method designed for refining comment-updating datasets. By calculating quality scores for each data sample, CupCleaner identifies and discards the ones with lower scores. The scoring mechanism takes into account the relationship between comments and code, as well as changes in both. The efficiency of CupCleaner is tested through human evaluations, three comment-updating datasets, and three code-centric models. The findings suggest that CupCleaner effectively eliminates unwanted data, enhancing model accuracy without modifying the test set. Moreover, the time taken by CupCleaner for data cleaning is reasonable when compared with the model training duration.
	-
	- **Comments**:
		- The paper is about an interesting problem and overall it fairly presents both the problems and the proposed solutions. Here are my suggestions to enhance the manuscript
			- In order to enhance the reader's understanding of the technical contribution presented in Section 3, it is recommended to include additional explanatory examples. For instance, when discussing "anchor search," providing introductory sentences about anchor text distribution and its practical utility can be beneficial. This would allow readers to grasp the details more effectively.
			- The same comment applies to the score calculation approach given in Section 3.1.
			- I would change RQ1 as "Can CupCleaner filter out true noise data?"
			- The authors state at the end of Sect 4.2 "However, due to a lack of exploration into the characteristics of comment updating, the constructed datasets still contain a significant amount of noisy data". This aspect should be adequately discussed in the threats to validity section as it points to the presence of a significant amount of noisy data in the constructed datasets.
			- Regarding Fig. 2 (sec. 3), the rationale behind selecting the thresholds of 0.65 and 0.8 needs further elaboration. It is crucial to provide a detailed explanation of the decision-making process behind choosing these specific values.
			- The scores presented in Figure 4 need to be described in terms of their representation and the distribution they measure. To aid inexperienced readers, it would be beneficial to provide introductory sentences explaining anchor search and anchor text distribution.
			- The authors applied specific thresholds to filter out certain percentages of data in the three datasets. On page 12 (line 615), they mentioned that approximately 33%, 12%, and 18% of the data were filtered out in the respective datasets. However, without further information, it is not possible to determine the specific rationale behind choosing those thresholds. It is essential to review the context in which these thresholds were discussed, such as the research objective, dataset characteristics, or any previous literature references, to understand the reasoning behind applying these particular cutoffs.
			- In the conclusion section, the authors state their intention to enhance the data cleaning approach in future work. It would be beneficial to elaborate on the specific improvements they intend to implement. Additionally, I suggest emphasizing the limitations of CupCleaner earlier in the paper and outlining their proposed strategies for addressing these limitations.
			-
	-
	-
-
- [[REVIEWS/Notes]]
-
-