22 KiB
22 KiB
type:: REVIEWS tags:: year:: 2024 venue:: TOSEM full-title:: Towards Understanding Refactoring Engine Bugs date-start:: 16-12-2024 - 12:55 date-submitted:: external-links:: status:: DONE deadline-submission:: file:: @TOSEM-2024-0648_Proof_hi parent:: todoist:: https://app.todoist.com/app/task/tosem-2024-0648-reviewer-agreed-6WQgFVcxWqGWQvWg
- ### [[Highlights]]
collapsed:: true
- # Annotazioni
(16/12/2024, 12:55:04)
- “Towards Understanding Refactoring Engine Bug” ([“TOSEM-2024-0648_Proof_hi”, p. 1](zotero://select/library/items/GMH8846I)) ([pdf](zotero://open-pdf/library/items/J9KUUMWP?page=2&annotation=6NJKP67Q)) #a28ae5
- “refactoring engines can generate incorrect refactored programs, resulting in unexpected behaviors or even crashes.” ([“TOSEM-2024-0648_Proof_hi”, p. 2](zotero://select/library/items/GMH8846I)) ([pdf](zotero://open-pdf/library/items/J9KUUMWP?page=3&annotation=D7QEVCCX)) #5fb236
- “Refactoring is a critical process in software development, aiming at improving the internal structure of code while preserving its external behavior.” ([“TOSEM-2024-0648_Proof_hi”, p. 2](zotero://select/library/items/GMH8846I)) ([pdf](zotero://open-pdf/library/items/J9KUUMWP?page=3&annotation=LCE3Z7MU)) #5fb236
- “first systematic study of refactoring engine bugs” ([“TOSEM-2024-0648_Proof_hi”, p. 2](zotero://select/library/items/GMH8846I)) ([pdf](zotero://open-pdf/library/items/J9KUUMWP?page=3&annotation=RUZWQD4P)) #a28ae5
- “12 findings and provided a series of valuable guidelines for future work” ([“TOSEM-2024-0648_Proof_hi”, p. 2](zotero://select/library/items/GMH8846I)) ([pdf](zotero://open-pdf/library/items/J9KUUMWP?page=3&annotation=W2TKIR9L)) #5fb236
- “developers often experience unexpected results due to the flaws in the refactoring engines.” ([“TOSEM-2024-0648_Proof_hi”, p. 3](zotero://select/library/items/GMH8846I)) ([pdf](zotero://open-pdf/library/items/J9KUUMWP?page=4&annotation=66U3B8SS)) #a28ae5
- “hose bugs could silently change the program behavior or induce inconsistencies in the code base, thus producing very serious effects on real-world applications or improving the complexity of maintenance.” ([“TOSEM-2024-0648_Proof_hi”, p. 3](zotero://select/library/items/GMH8846I)) ([pdf](zotero://open-pdf/library/items/J9KUUMWP?page=4&annotation=MXK2KDHF)) #e56eee
- “how refactoring engine bugs are induced (i.e., root causes)” ([“TOSEM-2024-0648_Proof_hi”, p. 3](zotero://select/library/items/GMH8846I)) ([pdf](zotero://open-pdf/library/items/J9KUUMWP?page=4&annotation=QRZBJHBD)) #5fb236
- “how these bugs affect the software” ([“TOSEM-2024-0648_Proof_hi”, p. 3](zotero://select/library/items/GMH8846I)) ([pdf](zotero://open-pdf/library/items/J9KUUMWP?page=4&annotation=PC8JV8W9)) #5fb236
- “how these bugs can be found” ([“TOSEM-2024-0648_Proof_hi”, p. 3](zotero://select/library/items/GMH8846I)) ([pdf](zotero://open-pdf/library/items/J9KUUMWP?page=4&annotation=4PJRHT7K)) #5fb236
- “analyze bugs in refactoring engines” ([“TOSEM-2024-0648_Proof_hi”, p. 3](zotero://select/library/items/GMH8846I)) ([pdf](zotero://open-pdf/library/items/J9KUUMWP?page=4&annotation=S9VLC8SK)) #a28ae5
- “more likely to trigger refactoring engine bugs” ([“TOSEM-2024-0648_Proof_hi”, p. 3](zotero://select/library/items/GMH8846I)) ([pdf](zotero://open-pdf/library/items/J9KUUMWP?page=4&annotation=6HD8CKYR)) #5fb236
- “ug Symptoms” ([“TOSEM-2024-0648_Proof_hi”, p. 3](zotero://select/library/items/GMH8846I)) ([pdf](zotero://open-pdf/library/items/J9KUUMWP?page=4&annotation=IAYHX3QZ)) #a28ae5
- “Root Causes” ([“TOSEM-2024-0648_Proof_hi”, p. 3](zotero://select/library/items/GMH8846I)) ([pdf](zotero://open-pdf/library/items/J9KUUMWP?page=4&annotation=L9BVIEFZ)) #a28ae5
- “Symptoms & Root Cause” ([“TOSEM-2024-0648_Proof_hi”, p. 3](zotero://select/library/items/GMH8846I)) ([pdf](zotero://open-pdf/library/items/J9KUUMWP?page=4&annotation=QEBX54QU)) #a28ae5
- “Triggering Condition” ([“TOSEM-2024-0648_Proof_hi”, p. 3](zotero://select/library/items/GMH8846I)) ([pdf](zotero://open-pdf/library/items/J9KUUMWP?page=4&annotation=R4FTDJ2W)) #a28ae5
- “The input includes the input program and configuration for a refactoring, the output is the refactored program.” ([“TOSEM-2024-0648_Proof_hi”, p. 4](zotero://select/library/items/GMH8846I)) ([pdf](zotero://open-pdf/library/items/J9KUUMWP?page=5&annotation=N7TJL8AF)) #ffd400
*It is important to consider also the goal or motivations for the refactoring. Is it for updating used third-party libraries? Apply some design-patterns? I think the Configuration box in Fig. 1 should be expanded to cover such aspects.*
- “518 bugs” ([“TOSEM-2024-0648_Proof_hi”, p. 4](zotero://select/library/items/GMH8846I)) ([pdf](zotero://open-pdf/library/items/J9KUUMWP?page=5&annotation=Q96QABNS)) #a28ae5
- “six root causes” ([“TOSEM-2024-0648_Proof_hi”, p. 4](zotero://select/library/items/GMH8846I)) ([pdf](zotero://open-pdf/library/items/J9KUUMWP?page=5&annotation=AEUXARTK)) #e56eee
- “nine bug symptoms” ([“TOSEM-2024-0648_Proof_hi”, p. 4](zotero://select/library/items/GMH8846I)) ([pdf](zotero://open-pdf/library/items/J9KUUMWP?page=5&annotation=FEPJDU5R)) #e56eee
- “eight main input program characteristics” ([“TOSEM-2024-0648_Proof_hi”, p. 4](zotero://select/library/items/GMH8846I)) ([pdf](zotero://open-pdf/library/items/J9KUUMWP?page=5&annotation=ING5HE48)) #e56eee
- “better understanding” ([“TOSEM-2024-0648_Proof_hi”, p. 4](zotero://select/library/items/GMH8846I)) ([pdf](zotero://open-pdf/library/items/J9KUUMWP?page=5&annotation=7EEHFZI2)) #e56eee
- “RefactorBench” ([“TOSEM-2024-0648_Proof_hi”, p. 4](zotero://select/library/items/GMH8846I)) ([pdf](zotero://open-pdf/library/items/J9KUUMWP?page=5&annotation=V9NGCI9S)) #a28ae5
- “Initialization” ([“TOSEM-2024-0648_Proof_hi”, p. 4](zotero://select/library/items/GMH8846I)) ([pdf](zotero://open-pdf/library/items/J9KUUMWP?page=5&annotation=6EH8HIDR)) #2ea8e5
- “Initial conditions checking” ([“TOSEM-2024-0648_Proof_hi”, p. 4](zotero://select/library/items/GMH8846I)) ([pdf](zotero://open-pdf/library/items/J9KUUMWP?page=5&annotation=NMTSQGYA)) #2ea8e5
- “Additional information” ([“TOSEM-2024-0648_Proof_hi”, p. 4](zotero://select/library/items/GMH8846I)) ([pdf](zotero://open-pdf/library/items/J9KUUMWP?page=5&annotation=ITZNXRFQ)) #2ea8e5
- “Our study found that “Incorrect Transformations” (165) is the most common root cause for both Eclipse and IntelliJ IDEA, accounting for 62 and 1” ([“TOSEM-2024-0648_Proof_hi”, p. 5](zotero://select/library/items/GMH8846I)) ([pdf](zotero://open-pdf/library/items/J9KUUMWP?page=6&annotation=NXCYDEF2)) #5fb236
- “workflow” ([“TOSEM-2024-0648_Proof_hi”, p. 6](zotero://select/library/items/GMH8846I)) ([pdf](zotero://open-pdf/library/items/J9KUUMWP?page=7&annotation=RFC26CUY)) #ffd400
*To make the figure consistent with the text, "Final preconditions" should be "Final conditions checking", isn't it?*
- “Final conditions checking” ([“TOSEM-2024-0648_Proof_hi”, p. 6](zotero://select/library/items/GMH8846I)) ([pdf](zotero://open-pdf/library/items/J9KUUMWP?page=7&annotation=P3M27FRU)) #2ea8e5
- “For example, the Make Static refactoring cannot be executed if the refactored method is overridden in a child class because a static method cannot be overridden” ([“TOSEM-2024-0648_Proof_hi”, p. 6](zotero://select/library/items/GMH8846I)) ([pdf](zotero://open-pdf/library/items/J9KUUMWP?page=7&annotation=SR8SR9ME)) #ffd400
*For this specific example, why such a check cannot be verified during the "Initial preconditions" phase? The differences between Initial and final condition checks are not very clear.*
- “Create change” ([“TOSEM-2024-0648_Proof_hi”, p. 6](zotero://select/library/items/GMH8846I)) ([pdf](zotero://open-pdf/library/items/J9KUUMWP?page=7&annotation=VBHN27P9)) #2ea8e5
- “refactoring engine testing tools fail to trigger the bug” ([“TOSEM-2024-0648_Proof_hi”, p. 6](zotero://select/library/items/GMH8846I)) ([pdf](zotero://open-pdf/library/items/J9KUUMWP?page=7&annotation=J925Y2AA)) #a28ae5
- “insights we could get by analyzing existing bug reports.” ([“TOSEM-2024-0648_Proof_hi”, p. 6](zotero://select/library/items/GMH8846I)) ([pdf](zotero://open-pdf/library/items/J9KUUMWP?page=7&annotation=HCJ97L93)) #a28ae5
- “Running the initial input program in Figure 2 would produce “Counter: 15”. However, after performing Make Static refactoring (RQ1: refactoring type is Make Static refactoring) using the refactoring engine of Eclipse on the method “toBeRefactored()”, the output becomes “Counter: 5”. The refactored program does not contain any syntax error and Eclipse just silently changes the program behavior without any warning (RQ2: the symptom is “Behavior Change”). This happens because Eclipse erroneously interprets a method call within an anonymous class inside the refactored method as a call on “this” and thus changes the method invocation to be performed on the added input parameter of the method (RQ3). However, the call to the method “toCall()” was made on an instance of” ([“TOSEM-2024-0648_Proof_hi”, p. 6](zotero://select/library/items/GMH8846I)) ([pdf](zotero://open-pdf/library/items/J9KUUMWP?page=7&annotation=VXMFKJNW)) #ffd400
*The description of the bug should be improved. The "Counter" references are not clear and the RQx references should be better described.*
- “producing unexpected behavior on real-world applications or making it difficult to maintain.” ([“TOSEM-2024-0648_Proof_hi”, p. 7](zotero://select/library/items/GMH8846I)) ([pdf](zotero://open-pdf/library/items/J9KUUMWP?page=8&annotation=X7WQQ28L)) #5fb236
- “study the refactoring engine bugs, paves the road for related future research, and provides guidelines for engine developers and researchers to test and improve the reliability of the refactoring engines.” ([“TOSEM-2024-0648_Proof_hi”, p. 7](zotero://select/library/items/GMH8846I)) ([pdf](zotero://open-pdf/library/items/J9KUUMWP?page=8&annotation=C2PFU5TE)) #2ea8e5
- “(1) Java Development Tools (JDT) from Eclipse” ([“TOSEM-2024-0648_Proof_hi”, p. 7](zotero://select/library/items/GMH8846I)) ([pdf](zotero://open-pdf/library/items/J9KUUMWP?page=8&annotation=ZC82EFJY)) #5fb236
- “(2) the Java refactoring component of IntelliJ IDEA” ([“TOSEM-2024-0648_Proof_hi”, p. 7](zotero://select/library/items/GMH8846I)) ([pdf](zotero://open-pdf/library/items/J9KUUMWP?page=8&annotation=F3PZ9PQQ)) #5fb236
- “(3) Netbeans refactoring engine” ([“TOSEM-2024-0648_Proof_hi”, p. 7](zotero://select/library/items/GMH8846I)) ([pdf](zotero://open-pdf/library/items/J9KUUMWP?page=8&annotation=6AQW4KPA)) #5fb236
- “Table 2. Statistical Information of Datasets” ([“TOSEM-2024-0648_Proof_hi”, p. 8](zotero://select/library/items/GMH8846I)) ([pdf](zotero://open-pdf/library/items/J9KUUMWP?page=9&annotation=YMV2JQWM)) #5fb236
- “we need to identify the bug reports that aim to fix refactoring engine bugs only” ([“TOSEM-2024-0648_Proof_hi”, p. 8](zotero://select/library/items/GMH8846I)) ([pdf](zotero://open-pdf/library/items/J9KUUMWP?page=9&annotation=82WTB2KJ)) #a28ae5
- “(i.e., “refactoring” and “refactor”)” ([“TOSEM-2024-0648_Proof_hi”, p. 8](zotero://select/library/items/GMH8846I)) ([pdf](zotero://open-pdf/library/items/J9KUUMWP?page=9&annotation=83YLM5RE)) #ffd400
*It is not clear how authors manage to retrieve refactoring engine bugs starting from keywords like these two.*
- “collected bug reports” ([“TOSEM-2024-0648_Proof_hi”, p. 8](zotero://select/library/items/GMH8846I)) ([pdf](zotero://open-pdf/library/items/J9KUUMWP?page=9&annotation=55BGCM5Q)) #2ea8e5
- “we identified 637 bug reports related to refactoring engine bugs.” ([“TOSEM-2024-0648_Proof_hi”, p. 8](zotero://select/library/items/GMH8846I)) ([pdf](zotero://open-pdf/library/items/J9KUUMWP?page=9&annotation=4JANC4WV)) #ffd400
*It is not clear how such a crucial phase has been performed.*
- “Finally, we collected 326 bug reports containing both the input program and patch (IPP dataset).” ([“TOSEM-2024-0648_Proof_hi”, p. 9](zotero://select/library/items/GMH8846I)) ([pdf](zotero://open-pdf/library/items/J9KUUMWP?page=10&annotation=LNYXP7J4)) #2ea8e5
- “IPP” ([“TOSEM-2024-0648_Proof_hi”, p. 9](zotero://select/library/items/GMH8846I)) ([pdf](zotero://open-pdf/library/items/J9KUUMWP?page=10&annotation=HPPRBDN6)) #a28ae5
- “1) the refactoring type of the bug, 2) the root cause of the bug, 3) the symptom that the bug exhibits, and 4) the characteristics of the input program triggering the bug.” ([“TOSEM-2024-0648_Proof_hi”, p. 9](zotero://select/library/items/GMH8846I)) ([pdf](zotero://open-pdf/library/items/J9KUUMWP?page=10&annotation=UYPAPUJY)) #ffd400
*These four aspects are clear and I can understand their motivations. However, it is necessary to give explanatory examples and improve the discussion on how authors manage to recognize and retrieve bugs exibhiting such aspects that are crucial for the goal of the paper.
Moreover, concerning four aspects, it is not clear if authors defined 4 set of labels to cover these four aspects.*
- “ngine bugs” ([“TOSEM-2024-0648_Proof_hi”, p. 9](zotero://select/library/items/GMH8846I)) ([pdf](zotero://open-pdf/library/items/J9KUUMWP?page=10&annotation=5G7ZTTWZ)) #ffd400
*it is necessary to clarify if with "refactoring engine bugs" authors refer to bugs that are added by the employed refactoring engines, or to malfunctioning of the refactoring engines that do not necessarily manifest bugs in the generated code.*
- “error-prone refactoring transformations,” ([“TOSEM-2024-0648_Proof_hi”, p. 9](zotero://select/library/items/GMH8846I)) ([pdf](zotero://open-pdf/library/items/J9KUUMWP?page=10&annotation=VX4IC8KV)) #5fb236
- “Table 3. Statistics of bugs found across different refactoring types” ([“TOSEM-2024-0648_Proof_hi”, p. 10](zotero://select/library/items/GMH8846I)) ([pdf](zotero://open-pdf/library/items/J9KUUMWP?page=11&annotation=SCU5XKYF)) #ffd400
*It is necessary to clarify the focus of the work with respect to the considered technologies. In particular, in some parts of the paper it seems that authors intend to analyze Eclipse, IntelliJ Idea, and netbeans, whereas while discussing Table 3 in section 4, the discussion of these tools is mixed up with ASTGen, SAFEREFACTOR, and Gligoric. Please clarify how you have considered such tecnhologies together in your investigation.*
- “more challenging for the refactoring engines to take into account all the complicated usage scenarios.” ([“TOSEM-2024-0648_Proof_hi”, p. 11](zotero://select/library/items/GMH8846I)) ([pdf](zotero://open-pdf/library/items/J9KUUMWP?page=12&annotation=MRFYA8HS)) #a28ae5
- “Our analysis revealed that existing” ([“TOSEM-2024-0648_Proof_hi”, p. 11](zotero://select/library/items/GMH8846I)) ([pdf](zotero://open-pdf/library/items/J9KUUMWP?page=12&annotation=S4RNB2HM)) #ffd400
*How did you investigate that? How did you get this result?*
- “The refactored program is syntactically incorrect, resulting in compile errors.” ([“TOSEM-2024-0648_Proof_hi”, p. 12](zotero://select/library/items/GMH8846I)) ([pdf](zotero://open-pdf/library/items/J9KUUMWP?page=13&annotation=JD5K4NEN)) #ffd400
*Here we are referring to bugs in the generated code.*
- “The refactoring engine terminates unexpectedly during refactoring, which usually produces an error message.” ([“TOSEM-2024-0648_Proof_hi”, p. 12](zotero://select/library/items/GMH8846I)) ([pdf](zotero://open-pdf/library/items/J9KUUMWP?page=13&annotation=4J7HJEXS)) #ffd400
*Here, instead you are referring to bugs of the refactoring engine occurring during the refactoring process. This reinforces my previous comment when I say that it is not always clear if you are referring to bugs of the artifacts produced by the refactoring engines or to bugs of the refactoring engines itself. Finding 2 of the paper should be improved after having solved such an issue.*
- ““Compile Error” (242), “Crash” (106), and “Behavior Change” (66) are the top three most common symptoms of refactoring engine bugs.” ([“TOSEM-2024-0648_Proof_hi”, p. 13](zotero://select/library/items/GMH8846I)) ([pdf](zotero://open-pdf/library/items/J9KUUMWP?page=14&annotation=5SLWDEZ7)) #ffd400
- “Bug Distribution by Root Causes” ([“TOSEM-2024-0648_Proof_hi”, p. 16](zotero://select/library/items/GMH8846I)) ([pdf](zotero://open-pdf/library/items/J9KUUMWP?page=17&annotation=U3JPDUCW)) #ffd400
*Can you give a qualitative discussion on why Netbeans is not considered in Fig. 6?*
- “The result indicates that handling transformation issues in refactoring engines is very challenging and deserves more attention.” ([“TOSEM-2024-0648_Proof_hi”, p. 16](zotero://select/library/items/GMH8846I)) ([pdf](zotero://open-pdf/library/items/J9KUUMWP?page=17&annotation=CKP58A5V)) #5fb236
- “First, those input programs could be used to design unit tests to expose bugs in refactoring engines” ([“TOSEM-2024-0648_Proof_hi”, p. 17](zotero://select/library/items/GMH8846I)) ([pdf](zotero://open-pdf/library/items/J9KUUMWP?page=18&annotation=58C39ULF)) #5fb236
- “The decision to focus on the top 10 is motivated by their higher likelihood of triggering bugs, as well as the fact that manually validating all CRTs would be non-trivial and time-consuming.” ([“TOSEM-2024-0648_Proof_hi”, p. 24](zotero://select/library/items/GMH8846I)) ([pdf](zotero://open-pdf/library/items/J9KUUMWP?page=25&annotation=4JGGSXI8)) #5fb236
- “a partition, we first check the bug-tracking systems of the target IDEs to avoid duplicate reports before manual validation” ([“TOSEM-2024-0648_Proof_hi”, p. 24](zotero://select/library/items/GMH8846I)) ([pdf](zotero://open-pdf/library/items/J9KUUMWP?page=25&annotation=5TY7Z6IA)) #ffd400
*Such a checking activity needs to be clarified. It is not clear how authors have done it.*
- “Among the 21 bugs we submitted, 10 bugs are confirmed by their developers, and seven of them have already been fixed.” ([“TOSEM-2024-0648_Proof_hi”, p. 28](zotero://select/library/items/GMH8846I)) ([pdf](zotero://open-pdf/library/items/J9KUUMWP?page=29&annotation=LPQR7ABN)) #a28ae5
- “quality of refactoring engines.” ([“TOSEM-2024-0648_Proof_hi”, p. 28](zotero://select/library/items/GMH8846I)) ([pdf](zotero://open-pdf/library/items/J9KUUMWP?page=29&annotation=8N7GRY4E)) #a28ae5
- ### [[Comments]]
- SUMMARY: The paper analyzes refactoring engine bugs by specifically focusing on Eclipse, IntelliJ IDEA, and NetBeans. The analysis considers 518 bugs to identify their root causes, symptoms, and characteristics. The authors also propose guidelines for developers to enhance the reliability of refactoring engines. To perform the analysis, the authors curated RefactorBench, a dataset of refactoring bugs, and validated their findings by submitting bug reports to the developers of popular refactoring tools.
- COMMENTS: The paper addresses an important issue in software development, as refactoring is critical for improving code quality. However, there are several areas where the discussion can be clarified and improved, as detailed below:
- The distinction between bugs in the artifacts produced by refactoring engines and bugs in the engines themselves must be more clear. For example, the comments related to "Finding 2" and the discussion on "Compile Errors" and "Crash" need refinement to distinguish these two categories effectively.
- The paper does not explain how the authors retrieved bug reports using keywords like "refactoring" and "refactor". For instance, on page 8 of the paper, it needs to be clarified how authors retrieved refactoring engine bugs starting from keywords like these two. This phase is crucial and requires better elaboration.
- The manual validation process to identify unique bug reports from bug-tracking systems is briefly mentioned but needs a more detailed description.
- Some figures and tables are inconsistent with the text. For example:
- In Table 3, the discussion mixes Eclipse, IntelliJ, and NetBeans with tools like ASTGen and SAFEREFACTOR. The relationship between these technologies should be clarified.
- Figure 6 excludes NetBeans without explanation. It is necessary to give a qualitative discussion on why NetBeans is not considered in Fig. 6.
- The description of Figure 1's "Configuration box" could be expanded to include motivations for the refactoring process, such as updating third-party libraries or applying design patterns.
- While the paper identifies different root causes and bug symptoms, it could provide more qualitative discussions or examples to help readers understand these categories better.
- The distinction between "Initial preconditions" and "Final conditions checking" (e.g., for the "Make Static" refactoring) is unclear. This differentiation should be explained in greater depth.
- The paper mentions that the authors categorized bugs based on four aspects: 1) refactoring type, 2) root cause, 3) symptom, and 4) input characteristics, but it does not clearly explain how these labels were assigned. Explanatory examples and a discussion on the labeling process are necessary.
- ####
- ### [[REVIEWS/Notes]]
- ### YELLOW CONCERNS
background-color:: yellow
- {{query (and [[ffd400]] [[TOSEM-2024-0648]] )}}
collapsed:: true
- ### ❓️Questions
- {{query (and [[question]] [[TOSEM-2024-0648]] )[[question]]}}
query-table:: true
query-properties:: [:block]