6.8 KiB
6.8 KiB
type:: REVIEWS tags:: year:: 2025 venue:: ICSE full-title:: Comparing the Efficiency of Semantic Code Clones date-start:: 17-05-2025 - 10:11 date-submitted:: external-links:: status:: DONE deadline-submission:: file:: @Comparing the efficiency of semantic code clones parent:: todoist:: https://app.todoist.com/app/task/1894-comparing-the-efficiency-of-semantic-code-clones-6XXr6PxFFpc2wrW6
- ### [[Highlights]]
- ### [[Comments]]
- #.tabular
- ### Paper summary
- This paper proposes PMA (Property/Metric Analyzer), an automated tool for comparing the performance of code clones, i.e.g, programs that produce the same results from the same inputs. PMA analyzes efficiency based on various input characteristics and performance metrics. The proposed framework involves performance profiling, statistical modeling, and comparative analysis across multiple implementations. It's evaluated on 20 programs from diverse domains and languages.
- ### Strengths
- + The paper is about a significant and complex challenge related to the efficiency evaluation and comparison of semantically equivalent programs
- + The evaluation covers a variety of subject areas and programming languages
- ### Weaknesses
- - The degree of automation concerning the definition of input properties is unclear and appears to rely heavily on user specification
- - The general applicability and scalability of the approach are not convincingly shown.
- - The claims of novelty are not well-established in relation to existing literature on profiling and performance analysis.
- ### Detailed comments for authors
- Novelty: While the idea of comparing the efficiency of semantically equivalent programs is interesting, the paper does not clearly distinguish itself from conventional profiling and performance analysis tools. Moreover, it overlooks fundamental challenges and theoretical issues in defining and identifying semantic code clones, which weakens the reliability of the proposal. The absence of a comparison with established profiling tools is a significant oversight. The related work section highlights several shortcomings:
- “The approach proposed in this paper automatically builds and analyzes performance profiles of the programs under test, closely related to profiling” (p. 10) -> However, there is no comparison with existing profiling mechanisms and tools.
- “In contrast, our approach systematically compares multiple equivalent implementations of a function or system, rather than focusing only on one, and incorporates multiple input properties in its analysis.” (p. 10) -> This claim aims to emphasize a key difference from existing work, but it is unconvincing. The definition of code clones used in the paper is vague, and the inherent theoretical limitations in identifying clones are not acknowledged. Since the proposed approach is approximate by nature, potential errors and uncertainties in clone identification should be explicitly considered and discussed in the following phases.
- “Typically, these tools are complex to use, in contrast to our approach. Their goal is also complementary as they are designed to operate at a huge scale; in contrast, our method can be applied at a small scale” (p. 10) -> This point is critical. To substantiate the claims, the authors should both demonstrate the usability of their approach and provide evidence that existing tools cannot be effectively used at small scale.
- **Rigor**: The study evaluates the proposed approach on 20 programs across various domains and programming languages, highlighting its scope of application. However, several methodological aspects lack clarity. The level of automation in defining input properties appears limited and is not properly discussed. Furthermore, the approach relies on strong assumptions of determinism and homogeneous execution environments, which are not justified. The generalizability of the approach is questionable. Efficiency differences may arise due to factors not accounted for, such as system architecture, compilers, or runtime environments. The paper also fails to position itself in the context of existing work on non-functional properties and performance analysis. The actual focus of the contribution is not clearly framed. By the end, it seems more aligned with analyzing alternative implementations of similar tasks rather than truly handling code clone analysis. Moreover, programs assumed to be clones may in fact differ semantically or in their execution behavior due to unmodeled environmental characteristics.
- Relevance: The research addresses a relevant problem in performance analysis and optimization. However, its broader impact is limited by the lack of comparison with existing performance tuning frameworks and tools. Moreover, the absence of connections with the broader body of work on non-functional properties further weakens the positioning of the contribution.
- Verifiability & transparency: The authors provide a Zenodo link with the implementation of the proposed approach, which contributes positively to transparency. However, the relationship between the tool implementation and the process described in the paper could be clarified and better documented.
- Presentation: The paper is generally well written, but some aspects could be improved. In particular, a diagram should be added at the beginning of Section 3 to provide a high-level overview of the proposed approach. Currently, the phases are introduced sequentially without an upfront explanation of the main components and their roles. Some figures also require enhancement. For instance, in Fig. 1b, the data for quicksort is not visible for sortedness values below 0.8, which makes the intended message at the end of Section 2.2 unclear.
- Additional comments:
- **Page 1**: “if for some input i, P(i) crashes but P′(i) does not...”: The authors should consider that the fault might be in either P or P′, not just P.
- **Page 3**: “These properties are very specific...”: It remains unclear how the automation of property definition is handled.
- **Page 3**: “every input”: This is too strong and unrealistic. The phrasing should be revised to reflect feasibility.
- **Page 5**: The generalizability of the approach is questionable. Differences in efficiency could be due to many factors not considered in the paper (e.g., system architecture, compiler, environment).
- **Page 9**: “efficiency metrics are tied to the system...”: This limitation is acknowledged but not addressed.
- Questions
-
- ### [[REVIEWS/Notes]]
- ### YELLOW CONCERNS
background-color:: yellow
- {{query (and [[ffd400]] [[icse2026-paper1894]] )}}
collapsed:: true
- ### ❓️Questions
- {{query (and [[question]] [[icse2026-paper1894]] )[[question]]}}
query-table:: true
query-properties:: [:block]