logseq/pages/@Comparing the efficiency of semantic code clones.md

tags:: #zotero date:: 2017 title:: @Comparing the efficiency of semantic code clones item-type:: journalArticle original-title:: Comparing the efficiency of semantic code clones language:: en library-catalog:: Zotero links:: Local library, Web library

• Abstract
  • Two programs are said to be semantic code clones if they produce the same output for all inputs. But there is far more to code than its semantics; for instance a pair of semantic code clones can have more or fewer vulnerabilities, can be more or less readable, or can be more or less efficient with respect to run time, memory usage, etc. In this paper, we propose a framework and approach to systematically and automatically compare the efficiency of such semantic code clones. Our approach distills program inputs into multiple numeric input properties, leverages statistical and numerical methods to model and analyze the relationships between these inputs and various efficiency metrics related to program execution, and finally reports the best performing clones w.r.t. all efficiency metrics on the studied ranges of property values, as well as the models and visualizations of efficiency as it relates to the input properties. We implement this approach in a tool called PMA (for Property/Metric Analyzer), with support for programs written in Python, JavaScript, Rust, and R, and evaluate PMA in 8 subject areas, showing that PMA generates useful and sometimes surprising insights about semantic code clones across many languages and applications; insights that we believe developers could leverage to better understand and improve the efficiency of their applications.

• Attachments

  • PDF {{zotero-imported-file MFEDET3D, "2017 - Comparing the efficiency of semantic code clones.pdf"}}

• Notes

  collapsed:: true

  • I'm reviewing a research paper and I took the following notes:

    Annotazioni

    (17/5/2025, 10:05:19)

    • “Two programs are said to be semantic code clones if they produce the same output for all inputs.” (“Comparing the efficiency of semantic code clones”, 2017, p. 1) #5fb236

    • “In this paper, we propose a framework and approach to systematically and automatically compare the efficiency of such semantic code clones.” (“Comparing the efficiency of semantic code clones”, 2017, p. 1) #5fb236

    • “Our approach distills program inputs into multiple numeric input properties, leverages statistical and numerical methods to model and analyze the relationships between these inputs and various efficiency metrics related to program execution, and finally reports the best performing clones w.r.t. all efficiency metrics on the studied ranges of property values, as well as the models and visualizations of efficiency as it relates to the input properties” (“Comparing the efficiency of semantic code clones”, 2017, p. 1) #5fb236

    • “PMA (for Property/Metric Analyzer)” (“Comparing the efficiency of semantic code clones”, 2017, p. 1) #5fb236

    • “Two programs are said to be semantic code clones if they produce the same output for all inputs, and semantic code clones are everywhere.” (“Comparing the efficiency of semantic code clones”, 2017, p. 1) #5fb236

    • “These kinds of clones appear frequently in the context of differential testing, a strategy to test some program P by also executing other, semantically-equivalent programs, e.g., P′, with the same inputs as P and using the other executions as oracles.” (“Comparing the efficiency of semantic code clones”, 2017, p. 1) #5fb236

    • “if for some input i, P (i) crashes but P′ (i) does not, there is probably an issue with P.” (“Comparing the efficiency of semantic code clones”, 2017, p. 1) #ffd400 or the other way round.... the issue might be with P' and not with P

    • “in this work we present an approach that automatically and systematically compares the efficiency of semantic code clones by exercising them with the same inputs and observing efficiency metrics related to execution.” (“Comparing the efficiency of semantic code clones”, 2017, p. 1) #a28ae5

    • “the approach can automatically generate inputs and distills them into multiple numeric input properties in order to relate them to the collected efficiency metrics.” (“Comparing the efficiency of semantic code clones”, 2017, p. 1) #a28ae5

    • “our approach can leverage statistical and numerical methods to build, analyze, and compare models of program efficiency, and we develop novel modeling and analysis techniques that account for the probabilistic nature of recording the efficiency of code.” (“Comparing the efficiency of semantic code clones”, 2017, p. 1) #a28ae5

    • “PMA (for Property/Metric Analyzer), and evaluate it on 20 subject applications spanning eight application areas and four programming languages.” (“Comparing the efficiency of semantic code clones”, 2017, p. 1) #a28ae5

    • “PMA can discover which input properties contribute to efficiency and which do not, and find which application is most efficient on which ranges of input property values, as well as overall.” (“Comparing the efficiency of semantic code clones”, 2017, p. 1) #5fb236

    • “an approach for profiling and analyzing the efficiency of semantic code clones, w.r.t. several properties of inputs;” (“Comparing the efficiency of semantic code clones”, 2017, p. 1) #2ea8e5

    • “a prototype implementation of this approach in a tool called PMA,” (“Comparing the efficiency of semantic code clones”, 2017, p. 1) #2ea8e5

    • “evaluation where we applied PMA in various domains” (“Comparing the efficiency of semantic code clones”, 2017, p. 1) #2ea8e5

    • “quicksort, mergesort” (“Comparing the efficiency of semantic code clones”, 2017, p. 2) #5fb236

    • “bubblesort, and insertionsort” (“Comparing the efficiency of semantic code clones”, 2017, p. 2) #5fb236

    • “there’s quite a bit of variability in both runtime and memory consumed for lists of the same length, which implies that another property of the input lists may be playing a role in the efficiency of the algorithms;” (“Comparing the efficiency of semantic code clones”, 2017, p. 2) #5fb236

    • “sortedness”, which we compute as the % of adjacent elements in the list that are out of order.” (“Comparing the efficiency of semantic code clones”, 2017, p. 2) #5fb236

    • “considering sortedness explains the anomalous extremely high run time and peak memory values for quicksort:” (“Comparing the efficiency of semantic code clones”, 2017, p. 2) #ffd400 I appreciate the effort of producing Fig. 1b. However, it is not clear for some of the situations. For instance, quicksort is not visible for sotedness values <0.8 Thus the message that authrors want to convey at the end of Sec 2.2 is not fully clear.

    • “At its core, our approach aims to investigate the relationship between inputs and various efficiency metrics related to program execution” (“Comparing the efficiency of semantic code clones”, 2017, p. 2) #a28ae5

    • “computing numeric properties of inputs” (“Comparing the efficiency of semantic code clones”, 2017, p. 2) #5fb236

    • “Input Properties.” (“Comparing the efficiency of semantic code clones”, 2017, p. 2) #2ea8e5

    • “In the sorting example, these are the length of the input list, as well as the % of adjacent pairs of elements in the list that are out-of-order; length is a default property, and the latter is a custom property. Our approach also makes the distinction between primary and other input properties; by default the primary input property is the first supplied by the user, and forms the basis of the initial exploratory data analysis which will, among other things, determine which other properties are significant predictors of efficiency.” (“Comparing the efficiency of semantic code clones”, 2017, p. 3) #ffd400 These properties are very specific to the program at hand. Who is going to define them? Still not clear at this stage the automation level of the proposed approach.

    • “Efficiency Metrics.” (“Comparing the efficiency of semantic code clones”, 2017, p. 3) #2ea8e5

    • “during the execution of a program.” (“Comparing the efficiency of semantic code clones”, 2017, p. 3) #5fb236

    • “observing with some observer M the execution of a program S on some inputs i0, ..., in, i.e., M (S (i0, ..., in)) = m” (“Comparing the efficiency of semantic code clones”, 2017, p. 3) #5fb236

    • “profile the programs under test S0, ..., Sk , collecting various metrics m0, ..., m j by observing them during program execution” (“Comparing the efficiency of semantic code clones”, 2017, p. 3) #5fb236

    • “every input” (“Comparing the efficiency of semantic code clones”, 2017, p. 3) #ffd400 Every input needs to be further elaborated! Every is strong.

    • “Profiling produces a summary containing b entries for each of the k programs, and each entry contains the values of all r input properties and j metrics. This summary serves as the basis for all future analysis steps, which are discussed next.” (“Comparing the efficiency of semantic code clones”, 2017, p. 3) #5fb236

    • “to identify and discard true clones, which we define as clones with identical observable semantics and observable efficiency” (“Comparing the efficiency of semantic code clones”, 2017, p. 3) #ffd400 Why discard?

    • “statistically significant differences between the programs for each metric.” (“Comparing the efficiency of semantic code clones”, 2017, p. 3) #5fb236

    • “t x1 ∈ X1 will be greater than an element x2 ∈ X2” (“Comparing the efficiency of semantic code clones”, 2017, p. 3) #5fb236

    • “hen the programs are determined to be equivalent and one of the clones is discarded at random.” (“Comparing the efficiency of semantic code clones”, 2017, p. 3) #ffd400 Why is it discarded? It's not clear the discarding phase with respect to the overall goal of the approach as presented in the introduction "....compares the efficiency of semantic code clones...."

    • “In this stage, the analysis determines and reports all significant differences between the remaining “false clones”.” (“Comparing the efficiency of semantic code clones”, 2017, p. 3) #ffd400 Now it becoming a bit clear. So essentially during the profiling phase you aim at removing all the programs that according to the £detect true clones" phase are confirmed to be clones. This needs to be clearly stated. Why do you need to do so? It is necessary to introduce early in Section 3 a figure depicting all the phases of the proposed approach so that the reader can have a high level view of the approach and better follow the details.

    • “3.3.1 Relative Efficiency Analysis.” (“Comparing the efficiency of semantic code clones”, 2017, p. 3) #2ea8e5

    • “in Section 2 we find the turning range where insertionsort and mergesort are equally efficient is [108, 140], while [0, 108] is optimal for insertionsort and [140, 500] is optimal for mergesort.” (“Comparing the efficiency of semantic code clones”, 2017, p. 4) #5fb236

    • “3.3.2 Anomaly Investigation.” (“Comparing the efficiency of semantic code clones”, 2017, p. 4) #2ea8e5

    • “20 subject applications across eight subject areas spanning four programming languages.” (“Comparing the efficiency of semantic code clones”, 2017, p. 5) #5fb236

    • “Setup. We aim to find the optimal list length at which to switch between quicksort and insertion sort, and see if the developers were correct in setting it to 20.” (“Comparing the efficiency of semantic code clones”, 2017, p. 5) #ffd400

    • “For our input properties, we chose the length of the list, since this is the metric over which the sorting algorithm switch threshold is computed, and also sortedness (recall that sortedness referes to the % of adjacent elements that are in order).” (“Comparing the efficiency of semantic code clones”, 2017, p. 5) #ffd400 I'm not convinced about the generalizability of the work. Moreover, the paper is related to many works on non-functional aspects of software systems, including performance analysis that are completely neglected in the paper. The focus of the paper is not clear. In the end it seems that the work is on supporting the analysis of alternative implementations of similar tasks with respect to efficiency criteria. Programs under analysis that are supposed to be clones, might be not and the different on performance might due on several aspects that might go beyond the given input and on characteristics that might even be related to the execution environment.

    • “It is possible that our evaluation is not representative of real-world scenarios.” (“Comparing the efficiency of semantic code clones”, 2017, p. 9) #ffd400 Exactly. See my previous comment.

    • “Efficiency metrics are tied to the system on which a program is being run, and so it is possible that running our evaluation on a different system would yield different results.” (“Comparing the efficiency of semantic code clones”, 2017, p. 9) #ffd400 This is also related to my previous points. In my opinion the considered settings for comparison makes semplifications that need to be convincingly supported!

    • “The approach proposed in this paper automatically builds and analyzes performance profiles of the programs under test, closely related to profiling” (“Comparing the efficiency of semantic code clones”, 2017, p. 10) #ffd400 There is no comparison of the proposed approach with existing profiling mechanisms and tools.

    • “In contrast, our approach systematically compares multiple equivalent implementations of a function or system, rather than focusing only on one, and incorporates multiple input properties in its analysis.” (“Comparing the efficiency of semantic code clones”, 2017, p. 10) #ffd400 This is said in the related work and claimed to highlight the difference with respect to existing work. This is not convincing in my opinion. First of all because the definition of code clones is blurred in my opinion. There are theoretical limitation on identifying clones. The proposed approach is kind of approximation, and as a such it is necessary to consider potential errors while discussing the subsequent phases of the process that rely on such approximations.

    • “Typically, these tools are complex to use, in contrast to our approach. Their goal is also complementary as they are designed to operate at a huge scale; in contrast, our method can be applied at a small scale” (“Comparing the efficiency of semantic code clones”, 2017, p. 10) #ffd400 This is also a claim in the related work section. This is a critical point for comparison. To make the paper strong it is required to demonstrate that the proposed approach is on the one hand easier than existing approaches and on the other hand demonstrate that existing tools cannot be operated at small scale.

    • “we proposed a novel automated profiling and analysis approach that leverages multiple input properties and models and analyzes their relationship with efficiency metrics related to the execution of multiple semantically equivalent programs” (“Comparing the efficiency of semantic code clones”, 2017, p. 10) #5fb236

    • “With this approach, developers can compare functionally-equivalent programs in a single framework, visually examine the relative effects of multiple input properties on a given efficiency metric, and automated statistical and numerical analysis determines optimal ranges for programs and builds multi-dimensional models of efficiency.” (“Comparing the efficiency of semantic code clones”, 2017, p. 10) #5fb236

    • “we implemented our approach in a tool called PMA, and evaluated its ability to discern efficiency differences in eight subject areas across four programming languages.” (“Comparing the efficiency of semantic code clones”, 2017, p. 10) #5fb236

    COnsider that those that are tagget with #5fb236 are just highlights, those that are tagged with #e56eee and #a28ae5 are imporant sentences. Please pay attention instead to the notes that are tagged with #ffd400. Those that are tagged with #ff6666 are typos or errors. Could you please draft a review by organizing it as follows:

    SUMMARY: Just a few sentence to summarize the work

    STRENGHTS:

    WEAKNESSES:

    COMMENTS: Organize the notes with respect to the following criteria:

    Novelty

    Rigor

    Relevance (of the contribution)

    Verifiability and Transparency

    Presentation

    And then add a Detailed Comments section to report the notes that contain issues or typos. Can you also formulate three explicit questions by considering the comments above?