17 KiB
17 KiB
type:: JournalPaper
date:: 18-03-2025 - 10:36
full-title:: Is Code Coverage of Performance Tests Related to Source Code Features? An Empirical Study on Open-Source Systems
external-links:: Empirical Study - JSS Extended Article - Online LaTeX Editor Overleaf
year:: 2025
status:: ACCEPTED
venue:: EMSE
priority:: P1
file:: 
- #.tabular
collapsed:: true
- ## MEETINGS
collapsed:: true
- {{query (and [[PAPERS/EMSE-D-25-00044-IMRAN]] (block-property :type [[meeting]]) (not (page [[PAPERS/EMSE-D-25-00044-IMRAN]])))}}
- query-table:: true
- ## READINGS
collapsed:: true
- #+BEGIN_QUERY
{:title "Related readings in Omnivore"
:query [:find (pull ?b [*] )
:in $ ?current-page
:where
[?b :block/page ?p]
[?p :block/name ?p-name]
[?b :block/content ?c]
[(clojure.string/includes? ?c ?current-page)]
[(clojure.string/includes? ?p-name "omnivore")]
]
:inputs [:current-page]}
#+END_QUERY
-
- ## TASKs
collapsed:: true
-
- ## ❓️ QUESTIONS TO ANSWER
{{query (and [[question]][[PAPERS/EMSE-D-25-00044-IMRAN]])}}
- ## [[PAPERS/NOTES]]
- # Reviews
- ## COMMENTS FOR THE AUTHOR:
collapsed:: true
- It is recommended that the manuscript undergo a major revision before being accepted. Specifically, the authors should consider the following points, included but not limited to:
- E1) incorporating **expert-annotated labels to enhance the machine learning model's accuracy** in identifying methods that truly require performance testing rather than replicating past habits;
- We wanted to investigate if we can get some good results "for free". We wanted to use benchmarks as proxy. And benchmarks were developed by experts
- We should ask the developers to do so. It would be another paper.
- E2) providing a detailed explanation of the machine learning **model’s decision-making process** to improve transparency;
- Expand the description of the used algorithms?
- E3) **expanding the dataset** to include a more diverse range of applications beyond open-source Java projects to better assess generalizability;
- Enterprise applications are not open-source. Maybe we could ask Andre Bondi if there are
- E4) exploring and integrating advanced machine learning techniques, such as **deep learning models**, to capture complex relationships in code;
- Fine tuning CodeBERT and use it as we have done for the others.
- E5) justifying the **choice of code coverage** as a metric for performance test quality, recognizing that executing a method does not necessarily indicate its importance for performance bottlenecks;
- I think we have already some answer for this
- E6) clarifying the **criteria for selecting the open-source Java projects used in the study**;
- We inherited this from previous studies. The dataset comes from a published paper.
- E7) re-evaluating the granularity of their coverage analysis methodology;
- Current tools do not permit to go deeper than line coverage.
- E8) discussing the **rationale for not using k-fold cross-validation** in model evaluation;
- Riccardo can run everything with k-fold cross-validation.
- 9) revising any unclear terminology throughout the manuscript; and
- 10) referencing **relevant literature on load/performance test generation to enrich the study**.
- ## Reviewer 1
collapsed:: true
- *This paper studied the code coverage issue of performance tests. More specifically, using Java benchmarks JMH, this paper studied these benchmarks' code coverage and their time cost. Additionally, based on JMH's coverage, it built a machine learning model to classify which part of the code should be covered. This is an extension of a prior conference publication. The extension is mainly on studying the features of the code that are covered by performance tests.*
- Some suggestions for improving the paper:
- 1. **How to determine the number of testing iterations should be reported in the paper**. And the exact number of iterations should be reported as well. It may be better to choose a stop criterion, for example, you can use the method from [49].
- WE need a stronger reasonfor inheriting the number of iterations from the developers
- 2. I assume you used JMH as the training data? The main issue is that JMH may not be a very reliable example of performance testing set. I strongly recommend you add a paragraph to **Threads to Validity about the possibility that JMH may be missing important performance test coverage**. In fact, there are papers on load/performance test generation that can give better coverages.
- Just follow what he is suggesting
- 3. I am a bit surprised that MLP didn't make to top 5 after **hyperparameter tuning**. I recommend you do a larger search. Moreover, hyperparameter tuning is a must for building models. Hence, it can be merged into RQ3.1.
- Let's add more parameters for the hyperparameter tuning
- 4. Reference 45 and 46 are the same.
- 5. For load/perf test generation, you can start with "PerfFuzz: automatically generating pathological inputs" and "Automatic generation of load tests". **That probably can give you a better training data set, may be for future work.**
- We are not working on the fact that we want to identify the key inputs to identify bottleneck...
- 6. In the introduction, the authors mentioned studying the features. However, there lacks detailed discussion on the features' importance. Can the authors do a feature analysis using SHAP or LIME
- Riccardo will have a look at these two tools.
- ## Reviewer 2
collapsed:: true
- *This study analyzes 28 open-source Java projects, evaluating 2,190 JMH performance benchmarks and 256,233 JUnit functional tests to investigate performance test coverage, execution time cost, and the feasibility of predicting test selection using static code features. Key findings are: 1) Performance tests have significantly lower coverage than functional tests; 2) Performance tests are extremely time-consuming; and 3) Machine learning models struggle to predict which methods should be performance tested. In addition, this study highlights major challenges in performance testing, particularly low coverage, long execution times, and difficulties in selecting the right methods to test. It suggests the need for better test selection strategies, improved performance testing tools, and more efficient testing practices.*
- Strengths:
- 1. Clear and well-structured research: The methodology is systematic and rigorous, covering both dynamic (call stack analysis) and static (srcML-based) techniques.
- 2. The study gives hard numbers on how bad performance test coverage is and shows how inefficient many test suites are.
- 3. Impactful for both practitioners and researchers. For developers, the study shows that most performance test suites aren't doing enough and that they need better tools to understand what's actually being tested. For researchers, it raises big questions—like how can we improve performance test selection? And can we integrate runtime profiling data to make smarter test decisions?
- Weakness and questions:
- Your study design is well-structured and rigorous, however, there are several potential issues and limitations that should be addressed, such as:
- 1) First, the model predicts which methods are covered by performance tests, but this does not necessarily mean those methods should be tested. The model might just be replicating past testing habits, which could be biased or incomplete. **You should incorporate expert-annotated labels to indicate which methods actually need performance testing.**
- *Vedere E1*
background-color:: red
- 2) Also, the study applies machine learning but **doesn't explain why the models make certain predictions**. The study uses machine learning to find patterns in the code, but it doesn't explain why the model makes its decisions.
- See E2
background-color:: red
- 3) The study only uses open-source Java projects from GitHub, but **real-world software is much more diverse.**
- See E3
background-color:: red
- For your results:
- 1) The machine learning models used are outdated -- this study relies on traditional machine learning models (e.g., Decision Tree, Random Forest, kNN, Naïve Bayes, Logistic Regression), which are quite basic and outdated compared to modern deep learning or transformer-based models. These models struggle with complex relationships in code and often require manual feature engineering, which might miss hidden patterns. Code-based tasks (e.g., predicting performance bottlenecks) benefit from deep learning models like graph neural networks (GNNs), transformer-based models (e.g., CodeBERT, GraphCodeBERT), or CNN/RNN hybrids, which learn rich code representations without manual feature extraction. --> **You can refer to this paper: "HGNN4Perf: Detecting Performance Optimization Opportunities via Hypergraph Neural Network"**
- [HGNN4Perf: Detecting Performance Optimization Opportunities via Hypergraph Neural Network](https://dl-acm-org.univaq.idm.oclc.org/doi/pdf/10.1145/3671016.3671405)
- *See E4*
background-color:: red
- 2) The study assumes that performance test quality is linked to code coverage, but this is not necessarily true. Just because a test executes a method does not mean it effectively measures performance bottlenecks. A method might be covered in a performance test but not be performance-critical. Performance tests should focus on CPU-intensive, I/O-heavy, or memory-consuming methods, but the study does not validate whether the covered methods impact actual system performance. **Instead of only analyzing coverage, you can integrate dynamic profiling tools like async-profiler, JFR (Java Flight Recorder), or flame graphs to measure which methods actually impact performance.**
- *See E5*
background-color:: red
- 3) **The dataset is limited also.** The dataset consists of 28 open-source Java projects, but this is not representative of large-scale enterprise applications. **Suggestions are expanding the dataset to include Enterprise applications to check if results hold for large-scale systems, and multi-language projects (e.g., performance tests in C++, Python, Rust).**
- *See E3 and E6*
background-color:: red
- ## Reviewer 3
collapsed:: true
- Summary of the Paper
*In this paper, the authors extended previous work by examining 28 open-source projects to discover the limitations of performance testing suites. Specifically, the paper studied the coverage of the performance testing suites, the nature of the source code, and the time it takes to execute the testing suite with the intent to identify static code features/patterns to mark code blocks that should seek performance attention.*
- Strengths of the Paper
1. The paper tackles an interesting and significant topic performance testing.
2. The paper is well-structured, flowing smoothly, and being easy to follow.
3. The paper poses insightful and a well-defined set of research questions. It explores how performance tests cover code compared to functional tests and investigates if static code features can predict test coverage, an area that has been less explored in the literature. The paper analyzes 28 open-source Java projects and using both static and dynamic analyses.
4. The authors have prepared a publicly available replication package.
- Weaknesses of the Paper
- 1. On page 5, the authors state that performance tests are "62 times larger than those of JUnit tests." **Could this be attributed to the nature of performance testing itself, which generally takes longer? What other metrics could be used to evaluate the cost of testing?**
- Think about (Vittorio will take it)
- 2. If the study exclusively focuses on Java projects, **can the findings be generalized to projects developed in other programming languages?** Although the paper discusses "threats to validity," it may be beneficial to explain the rationale for investigating Java specifically regarding performance issues. **Additionally, the title of the paper might be considered an overstatement since it primarily examines open-source Java projects.**
- We put Java to the title
- 3. Is code coverage an appropriate metric for evaluating the quality of performance testing suites? The paper acknowledges that the validity of code coverage as a measure of code quality is debatable for functional testing, let alone performance testing. The few arguments supporting the use of code coverage in performance testing appear relevant to very specific types of bugs and environments (e.g., synchronization bugs in production environments), making it difficult to generalize their applicability. **The authors may consider further justify the choice of code coverage as a measure for performance testing suites.**
- Luca will expand
- 4. The effort to "explore the potential relationships between source code metrics and the likelihood of the analyzed source code being covered by performance tests" is commendable, but is it sensible? Even if there is a connection, could it lead to adverse outcomes? **For instance, if a code block is likely to be covered by performance tests, might this encourage developers to shift their focus elsewhere?**
- Da capire. Forse voleva dire che visto che c'e' performance test lo sviluppatore potrebbe essere portato a sviluppare test di altro
- 5. Regarding test coverage, if a method (m) appears in the call stack (either directly or indirectly), the method m is deemed covered by test t. Is this granularity too coarse? **Certain bugs may only manifest if particular segments of code are executed with specific input values.**
- Vedere E7
- 6. On page 17, the paper states that k-fold cross-validation is not used, and instead, the dataset was split into 80% for training and 20% for testing. **What is the difference between this approach and 5-fold cross-validation? How does this method better address "class imbalance"?**
- Vedere E8
- 7. **How were the open-source Java projects selected**? Do they cover a wide scope of application domains that are known to have performance issues (hence the performance testing)? Does it make sense to put the selected projects in the same pool for the analysis? If we group the project into different domains, what conclusions can we draw?
- Vedere E6
- Recheck the distribution of projects across domains.
- 8. The comparison between performance and functional testing coverage is anticipated (as acknowledged early in the paper). **While the effort to provide comparative data is appreciated, is there a specific point the paper aims to convey?** Would comparing among non-functional requirements (such as performance, robustness, security, etc.) make more sense?
- Vittorio will think about this
- 9. Related to the previous question, the same reasoning applies to comparing execution times. **What underlies the longer duration of performance testing?** Is it due to the nature of the testing (e.g., endurance testing)? **The observations in RQ2 are noteworthy but also widely recognized.**
- Luca will revise.
- Smaller Notes
- 1. On page 5, when we talk about something that is smaller in quality, we may want to say the performance testing code coverage is a quarter of the Junit tests instead of "4 times lower than the JUnit tests one". The same can be said for page 27.
- 2. The paper used dynamic analysis for JUnit callstacks for the model training, **does that contradict the goal of using static features to associate performance code**?
- Why not using other dynamic features considering that we are collecting JUnit callstacks?
-
- 3. As per Wikipedia, "In computing, a benchmark is the act of running a computer program, a set of programs, or other operations, in order to assess the relative performance of an object, normally by running several standard tests and trials against it." In section 4, the paper mentioned analyzing 2,190 **JMH benchmarks, do you mean test cases?**
- Add the definition of JMH benchmarks
- 4. Observations in **RQ3. Could that be a byproduct of the selected projects?**
- External threats to validity. This is true for any "data" based approach