tags:: [[#zotero]]
date:: 2026
title:: @A kNN-Based Recommender System for Test Case Reuse in Agile Software Development
item-type:: [[journalArticle]]
original-title:: A kNN-Based Recommender System for Test Case Reuse in Agile Software Development
language:: en
authors:: [[Anonymous Author]]
library-catalog:: Zotero
links:: [Local library](zotero://select/library/items/NUL9BMH7), [Web library](https://www.zotero.org/users/1039502/items/NUL9BMH7)

- [[Abstract]]
	- Context: Agile testing poses unique challenges, including short development cycles, evolving requirements, and the need to rapidly execute and update test suites. Efficient reuse of test cases can help address these demands, but remains difficult due to fragmented documentation and evolving repositories. Objective: This study proposes a Recommender System (RecSys) to support test case reuse in agile development contexts by leveraging historical user stories and a domain-specific taxonomy. Method: We implemented a K-Nearest Neighbors (KNN)-based RecSys to retrieve user stories similar to a given target and recommend their associated test cases. The algorithm’s transparent logic enables traceable and interpretable recommendations. The evaluation followed a multi-method design: first, an offline experiment with 217 user stories and 1077 test cases from two companies identified the best configuration based on recall, precision, and F-measure. Then, this configuration was applied in an online case study involving two agile projects to assess real-world impact on test reuse and suite completeness. Results: The best offline configuration achieved a recall of 67.5%, precision of 37.8%, and F-measure of 51.0%. In the online phase, 65.09% of the developed test cases aligned with existing assets, and the RecSyS increased the projects’ test suites by 38.78%. Conclusion: These results indicate that our taxonomy-based RecSyS effectively supports test case reuse in agile settings, achieving solid results both offline and in practice. Its transparent logic, minimal infrastructure needs, and alignment with agile workflows make it a lightweight and practical alternative to more complex reuse solutions.
- ### Attachments
	- [PDF](zotero://select/library/items/SM9JG55B) {{zotero-imported-file SM9JG55B, "Author - 2026 - A kNN-Based Recommender System for Test Case Reuse in Agile Software Development.pdf"}}
- ### Notes
	- # Annotations 
	   (18/09/2025, 01:58:43)
	  
	  - “Agile testing poses unique challenges, including short development cycles, evolving requirements, and the need to rapidly execute and update test suites.” (Author, 2026, p. 1) #5fb236
	  * *
	  
	   
	  
	  - “Efficient reuse of test cases can help address these demands, but remains difficult due to fragmented documentation and evolving repositories” (Author, 2026, p. 1) #e56eee
	  * *
	  
	   
	  
	  - “This study proposes a Recommender System (RecSys) to support test case reuse in agile development contexts by leveraging historical user stories and a domain-specific taxonomy” (Author, 2026, p. 1) #e56eee
	  * *
	  
	   
	  
	  - “We implemented a K-Nearest Neighbors (KNN)-based RecSys to retrieve user stories similar to a given target and recommend their associated test cases.” (Author, 2026, p. 1) #5fb236
	  * *
	  
	   
	  
	  - “first, an offline experiment with 217 user stories and 1077 test cases from two companies identified the best configuration based on recall, precision, and F-measure.” (Author, 2026, p. 1) #5fb236
	  * *
	  
	   
	  
	  - “this configuration was applied in an online case study involving two agile projects to assess real-world impact on test reuse and suite completeness.” (Author, 2026, p. 1) #5fb236
	  * *
	  
	   
	  
	  - “These results indicate that our taxonomy-based RecSyS effectively supports test case reuse in agile settings, achieving solid results both offline and in practice.” (Author, 2026, p. 1) #5fb236
	  * *
	  
	   
	  
	  - “developers must navigate increasingly complex information spaces—often spending disproportionate time seeking relevant artifacts at the expense of value-generating work” (Author, 2026, p. 1) #5fb236
	  * *
	  
	   
	  
	  - “Recommender Systems (RecSys) for software engineering have emerged as tools that assist developers with various tasks—ranging from code reuse to effective bug reporting—and aim to reduce cognitive load and improve productivity [” (Author, 2026, p. 1) #5fb236
	  * *
	  
	   
	  
	  - “reusing validated test cases offers a promising way to improve test coverage and efficiency without increasing effort” (Author, 2026, p. 1) #5fb236
	  * *
	  
	   
	  
	  - “When organizations accumulate substantial testing knowledge across features or projects, test reuse can reduce redundancy and accelerate software testing.” (Author, 2026, p. 1) #a28ae5
	  * *
	  
	   
	  
	  - “Industrial test repositories often suffer from inconsistent documentation, lack of structure, and frequent changes” (Author, 2026, p. 1) #a28ae5
	  * *
	  
	   
	  
	  - “However, these techniques often require complex infrastructure—semantic models, embeddings, or deep learning pipelines, and typically operate at the code level or support regression testing, rather than proactive reuse at the user story level in agile workflows.” (Author, 2026, p. 1) #5fb236
	  * *
	  
	   
	  
	  - “we developed a RecSys tailored to agile environments, particularly for early sprint planning and user story development.” (Author, 2026, p. 1) #5fb236
	  * *
	  
	   
	  
	  - “The system leverages two underutilized resources: structured taxonomies and historical user stories.” (Author, 2026, p. 1) #5fb236
	  * *
	  
	   
	  
	  - “referred to as offline validation in RecSys” (Author, 2026, p. 2) #5fb236
	  * *
	  
	   
	  
	  - “First, we used the data from 13 industrial projects from one organization, encompassing 217 user stories and 1077 distinct test cases, to identify the most effective RecSys configuration.” (Author, 2026, p. 2) #5fb236
	  * *
	  
	   
	  
	  - “in-situ case study in two projects of the same organization to assess the RecSys’s practical impact on agile test case development” (Author, 2026, p. 2) #5fb236
	  * *
	  
	   
	  
	  - “test cases linked to semantically similar user stories in agile development.” (Author, 2026, p. 2) #e56eee
	  * *
	  
	   
	  
	  - “textual similarity between requirements implies similarity in downstream artifacts (e.g., code and tests).” (Author, 2026, p. 2) #ffd400
	  *This is a crucial assumption that can give place to some threats to validity. It's ok if it is mentioned in that section. *
	  
	   
	  
	  - “Abbas et al. [2] provide empirical support for this assumption. Their study evaluated six NLP models—ranging from lexical to deep learning-based approaches—and found a moderately positive correlation between the similarity of requirements and the similarity of their associated software components” (Author, 2026, p. 2) #a28ae5
	  *That's important. *
	  
	   
	  
	  - “that semantically similar user stories are likely to be associated with reusable test cases.” (Author, 2026, p. 2) #e56eee
	  * *
	  
	   
	  
	  - “This assumption is further supported by broader work in requirements traceability and retrieval [24, 55], which shows that requirements labeled with semantic metadata—such as purpose, stakeholder, or behavior—can be more effectively retrieved. Collectively, these studies justify our design choice to use structured user story similarity as a proxy for identifying reusable test cases.” (Author, 2026, p. 2) #5fb236
	  * *
	  
	   
	  
	  - “Web Information Systems (WIS)” (Author, 2026, p. 2) #5fb236
	  * *
	  
	   
	  
	  - “Module, Operation” (Author, 2026, p. 2) #5fb236
	  * *
	  
	   
	  
	  - “. Each user story can be labeled with one or more such pairs, enabling multi-label classification and more expressive comparisons across stories.” (Author, 2026, p. 2) #5fb236
	  * *
	  
	   
	  
	  - “Any structured labeling scheme that enables meaningful comparison across requirements, especially with respect to the type of feature involved, can support our approach. This flexibility allows adaptation to other domains or evolving taxonomies.” (Author, 2026, p. 2) #5fb236
	  * *
	  
	   
	  
	  - “These solutions require mature projects with rich defect histories and focus on prioritization rather than early-stage reuse” (Author, 2026, p. 3) #e56eee
	  * *
	  
	   
	  
	  - “LLMs represent a more recent shift from reuse to generation.” (Author, 2026, p. 3) #e56eee
	  * *
	  
	   
	  
	  - “Empirical evaluations show that models like GPT-4 and Gemini can generate high-coverage unit tests given source code and examples” (Author, 2026, p. 3) #5fb236
	  * *
	  
	   
	  
	  - “However, these black-box approaches raise reproducibility and traceability concerns and do not capitalize on validated, existing test assets within an organization.” (Author, 2026, p. 3) #e56eee
	  * *
	  
	   
	  
	  - “We propose a RecSys that classifies User Stories using a validated taxonomy and applies a transparent similarity metric to retrieve reusable test cases, before coding begins.” (Author, 2026, p. 3) #5fb236
	  * *
	  
	   
	  
	  - “The RecSys receives as input a target requirement and produces a list of potential test cases to be reused by testers through the data analysis from previously executed projects or ones under execution.” (Author, 2026, p. 3) #5fb236
	  * *
	  
	   
	  
	  - “we assumed that test cases are requirements-driven and that, if the requirements are similar, the test cases are also similar, following the findings of Abbas et al. [2].” (Author, 2026, p. 3) #5fb236
	  * *
	  
	   
	  
	  - “In particular, we employed the WIS taxonomy proposed by Dilorenzo et al. [15], detailed in Section 2.1. For instance, if we have the set of user stories given by U = {U S1, U S2, · · · , U Sk }, where k is the total number of user stories, and that each U Sk ∈ U are of the type (Authentication, First login), we can infer that they are all similar, even if they have different descriptions.” (Author, 2026, p. 3) #ffd400
	  *This is a strong assumption. Similar user stories have similar tests is strong for me. Even minor variations of code implementing similar user stories can make tests different. It is interesting to see, what's the usage of the recommended tests. Are they used as starting point or as reference implementation to look at while developing the actual test cases? *
	  
	   
	  
	  - “Acceptance Criteria (AC)” (Author, 2026, p. 3) #e56eee
	  * *
	  
	   
	  
	  - “Let f be a utility function that measures the usefulness of a test case t to a user story u, i.e., f : U × T −→ R,” (Author, 2026, p. 3) #e56eee
	  * *
	  
	   
	  
	  - “Recommender: responsible for analyzing characteristics vectors, generated by the data transformer, calculating the similarity between target User Story and retrieved User Stories, and recommending test cases developed for the most similar User Stories ranked by their relevance.” (Author, 2026, p. 3) #5fb236
	  * *
	  
	   
	  
	  - “uses the collaborative filtering based on user attributes [23], and performs the recommendation of test cases from the nearest neighbors, to the target User Story.” (Author, 2026, p. 4) #ffd400
	  * *
	  
	   
	  
	  - “K Nearest Neighbors (KNN) algorithm” (Author, 2026, p. 4) #5fb236
	  * *
	  
	   
	  
	  - “We validated the proposed Recommender System using a multimethod design that combined a retrospective experiment using historical project data (offline validation) and a pilot in situ deployment in industry projects (online validation). This strategy allowed us to examine both the internal performance of the algorithm under controlled conditions and its practical usefulness in real-world agile teams.” (Author, 2026, p. 5) #5fb236
	  *This is important and well motivated. *
	  
	   
	  
	  - “After identifying the best-performing configuration, we proceeded to the online phase, where we deployed the RecSys in two ongoing software projects at an industrial partner.” (Author, 2026, p. 5) #5fb236
	  * *
	  
	   
	  
	  - “We did not include a comparative evaluation with existing test case recommenders such as those proposed by Bera et al.[9] or Ge and Liu[20], as these approaches rely on fundamentally different infrastructures, including structured requirement modeling, collaborative filtering, and knowledge graphs. Our method depends on a domain-specific taxonomy and requires semantic labeling of user stories, making direct comparisons non-trivial and potentially misleading. Furthermore, no benchmark dataset exists that would support a controlled comparison under equivalent conditions. Instead, we focused on assessing the internal effectiveness of our approach through multiple configurations and evaluating its external utility through industrial deployment.” (Author, 2026, p. 5) #ffd400
	  *Ok even though this is a potential bias that needs to be discussed. *
	  
	   
	  
	  - “4.1.1 Training Dataset Definition” (Author, 2026, p. 5) #2ea8e5
	  * *
	  
	   
	  
	  - “The lack of RecSys users (User Stories) and items (test cases) data problem is called User Cold-Start and Item Cold-Start, respectivel” (Author, 2026, p. 5) #5fb236
	  * *
	  
	   
	  
	  - “we collected User Stories data, AC, and test cases from project databases of two software development companies that execute projects with Scrum” (Author, 2026, p. 5) #5fb236
	  * *
	  
	   
	  
	  - “4.1.2 Experiment Design.” (Author, 2026, p. 5) #2ea8e5
	  * *
	  
	   
	  
	  - “support Scrum teams in reusing test cases and enhancing the projects’ test suites.” (Author, 2026, p. 6) #5fb236
	  * *
	  
	   
	  
	  - “RQ” (Author, 2026, p. 6) #ff6666
	  *RQ_on *
	  
	   
	  
	  - “RQon1: To what extent does the test cases recommender increase the test suite completeness for agile projects?” (Author, 2026, p. 6) #ffd400
	  *In contrast to the situation when no support is used? What's the baseline? *
	  
	   
	  
	  - “RQon2: To what extent does the test cases recommender enable the test cases reuse for agile projects?” (Author, 2026, p. 6) #ffd400
	  * *
	  
	   
	  
	  - “RQon1, we wanted to identify whether the test suite resulted from the recommendations and test cases developed by the project testers are more complete than the original test suite,” (Author, 2026, p. 6) #5fb236
	  * *
	  
	   
	  
	  - “RQon2, our goal was to identify if the RecSys provides testers with test cases that they would develop, that is, if it anticipates test cases that would be developed and, thus, allow the tester to reuse them.” (Author, 2026, p. 6) #5fb236
	  * *
	  
	   
	  
	  - “two projects under development from one organization” (Author, 2026, p. 6) #ffd400
	  *Can you say something more on the selected projects and on the organization? (business domain, kinds of projects, etc) *
	  
	   
	  
	  - “hen, we compared the recommended test cases with the ones created by the project’s testers. After RecSys execution, we discussed the results with the project’s testers and classified the test cases as follows:” (Author, 2026, p. 6) #ffd400
	  *This was a qualitative comparison, isn't it? *
	  
	   
	  
	  - “Accepted” (Author, 2026, p. 6) #ffd400
	  *Acepted how? I'm not sure it can be used as it is recommended. I guess some refinements/changes were required, isn't it? *
	  
	   
	  
	  - “a higher number of neighbors, whereas the recall value does not present a significant change.” (Author, 2026, p. 7) #5fb236
	  * *
	  
	   
	  
	  - “he configuration Euclidean distance, K = 3, heuristic = yes outperformed all 59 alternatives (p < 0.05, Nemenyi post-hoc). It achieved 67.5 % recall, 37.8 % precision, and an F2 score of 0.510. This setting is therefore carried forward to the online deployment.” (Author, 2026, p. 7) #5fb236
	  * *
	  
	   
	  
	  - “Additionally, we analyzed the relevance of the accepted test cases, based on the testers’ perception, to identify the tool’s efficiency. Thus, Figure 5 shows the relevance percentages.” (Author, 2026, p. 8) #5fb236
	  * *
	  
	   
	  
	  - “Figure 5: Accepted test cases relevance.” (Author, 2026, p. 8) #ffd400
	  *This is not very good, isn't it? It seems that among the accepted tests, those with low relevance was 47.5% and only 25.4% was considered to be highly relevant... *
	  
	   
	  
	  - “It is worth noting that the percentage of test cases reused is not directly related to the percentage of effort reduction for the development of these reused test cases” (Author, 2026, p. 8) #5fb236
	  * *
	  
	   
	  
	  - “Hence, two-thirds of manually authored tests could have been reused with minor adaptation effort.” (Author, 2026, p. 8) #5fb236
	  * *
	  
	   
	  
	  - “Impact on test reuse and suite completeness. Across 41 User Stories, the RecSys recommended 334 test cases. Of these, 132 matched tests already written by the testers (i.e., reused), and 79 were accepted as new, valuable additions—resulting in a 38.8% increase in the test suite.2 Altogether, approximately 63% of the recommended test cases were considered useful, showing that the system can meaningfully support reuse and enrich coverage with minimal overhead.” (Author, 2026, p. 9) #5fb236
	  * *
	  
	   
	  
	  - “Limitations and failure cases. Despite its benefits, the RecSys was not without limitations. The most common reason for rejection (86% of rejected cases) was incompatibility with implicit or project-specific business rules not captured by the taxonomy. This highlights an important improvement opportunity: augmenting user stories with additional semantic tags or rule-level metadata could help filter out false positives and improve recommendation relevance.” (Author, 2026, p. 9) #5fb236
	  * *
	  
	   
	  
	  - “Applicability and generalization. Although the evaluation was conducted in the context of Web Information Systems, the approach depends primarily on the availability of a domain-specific taxonomy. Substituting this taxonomy for others—tailored to different application domains—requires minimal adaptation effort. Our findings support the broader claim that lightweight, taxonomy-guided kNN-based recommenders can effectively promote early-stage test reuse in agile development, without requiring complex infrastructure or large-scale data.” (Author, 2026, p. 9) #5fb236
	  * *
	  
	   
	  
	  - “7 THREATS TO VALIDITY This section discusses the potential threats to the validity of our study, structured according to the categories proposed by Wohlin et al. [53]. Conclusion Validity. Although the offline dataset contains 217 user stories and 1 077 test cases, some taxonomy categories (e.g., Authentication) are under-represented. When a category appears fewer than K times in a fold, the RecSys cannot identify the required neighbors. We mitigated this by discarding such folds and reporting the effective sample size, but the reduced counts lower statistical power for those categories. Further, we compared 60 algorithm configurations, increasing the risk of Type-I error. We therefore applied the Friedman test followed by a post-hoc Nemenyi procedure and interpreted results at α = 0.05. Nonetheless, borderline-significant differences should be interpreted with caution. Finally, data Normality was rejected by Shapiro–Wilk, justifying non-parametric analysis, but other assumptions (e.g., independence of folds) remain. Cross-validation folds can be correlated if neighboring user stories share many test cases; future work could use nested cross-validation to reduce this risk. Internal Validity.User stories were manually mapped to taxonomy categories by the first author and later double-checked by two co-authors. Disagreements (6 %) were resolved by discussion. Nevertheless, latent bias may persist. A kappa statistic was not calculated; future work will involve independent raters and report inter-rater reliability. Additionally, during the online study, the Scrum team adopted a new test-management plug-in that automatically duplicates certain test cases. We controlled for this by excluding auto-generated duplicates from reuse counts, yet residual confounding is possible. Finally, testers may become more proficient over successive sprints, independently improving reuse. Because the pilot lasted only seven sprints (41 stories) and no time–series analysis was performed, maturation effects cannot be fully ruled out. Construct Validity. We relied on precision, recall, and F-measure (β = 2) for offline evaluation, assuming they correlate with tester utility. However, these metrics do not account for the effort required to inspect false positives or adapt reused tests. Complementary measures such as Mean Reciprocal Rank or time-to-acceptance could provide a richer picture. Further, we evaluated only kNN with six distance functions for viability reasons. Other families of RecSys (e.g., matrix factorization, BERT/LLM embeddings) were not explored. This limits construct coverage; future replications should vary the underlying model. Finally, similarity is computed on binary vectors derived from our taxonomy. If two stories are semantically similar but fall into different categories, similarity is underestimated. Conversely, stories in the same category but semantically dissimilar may inflate similarity. Introducing textual embeddings or hierarchical weights could alleviate this threat. External Validity. Training data came from two Brazilian software vendors; the online pilot ran in one of them. Both organizations develop information systems for web/mobile platforms. Results may differ in safety-critical domains (e.g., avionics) or where test documentation is richer/poorer. Further, uur dataset size (1 077 tests) is modest compared with repositories maintained by large enterprises. kNN’s computational cost is O (nK); additional engineering (e.g., ANN indexing) may be required for million-scale test suites. Additionally, the approach presumes (i) user stories conform to the chosen taxonomy, (ii) each story links to at least K = 3 test cases in the historical repository, and (iii) similar wording or semantics persist across projects. Organizations lacking these conditions may observe lower recall or face a cold-start problem. Finally, both pilot teams already emphasized test automation and collective ownership of test artifacts. In cultures where testers and developers work in silos, the willingness to reuse peer artifacts might differ.” (Author, 2026, p. 10) #ffd400
	  *Overall, I liked the paper. It presents a simple even though effective approach to support the discovery and reuse of testing. The performed experiments are encouraging even though additional investigations need to be n order to support developers in adopting the retrieved test. The efforts required to use the recommended test need to be further investigated. *