logseq/pages/@Generation of Unit Tests for Test-Driven Development using Large Language Models.md

tags:: #zotero title:: @Generation of Unit Tests for Test-Driven Development using Large Language Models item-type:: journalArticle original-title:: Generation of Unit Tests for Test-Driven Development using Large Language Models language:: en authors:: Nathanael Yao, Juergen Dingel, Ali Tizghadam library-catalog:: Zotero links:: Local library, Web library

• Abstract
  • Test-Driven Development (TDD) is a process that has been shown to reduce software defects yet is not always adopted in industrial software. Case studies on industrial teams have shown up to a 50% reduction in defect density when using TDD compared to ad-hoc unit testing. TDD can also reduce the cost of debugging software by finding code defects earlier and can contribute to better software design. In this paper, we present our ongoing work on an approach that uses generative AI to generate unit tests to facilitate test-driven development. Given a high-level goal, the approach generates formalized requirements in the form of a goal model, which is then used to generate unit tests in a test folder. We describe a current prototype implementation and show initial results for example high-level goals provided by our industrial partner.

• Attachments

  • PDF {{zotero-imported-file IU5ZGYX4, "Yao et al. - Generation of Unit Tests for Test-Driven Development using Large Language Models.pdf"}}

• Notes

  • I'm reviewing a research paper and I took the following notes:

    Annotations

    (30/7/2025, 12:06:11)

    • “Test-Driven Development (TDD) is a process that has been shown to reduce software defects yet is not always adopted in industrial software.” (Yao et al., p. 1) #5fb236

    • “50% reduction in defect density when using TDD compared to ad-hoc unit testing.” (Yao et al., p. 1) #5fb236

    • “cost of debugging software by finding code defects earlier and can contribute to better software design.” (Yao et al., p. 1) #5fb236

    • “ongoing work on an approach that uses generative AI to generate unit tests to facilitate test-driven development” (Yao et al., p. 1) #e56eee

    • “goal model,” (Yao et al., p. 1) #a28ae5

    • “which is then used to generate unit tests in a test folder.” (Yao et al., p. 1) #a28ae5

    • “During Test-Driven Development (TDD), developers start by creating a failing test that matches the given goals for the implementation.” (Yao et al., p. 1) #e56eee

    • “When the implementation code is written, developers write the minimum amount of code to pass the unit tests written in the previous step.” (Yao et al., p. 1) #e56eee

    • “Unit tests are a form of software testing that evaluates individual components or functions in isolation to ensure their correctness.” (Yao et al., p. 1) #5fb236

    • “unit tests are often written after the implementation code, using TDD can be more effective in reducing software defects.” (Yao et al., p. 1) #5fb236

    • “However, despite its benefits, a small subset of developers choose to not write tests [2].” (Yao et al., p. 1) #5fb236

    • “Using Large Language Models (LLMs) to generate unit tests offers an approach that may be less time consuming for developers.” (Yao et al., p. 1) #5fb236

    • “use of LLMs for the generation of unit tests from user-given goals to facilitate TDD.” (Yao et al., p. 1) #5fb236

    • “ongoing work on an approach and prototype for the generation of unit tests for TDD.” (Yao et al., p. 1) #a28ae5

    • “many diverse user-given goals, but the development of our prototype was heavily focused on generating unit tests for the network domain” (Yao et al., p. 1) #a28ae5

    • “We did not conduct an extensive evaluation, instead relying on manual checks and feedback from our industry partner.” (Yao et al., p. 1) #ffd400 This is ok for a workshop paper.

    • “II. UNIT TESTS IN TEST-DRIVEN DEVELOPMENT” (Yao et al., p. 1) #5fb236

    • “During Test-Driven development (TDD), unit tests are created based on requirements for new functionality before any implementation code is written” (Yao et al., p. 1) #5fb236

    • “Red-Green-Refactor: developers write a few failing unit tests (red), then write implementation code to pass the test cases (green) and refactor the code, making sure all the tests still pass (refactor).” (Yao et al., p. 1) #5fb236

    • “Unit tests are added incrementally as new features are added, and all previous unit tests must pass as new unit tests and implementation code are added.” (Yao et al., p. 1) #a28ae5

    • “Moreover, unit tests in TDD can also help reveal issues with the requirements.” (Yao et al., p. 1) #5fb236

    • “The purpose of writing unit tests is to be able to isolate a unit and verify its correctness.” (Yao et al., p. 1) #a28ae5

    • “R1: Given input that identifies the high-level goal g that the user is planning to write implementation code for, the approach and prototype implementation should for any goal g: a) refine the high-level goal g into a set of sub-goals and b) generate unit tests of sufficient quality for goal g using the refined requirements. R2: The approach should represent the requirements in a way that generates good quality unit tests. R3: The approach should work for arbitrary user goals g.” (Yao et al., p. 2) #ffd400 Section 5 that presents the approach is not effective in explaining how these steps work. In particular, it is not clear how the human involvement is managed, how the iterative nature of the process is supported, how human feedback is considered by the subsequent steps of the test driven process.

    • “Most recent work on test generation uses the implementation code as input. For instance, Amazon Q developer [6] and GitHub Copilot [7] provide a chatbot that allows users to generate unit tests with appropriate mocks for specified methods,” (Yao et al., p. 2) #a28ae5

    • “TESTPILOT, which used the function signature as well as other available information such as documentation and usage examples to generate unit tests with GAI” (Yao et al., p. 2) #5fb236

    • “A previous study found correctness issues in some unit tests generated by ChatGPT but finds that the correct tests are of comparable quality to tests written by developers [3]” (Yao et al., p. 2) #5fb236

    • “Our approach differs from these works as it does not use the implementation code as input and can be used to support TDD.” (Yao et al., p. 2) #5fb236

    • “The MAPE-K loop mechanism has been used to assist in LLM-based goal model generation by continuously adding goals during the loops.” (Yao et al., p. 2) #5fb236

    • “process assigns different roles to the LLM, allowing it to act as different domain experts when performing different tasks within the MAPE-K loop” (Yao et al., p. 2) #5fb236

    • “This also showed promising results, but included steps that required human decision making” (Yao et al., p. 2) #5fb236

    • “GAI has been used as a collaborative tool for developers during TDD showing promising results but requiring developer supervision.” (Yao et al., p. 2) #5fb236

    • “Many techniques are used to try and improve the quality of LLM-generated answer” (Yao et al., p. 2) #5fb236

    • “Our approach is novel, as it allows for few-shot examples of unit tests to be given in our unit test generation step that are aligned with the basic actions selected in the generated goal models.” (Yao et al., p. 2) #e56eee

    • “Our industrial partner tasked us with creating a tool that generates unit tests from user-defined goals.” (Yao et al., p. 2) #5fb236

    • “Generating a goal model skeleton” (Yao et al., p. 2) #2ea8e5

    • “filling in the values in the goal model skeleton using GenAI” (Yao et al., p. 2) #2ea8e5

    • “unit test generation” (Yao et al., p. 2) #2ea8e5

    • “1) Pool of Basic Actions” (Yao et al., p. 2) #2ea8e5

    • “Fig. 1: Algorithm for generating unit tests” (Yao et al., p. 3) #ffd400 I'm not sure this algorithm as presented in the paper is effective in the presentation. I would find an alternative way to present the process that in the end consists of three main steps.

    • “Fig. 2: Snippet of the pool of basic actions used to build our goal models” (Yao et al., p. 3) #ffd400 Who defines them? Are they domain/application specific?

    • “It contains two template variables: a goal g representing the user-defined goal” (Yao et al., p. 3) #ffd400 Are goal models written in a specific syntax/format?

    • “3) Querying the LLM: Next, the LLM is queried using the goal model skeleton prompt and few shot examples to generate a goal model skeleton (line 2 in Figure 1).” (Yao et al., p. 3) #ffd400 This is not clear. The LLM is queried with a goal model skeleton prompt to generate a goal model skeleton. Different parts of the paper are affected by presentation issues that reduce the readability of the work. It is not clear what is general and what is application specific that is valid only for the application at hand and made working with ad-hoc steps.

    • “validate, get, return” (Yao et al., p. 3) #5fb236

    • “C. Unit Test Generation 1) Few Shot Examples: The list of chosen basic actions generated in the previous step will be used to select example unit tests to aid in unit test generation (line 9 of Figure 1). Each basic action in our pool has a corresponding set of example requirements and unit tests. We tried to include at least two few-shot examples of requirement and unit test pairs for each basic action in our pool. Figure 7 is a snippet of a few-shot example for the ’lookup’ basic action (Jinja2 statements are shown in blue).” (Yao et al., p. 4) #ffd400 This step is not clear neither. First of all it is necessary to clarify that the work is Python specific. Then, it is not clear how unit tests for the basic actions can be "composed" to generate unit tests that are specific for the application at hand. Moreover, what's the granularity of the wanted unit tests?

    • “However, it struggled with more complex examples and on examples where the high-level goals were less well defined.” (Yao et al., p. 5) #ffd400 This is not surprising to me. I'm really missing in the approach a clear specification of the granularity of the unit tests, what they are supposed to test, etc.

    • “Our approach takes a high-level goal, refines it into a goal model constructed using a pool of basic actions, and generates unit tests based on the subgoals in the goal model” (Yao et al., p. 5) #5fb236

    • “Despite the mixed results, our approach is novel as it is able to provide few-shot examples of relevant unit tests for many different high-level goals.” (Yao et al., p. 5) #5fb236

    • “Chain-of-thought prompting could be used to attempt to improve the LLMs reasoning capability.” (Yao et al., p. 5) #5fb236

    COnsider that those that are tagget with #5fb236 are just highlights, those that are tagged with #e56eee and #a28ae5 are imporant sentences. Please pay attention instead to the notes that are tagged with #ffd400. Those that are tagged with #ff6666 are typos or errors. Could you please draft a review by organizing it as follows: SUMMARY: Just a few sentence to summarize the work COMMENTS: Organize the notes especially those that contain issues or typos. Please do not make an extrause of bullet points. Make the comments fluent without being too verbose