logseq/pages/@Evolutionary prompt engineering for cost-effective code generation with large language models.md

tags:: #zotero date:: 2017 title:: @Evolutionary prompt engineering for cost-effective code generation with large language models item-type:: journalArticle original-title:: Evolutionary prompt engineering for cost-effective code generation with large language models language:: en library-catalog:: Zotero links:: Local library, Web library

• Abstract
  • Large Language Models (LLMs) have seen increasing use in various software development tasks, especially in code generation. The most advanced recent methods attempt to incorporate feedback from code execution into prompts to help guide LLMs in generating correct code in an iterative process. While effective, these methods could be costly due to numerous interactions with the LLM and extensive token usage. To address this issue, we propose an alternative approach named Evolutionary Prompt Engineering for Code (EPiC), which leverages a lightweight evolutionary algorithm to refine the original prompts into improved versions that generate high-quality code, with minimal interactions with the LLM. Our evaluation against state-of-the-art (SOTA) LLM-based code generation agents shows that EPiC not only achieves up to 6% improvement in pass@k but is also 2–10 times more cost-effective than the baselines.

• Attachments

  • PDF {{zotero-imported-file 9RXNTKG3, "2017 - Evolutionary prompt engineering for cost-effective code generation with large language models.pdf"}}

• Notes

  • I'm reviewing a research paper and I took the following notes:

    Annotazioni

    (16/5/2025, 11:18:31)

    • “Cost-Effective Code Generation” (“Evolutionary prompt engineering for cost-effective code generation with large language models”, 2017, p. 1) #5fb236

    • “feedback from code execution into prompts” (“Evolutionary prompt engineering for cost-effective code generation with large language models”, 2017, p. 1) #ffd400 What kind of code execution information are fed into prompts?

    • “6% improvement in pass@k” (“Evolutionary prompt engineering for cost-effective code generation with large language models”, 2017, p. 1) #5fb236

    • “cost-effective” (“Evolutionary prompt engineering for cost-effective code generation with large language models”, 2017, p. 1) #ffd400 what kind of cost is considered here?

    • “MBPP dataset” (“Evolutionary prompt engineering for cost-effective code generation with large language models”, 2017, p. 1) #ffd400 what is it?

    • “LATS” (“Evolutionary prompt engineering for cost-effective code generation with large language models”, 2017, p. 1) #ffd400 what is it?

    • “One key limitation of these approaches is that the initial prompt fed to the LLM is often suboptimal.” (“Evolutionary prompt engineering for cost-effective code generation with large language models”, 2017, p. 1) #a28ae5

    • “If we had an approach that systematically improves the prompt with minimal LLM calls, we could find the optimal prompt.” (“Evolutionary prompt engineering for cost-effective code generation with large language models”, 2017, p. 1) #5fb236 this is s a kind of teacher-student pattern.

    • “if we automatically mutate the initial prompt, systematically test it, and then finalize an improved version, we can converge on correct code more efficiently.” (“Evolutionary prompt engineering for cost-effective code generation with large language models”, 2017, p. 1) #5fb236

    • “Evolutionary Prompt Engineering for Code (EPiC)” (“Evolutionary prompt engineering for cost-effective code generation with large language models”, 2017, p. 1) #a28ae5

    • “to refine prompts in a structured and costeffective way.” (“Evolutionary prompt engineering for cost-effective code generation with large language models”, 2017, p. 1) #e56eee

    • “augmented” (“Evolutionary prompt engineering for cost-effective code generation with large language models”, 2017, p. 1) #ffd400 why necessarely augmented? withb respect to what?

    • “The mutation is carried out using two approaches: one utilizes an LLM 1 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 Conference’17, July 2017, Washington, DC, USA Anon. 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 guided by a prompt that specifies how to perform the mutation, and the other employs vector embeddings of words to find and replace similar words.” (“Evolutionary prompt engineering for cost-effective code generation with large language models”, 2017, p. 1) #a28ae5

    • “pass@k metric,” (“Evolutionary prompt engineering for cost-effective code generation with large language models”, 2017, p. 2) #a28ae5

    • “as introduced in [3], for accuracy (effectiveness), which estimates the probability that at least one of the top k generated code samples is correct” (“Evolutionary prompt engineering for cost-effective code generation with large language models”, 2017, p. 2) #a28ae5

    • “Additional Token Usage per Solved Problem” (“Evolutionary prompt engineering for cost-effective code generation with large language models”, 2017, p. 2) #e56eee

    • “code generation task from the cost-effectiveness perspective” (“Evolutionary prompt engineering for cost-effective code generation with large language models”, 2017, p. 2) #e56eee

    • “novel framework EPiC that leverages a lightweight evolutionary algorithm to evolve the original prompts toward better ones that produce high-quality code” (“Evolutionary prompt engineering for cost-effective code generation with large language models”, 2017, p. 2) #e56eee

    • “cost-effectiveness of EPiC in code generation compared to baseline methods” (“Evolutionary prompt engineering for cost-effective code generation with large language models”, 2017, p. 2) #e56eee

    • “we briefly explain prompt engineering for LLMs in general and report the most related work in the context of prompt engineering of LLM for code.” (“Evolutionary prompt engineering for cost-effective code generation with large language models”, 2017, p. 2) #5fb236

    • “Prompt engineering refers to the process of designing and refining prompts to achieve desired outcomes when using LLMs1” (“Evolutionary prompt engineering for cost-effective code generation with large language models”, 2017, p. 2) #5fb236

    • “approaches without training” (“Evolutionary prompt engineering for cost-effective code generation with large language models”, 2017, p. 2) #5fb236

    • “reasoning and logic” (“Evolutionary prompt engineering for cost-effective code generation with large language models”, 2017, p. 2) #5fb236

    • “reducing hallucination” (“Evolutionary prompt engineering for cost-effective code generation with large language models”, 2017, p. 2) #5fb236

    • “evolutionary-based methods” (“Evolutionary prompt engineering for cost-effective code generation with large language models”, 2017, p. 2) #5fb236

    • “reasoning and logic-based methods” (“Evolutionary prompt engineering for cost-effective code generation with large language models”, 2017, p. 2) #5fb236

    • “Few-shot prompting [1] provides limited examples to guide understanding but requires more tokens, making it less practical for long texts and susceptible to bias from example selection.” (“Evolutionary prompt engineering for cost-effective code generation with large language models”, 2017, p. 2) #5fb236

    • “EvoPrompt [34] automates this by iteratively refining prompts using mutation and crossover.” (“Evolutionary prompt engineering for cost-effective code generation with large language models”, 2017, p. 2) #a28ae5

    • “Using Parsel, they break down algorithmic tasks into structured descriptions written in natural language, then explore various combinations of function implementations using tests” (“Evolutionary prompt engineering for cost-effective code generation with large language models”, 2017, p. 2) #5fb236

    • “Reflexion [29] is a reinforcement-based framework in which language agents learn from linguistic feedback.” (“Evolutionary prompt engineering for cost-effective code generation with large language models”, 2017, p. 2) #5fb236

    • “EPiC is the first evolutionary-based prompt engineering method for code generation.” (“Evolutionary prompt engineering for cost-effective code generation with large language models”, 2017, p. 3) #e56eee

    • “It employs a lightweight process to identify the optimal solution in a cost-effective manner.” (“Evolutionary prompt engineering for cost-effective code generation with large language models”, 2017, p. 3) #e56eee

    • “EPiC utilizes a local embedding function to implement mutation operators on text to reduce the cost of iterative prompt engineering for code generation.” (“Evolutionary prompt engineering for cost-effective code generation with large language models”, 2017, p. 3) #a28ae5

    • “guides the search over iterations using the fitness function in Section 4.4, which helps in finding the best prompts.” (“Evolutionary prompt engineering for cost-effective code generation with large language models”, 2017, p. 3) #a28ae5

    • “without requiring gradient information” (“Evolutionary prompt engineering for cost-effective code generation with large language models”, 2017, p. 3) #a28ae5

    • “where the pass rate of generated code is a discrete and non-differentiable function.” (“Evolutionary prompt engineering for cost-effective code generation with large language models”, 2017, p. 3) #e56eee

    • “initialization, evaluation, selection, variation, and iteration.” (“Evolutionary prompt engineering for cost-effective code generation with large language models”, 2017, p. 3) #2ea8e5

    • “evaluating fitness is relatively straightforward” (“Evolutionary prompt engineering for cost-effective code generation with large language models”, 2017, p. 3) #ffd400 How can you say that? You are supposed to measure the quality of the output produced with the prompt at hand, and this can require some time/effort! #question

    • “optimized” (“Evolutionary prompt engineering for cost-effective code generation with large language models”, 2017, p. 3) #ffd400 Do you know if each iteration improves the previous ones? #question

    • “sufficient to generate the correct implementatio” (“Evolutionary prompt engineering for cost-effective code generation with large language models”, 2017, p. 3) #ffd400 Here we have "sufficient" and "correct" that need to be concretize. They refer crucial aspects that need to be materialized otherwise it's not clear how they can be achieved in practice.

    • “In” (“Evolutionary prompt engineering for cost-effective code generation with large language models”, 2017, p. 3) #ffd400 Maybe you can start the paragraph by saying, "By referring to Fig. 2 ..."

    • “muCandidates ← chooseCandidates (candidates, N − 1)” (“Evolutionary prompt engineering for cost-effective code generation with large language models”, 2017, p. 4) #5fb236

    • “test cases.” (“Evolutionary prompt engineering for cost-effective code generation with large language models”, 2017, p. 4) #ffd400 It seems that also test cases are generated. What if generated test cases are semantically wrong? This is a possible "point of failure" of the whole process.

    • “If any test case fails on the code, we continue with the EPE phase. If all test cases pass, we report the generated code as the final answer” (“Evolutionary prompt engineering for cost-effective code generation with large language models”, 2017, p. 4) #ffd400 See my previous comment. Test cases might pass because tests are wrong if those are not given as input and instead also part of the generation.

    • “Note that test cases can be provided in various ways. One approach is to use developer-provided test cases for evaluation, while another is to generate test cases using the LLM.” (“Evolutionary prompt engineering for cost-effective code generation with large language models”, 2017, p. 4) #ffd400 See my previous point. This can be an important point of failure of the process.

    • “we also opted to use LLMs for test case generation to ensure a fully automated approach, assuming no developer-provided test cases. To ensure the functional correctness of these test cases, we validated them by parsing their Abstract Syntax Trees (AST)” (“Evolutionary prompt engineering for cost-effective code generation with large language models”, 2017, p. 4) #ffd400 This is a critical point. It is not clear how it works. When test cases are generated, starting from what? From the same prompt used to generate code? #question

    • “prompts for code and test case generation” (“Evolutionary prompt engineering for cost-effective code generation with large language models”, 2017, p. 4) #ffd400 These look like completely disconnected. How can you be sure that test cases are semantically connected to the wanted generated code? #question

    • “assert add3Numbers(1, 2, 3) == 6 assert add3Numbers(-1, 2, 3) == 4 assert add3Numbers(1, -2, 3) == 2 assert add3Numbers(1, 2, -3) == 0 assert add3Numbers(-3, -2, -1) == -6 assert add3Numbers(0, 0, 0) == 0” (“Evolutionary prompt engineering for cost-effective code generation with large language models”, 2017, p. 4) #ffd400 I'm skeptical about the semantic correctness of the generated test cases.

    • “we generate multiple prompts by modifying the initial prompt using an LLM agent. Generating this prompt population forms an important part of our evolutionary algorithm” (“Evolutionary prompt engineering for cost-effective code generation with large language models”, 2017, p. 5) #ffd400 What are the mutation operators to generate multiple prompts?

    • “We define a fitness function based on the ratio of test cases passed.” (“Evolutionary prompt engineering for cost-effective code generation with large language models”, 2017, p. 5) #ffd400 See my previous content.

    • “randomly mutates” (“Evolutionary prompt engineering for cost-effective code generation with large language models”, 2017, p. 5) #ffd400 This triggers many questions related to the way mutants are generated.

    • “adding the elite prompt to the pool of mutated prompts.” (“Evolutionary prompt engineering for cost-effective code generation with large language models”, 2017, p. 5) #5fb236

    • “LLM_as_mutator , we provide LLMs with predefined instructions on how to implement the prompt mutation.” (“Evolutionary prompt engineering for cost-effective code generation with large language models”, 2017, p. 6) #2ea8e5

    • “alter the prompts by substituting words with their synonyms (sim_words_as_mutator )” (“Evolutionary prompt engineering for cost-effective code generation with large language models”, 2017, p. 6) #2ea8e5

    • “You are a mutation tool. This is a Python function and its description. Please change the description by enhancing its clarity and comprehensibility for sophisticated language models. Please put the changed description between #Explanation and #End. Use at most 600 words.” (“Evolutionary prompt engineering for cost-effective code generation with large language models”, 2017, p. 6) #ffd400 Sometimes LLMs even fail in producing outputs respecting the given number of words limit!?!?!?

    • “selected words” (“Evolutionary prompt engineering for cost-effective code generation with large language models”, 2017, p. 6) #ffd400 What's the criteria that is used to select words to changed for mutating prompts? #question

    • “[21],” (“Evolutionary prompt engineering for cost-effective code generation with large language models”, 2017, p. 6) #ff6666 It should be [3], isn't it?

    • “For a single problem, this metric estimates the probability that at least one of the top k samples is correct” (“Evolutionary prompt engineering for cost-effective code generation with large language models”, 2017, p. 7) #2ea8e5

    • “(Pm − Pb ) × N” (“Evolutionary prompt engineering for cost-effective code generation with large language models”, 2017, p. 7) #ffd400 The denominator of the ATSP equation can be 0, isn't it?

    • “Total token usage for method m and baseline b” (“Evolutionary prompt engineering for cost-effective code generation with large language models”, 2017, p. 7) #5fb236

    • “pass@1” (“Evolutionary prompt engineering for cost-effective code generation with large language models”, 2017, p. 7) #5fb236 So this is the probability that the first generated sample is correct.

    • “The goal of the EPiC framework is to optimize the fitness function F based on the provided test cases T by identifying the optimal input x within the prompt space X.” (“Evolutionary prompt engineering for cost-effective code generation with large language models”, 2017, p. 7) #5fb236

    • “their cost is not disproportionately high.” (“Evolutionary prompt engineering for cost-effective code generation with large language models”, 2017, p. 7) #ffd400 What do you mean? What's the threshold that you considered for this? #question

    • “baseline configurations” (“Evolutionary prompt engineering for cost-effective code generation with large language models”, 2017, p. 7) #ffd400 Can you say something more about about the used baseline configurations? Are they coming from the original works and presented as those that permit the corresponding approaches at their best? #question

    • “To enable a fairer comparison among the agents, we adopted a consistent test generation approach to produce test cases rather than using the ground-truth tests for internal evaluation.” (“Evolutionary prompt engineering for cost-effective code generation with large language models”, 2017, p. 8) #5fb236

    • “For the ablation study in Section 6,” (“Evolutionary prompt engineering for cost-effective code generation with large language models”, 2017, p. 8) #ffd400 The reference to the performed ablation study suddenly appears without a proper motivation and introduction.

    • “temperature of 0.0” (“Evolutionary prompt engineering for cost-effective code generation with large language models”, 2017, p. 8) #ffd400 How this temperature setting is connected to the temperature value given as 0.6 for the initial population builder phase described in Sec 3.2.1? #question

    • “RQ1: How does EPiC perform across different SOTA LLMs?” (“Evolutionary prompt engineering for cost-effective code generation with large language models”, 2017, p. 8) #ffd400 Do you have any estimation of the costs of the considered baselines? #question

    • “achieves higher functional correctness at the cost of increased token usage.” (“Evolutionary prompt engineering for cost-effective code generation with large language models”, 2017, p. 8) #5fb236

    • “RQ2: How does EPiC compare to other iterative-based agents?” (“Evolutionary prompt engineering for cost-effective code generation with large language models”, 2017, p. 9) #ffd400 What is the LLM that is used to get the results shown in Table 3?

    • “To alleviate the threat, we have shown that the LLM-generated tests are not that far off from the results using the original (developer-written) tests.” (“Evolutionary prompt engineering for cost-effective code generation with large language models”, 2017, p. 10) #ffd400 How can you be sure about that? #question

    COnsider that those that are tagget with #5fb236 are just highlights, those that are tagged with #e56eee and #a28ae5 are imporant sentences. Please pay attention instead to the notes that are tagged with #ffd400. Those that are tagged with #ff6666 are typos or errors. Could you please draft a review by organizing it as follows:

    SUMMARY: Just a few sentence to summarize the work

    STRENGHTS:

    WEAKNESSES:

    COMMENTS: Organize the notes with respect to the following criteria:

    Novelty

    Rigor

    Relevance (of the contribution)

    Verifiability and Transparency

    Presentation

    And then add a Detailed Comments section to report the notets that contain issues or typos.