218 lines
36 KiB
Markdown
218 lines
36 KiB
Markdown
type:: [[REVIEWS]]
|
||
tags::
|
||
year:: 2025
|
||
venue:: [[ICSE]]
|
||
full-title:: Evolutionary Prompt Engineering for Cost-Effective Code Generation with Large Language Models
|
||
date-start:: [[16-05-2025]] - 11:20
|
||
date-submitted::
|
||
external-links::
|
||
status:: [[DONE]]
|
||
deadline-submission::
|
||
file:: [[@Evolutionary prompt engineering for cost-effective code generation with large language models]]
|
||
parent::
|
||
todoist:: https://app.todoist.com/app/task/1640-evolutionary-prompt-engineering-for-cost-effective-code-generation-with-lar-6XXr6Q34Vg79qGqc
|
||
|
||
- ### [[Highlights]]
|
||
collapsed:: true
|
||
- # **Annotazioni**
|
||
collapsed:: true
|
||
|
||
(16/5/2025, 11:18:31)
|
||
- “Cost-Effective Code Generation” ([“Evolutionary prompt engineering for cost-effective code generation with large language models”, 2017, p. 1](zotero://select/library/items/VKR2ARTC)) ([pdf](zotero://open-pdf/library/items/9RXNTKG3?page=1&annotation=BBCTY4QS)) #5fb236
|
||
- “feedback from code execution into prompts” ([“Evolutionary prompt engineering for cost-effective code generation with large language models”, 2017, p. 1](zotero://select/library/items/VKR2ARTC)) ([pdf](zotero://open-pdf/library/items/9RXNTKG3?page=1&annotation=ZPNRMY34)) #ffd400
|
||
|
||
**What kind of code execution information are fed into prompts?**
|
||
- “6% improvement in pass@k” ([“Evolutionary prompt engineering for cost-effective code generation with large language models”, 2017, p. 1](zotero://select/library/items/VKR2ARTC)) ([pdf](zotero://open-pdf/library/items/9RXNTKG3?page=1&annotation=CD5UJJVF)) #5fb236
|
||
- “cost-effective” ([“Evolutionary prompt engineering for cost-effective code generation with large language models”, 2017, p. 1](zotero://select/library/items/VKR2ARTC)) ([pdf](zotero://open-pdf/library/items/9RXNTKG3?page=1&annotation=Y2E9ZYH4)) #ffd400
|
||
|
||
**what kind of cost is considered here?**
|
||
- “MBPP dataset” ([“Evolutionary prompt engineering for cost-effective code generation with large language models”, 2017, p. 1](zotero://select/library/items/VKR2ARTC)) ([pdf](zotero://open-pdf/library/items/9RXNTKG3?page=1&annotation=FX88I9DD)) #ffd400
|
||
|
||
**what is it?**
|
||
- “LATS” ([“Evolutionary prompt engineering for cost-effective code generation with large language models”, 2017, p. 1](zotero://select/library/items/VKR2ARTC)) ([pdf](zotero://open-pdf/library/items/9RXNTKG3?page=1&annotation=XRYJZVAI)) #ffd400
|
||
|
||
**what is it?**
|
||
- “One key limitation of these approaches is that the initial prompt fed to the LLM is often suboptimal.” ([“Evolutionary prompt engineering for cost-effective code generation with large language models”, 2017, p. 1](zotero://select/library/items/VKR2ARTC)) ([pdf](zotero://open-pdf/library/items/9RXNTKG3?page=1&annotation=IFJCQZME)) #a28ae5
|
||
- “If we had an approach that systematically improves the prompt with minimal LLM calls, we could find the optimal prompt.” ([“Evolutionary prompt engineering for cost-effective code generation with large language models”, 2017, p. 1](zotero://select/library/items/VKR2ARTC)) ([pdf](zotero://open-pdf/library/items/9RXNTKG3?page=1&annotation=TJDG37T4)) #5fb236
|
||
|
||
**this is s a kind of teacher-student pattern.**
|
||
- “if we automatically mutate the initial prompt, systematically test it, and then finalize an improved version, we can converge on correct code more efficiently.” ([“Evolutionary prompt engineering for cost-effective code generation with large language models”, 2017, p. 1](zotero://select/library/items/VKR2ARTC)) ([pdf](zotero://open-pdf/library/items/9RXNTKG3?page=1&annotation=5L9DH23Y)) #5fb236
|
||
- “Evolutionary Prompt Engineering for Code (EPiC)” ([“Evolutionary prompt engineering for cost-effective code generation with large language models”, 2017, p. 1](zotero://select/library/items/VKR2ARTC)) ([pdf](zotero://open-pdf/library/items/9RXNTKG3?page=1&annotation=LAH9ME4J)) #a28ae5
|
||
- “to refine prompts in a structured and costeffective way.” ([“Evolutionary prompt engineering for cost-effective code generation with large language models”, 2017, p. 1](zotero://select/library/items/VKR2ARTC)) ([pdf](zotero://open-pdf/library/items/9RXNTKG3?page=1&annotation=U522KQDM)) #e56eee
|
||
- “augmented” ([“Evolutionary prompt engineering for cost-effective code generation with large language models”, 2017, p. 1](zotero://select/library/items/VKR2ARTC)) ([pdf](zotero://open-pdf/library/items/9RXNTKG3?page=1&annotation=S7J5NBC6)) #ffd400
|
||
|
||
**why necessarely augmented? withb respect to what?**
|
||
- “The mutation is carried out using two approaches: one utilizes an LLM 1 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 Conference’17, July 2017, Washington, DC, USA Anon. 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 guided by a prompt that specifies how to perform the mutation, and the other employs vector embeddings of words to find and replace similar words.” ([“Evolutionary prompt engineering for cost-effective code generation with large language models”, 2017, p. 1](zotero://select/library/items/VKR2ARTC)) ([pdf](zotero://open-pdf/library/items/9RXNTKG3?page=1&annotation=TSZT6B3J)) #a28ae5
|
||
- “pass@k metric,” ([“Evolutionary prompt engineering for cost-effective code generation with large language models”, 2017, p. 2](zotero://select/library/items/VKR2ARTC)) ([pdf](zotero://open-pdf/library/items/9RXNTKG3?page=2&annotation=HSP2ZTKR)) #a28ae5
|
||
- “as introduced in [3], for accuracy (effectiveness), which estimates the probability that at least one of the top k generated code samples is correct” ([“Evolutionary prompt engineering for cost-effective code generation with large language models”, 2017, p. 2](zotero://select/library/items/VKR2ARTC)) ([pdf](zotero://open-pdf/library/items/9RXNTKG3?page=2&annotation=VIJCEMRG)) #a28ae5
|
||
- “Additional Token Usage per Solved Problem” ([“Evolutionary prompt engineering for cost-effective code generation with large language models”, 2017, p. 2](zotero://select/library/items/VKR2ARTC)) ([pdf](zotero://open-pdf/library/items/9RXNTKG3?page=2&annotation=W8PL2T7G)) #e56eee
|
||
- “code generation task from the cost-effectiveness perspective” ([“Evolutionary prompt engineering for cost-effective code generation with large language models”, 2017, p. 2](zotero://select/library/items/VKR2ARTC)) ([pdf](zotero://open-pdf/library/items/9RXNTKG3?page=2&annotation=KEUX9WLG)) #e56eee
|
||
- “novel framework EPiC that leverages a lightweight evolutionary algorithm to evolve the original prompts toward better ones that produce high-quality code” ([“Evolutionary prompt engineering for cost-effective code generation with large language models”, 2017, p. 2](zotero://select/library/items/VKR2ARTC)) ([pdf](zotero://open-pdf/library/items/9RXNTKG3?page=2&annotation=8NDJURJZ)) #e56eee
|
||
- “cost-effectiveness of EPiC in code generation compared to baseline methods” ([“Evolutionary prompt engineering for cost-effective code generation with large language models”, 2017, p. 2](zotero://select/library/items/VKR2ARTC)) ([pdf](zotero://open-pdf/library/items/9RXNTKG3?page=2&annotation=PJRNIUYP)) #e56eee
|
||
- “we briefly explain prompt engineering for LLMs in general and report the most related work in the context of prompt engineering of LLM for code.” ([“Evolutionary prompt engineering for cost-effective code generation with large language models”, 2017, p. 2](zotero://select/library/items/VKR2ARTC)) ([pdf](zotero://open-pdf/library/items/9RXNTKG3?page=2&annotation=MRIJGBEL)) #5fb236
|
||
- “Prompt engineering refers to the process of designing and refining prompts to achieve desired outcomes when using LLMs1” ([“Evolutionary prompt engineering for cost-effective code generation with large language models”, 2017, p. 2](zotero://select/library/items/VKR2ARTC)) ([pdf](zotero://open-pdf/library/items/9RXNTKG3?page=2&annotation=7MEHYGFW)) #5fb236
|
||
- “approaches without training” ([“Evolutionary prompt engineering for cost-effective code generation with large language models”, 2017, p. 2](zotero://select/library/items/VKR2ARTC)) ([pdf](zotero://open-pdf/library/items/9RXNTKG3?page=2&annotation=KLPN9U2Y)) #5fb236
|
||
- “reasoning and logic” ([“Evolutionary prompt engineering for cost-effective code generation with large language models”, 2017, p. 2](zotero://select/library/items/VKR2ARTC)) ([pdf](zotero://open-pdf/library/items/9RXNTKG3?page=2&annotation=RXWAQGCP)) #5fb236
|
||
- “reducing hallucination” ([“Evolutionary prompt engineering for cost-effective code generation with large language models”, 2017, p. 2](zotero://select/library/items/VKR2ARTC)) ([pdf](zotero://open-pdf/library/items/9RXNTKG3?page=2&annotation=M27G584B)) #5fb236
|
||
- “evolutionary-based methods” ([“Evolutionary prompt engineering for cost-effective code generation with large language models”, 2017, p. 2](zotero://select/library/items/VKR2ARTC)) ([pdf](zotero://open-pdf/library/items/9RXNTKG3?page=2&annotation=X44EQEAI)) #5fb236
|
||
- “reasoning and logic-based methods” ([“Evolutionary prompt engineering for cost-effective code generation with large language models”, 2017, p. 2](zotero://select/library/items/VKR2ARTC)) ([pdf](zotero://open-pdf/library/items/9RXNTKG3?page=2&annotation=FUZAYYDS)) #5fb236
|
||
- “Few-shot prompting [1] provides limited examples to guide understanding but requires more tokens, making it less practical for long texts and susceptible to bias from example selection.” ([“Evolutionary prompt engineering for cost-effective code generation with large language models”, 2017, p. 2](zotero://select/library/items/VKR2ARTC)) ([pdf](zotero://open-pdf/library/items/9RXNTKG3?page=2&annotation=XE24NGIH)) #5fb236
|
||
- “EvoPrompt [34] automates this by iteratively refining prompts using mutation and crossover.” ([“Evolutionary prompt engineering for cost-effective code generation with large language models”, 2017, p. 2](zotero://select/library/items/VKR2ARTC)) ([pdf](zotero://open-pdf/library/items/9RXNTKG3?page=2&annotation=BK5THEH9)) #a28ae5
|
||
- “Using Parsel, they break down algorithmic tasks into structured descriptions written in natural language, then explore various combinations of function implementations using tests” ([“Evolutionary prompt engineering for cost-effective code generation with large language models”, 2017, p. 2](zotero://select/library/items/VKR2ARTC)) ([pdf](zotero://open-pdf/library/items/9RXNTKG3?page=2&annotation=A2YIKWVR)) #5fb236
|
||
- “Reflexion [29] is a reinforcement-based framework in which language agents learn from linguistic feedback.” ([“Evolutionary prompt engineering for cost-effective code generation with large language models”, 2017, p. 2](zotero://select/library/items/VKR2ARTC)) ([pdf](zotero://open-pdf/library/items/9RXNTKG3?page=2&annotation=VFWI69JI)) #5fb236
|
||
- “EPiC is the first evolutionary-based prompt engineering method for code generation.” ([“Evolutionary prompt engineering for cost-effective code generation with large language models”, 2017, p. 3](zotero://select/library/items/VKR2ARTC)) ([pdf](zotero://open-pdf/library/items/9RXNTKG3?page=3&annotation=QLMMT62M)) #e56eee
|
||
- “It employs a lightweight process to identify the optimal solution in a cost-effective manner.” ([“Evolutionary prompt engineering for cost-effective code generation with large language models”, 2017, p. 3](zotero://select/library/items/VKR2ARTC)) ([pdf](zotero://open-pdf/library/items/9RXNTKG3?page=3&annotation=P2UXNUGX)) #e56eee
|
||
- “EPiC utilizes a local embedding function to implement mutation operators on text to reduce the cost of iterative prompt engineering for code generation.” ([“Evolutionary prompt engineering for cost-effective code generation with large language models”, 2017, p. 3](zotero://select/library/items/VKR2ARTC)) ([pdf](zotero://open-pdf/library/items/9RXNTKG3?page=3&annotation=KIPA5GP6)) #a28ae5
|
||
- “guides the search over iterations using the fitness function in Section 4.4, which helps in finding the best prompts.” ([“Evolutionary prompt engineering for cost-effective code generation with large language models”, 2017, p. 3](zotero://select/library/items/VKR2ARTC)) ([pdf](zotero://open-pdf/library/items/9RXNTKG3?page=3&annotation=T6CJ5HG3)) #a28ae5
|
||
- “without requiring gradient information” ([“Evolutionary prompt engineering for cost-effective code generation with large language models”, 2017, p. 3](zotero://select/library/items/VKR2ARTC)) ([pdf](zotero://open-pdf/library/items/9RXNTKG3?page=3&annotation=DR5UG8HH)) #a28ae5
|
||
- “where the pass rate of generated code is a discrete and non-differentiable function.” ([“Evolutionary prompt engineering for cost-effective code generation with large language models”, 2017, p. 3](zotero://select/library/items/VKR2ARTC)) ([pdf](zotero://open-pdf/library/items/9RXNTKG3?page=3&annotation=B4P6IK2J)) #e56eee
|
||
- “initialization, evaluation, selection, variation, and iteration.” ([“Evolutionary prompt engineering for cost-effective code generation with large language models”, 2017, p. 3](zotero://select/library/items/VKR2ARTC)) ([pdf](zotero://open-pdf/library/items/9RXNTKG3?page=3&annotation=IL8H3U3V)) #2ea8e5
|
||
- “evaluating fitness is relatively straightforward” ([“Evolutionary prompt engineering for cost-effective code generation with large language models”, 2017, p. 3](zotero://select/library/items/VKR2ARTC)) ([pdf](zotero://open-pdf/library/items/9RXNTKG3?page=3&annotation=C79XF3H9)) #ffd400
|
||
|
||
**How can you say that? You are supposed to measure the quality of the output produced with the prompt at hand, and this can require some time/effort! **#**question**
|
||
- “optimized” ([“Evolutionary prompt engineering for cost-effective code generation with large language models”, 2017, p. 3](zotero://select/library/items/VKR2ARTC)) ([pdf](zotero://open-pdf/library/items/9RXNTKG3?page=3&annotation=MQN5EPPF)) #ffd400
|
||
|
||
**Do you know if each iteration improves the previous ones? **#**question**
|
||
- “sufficient to generate the correct implementatio” ([“Evolutionary prompt engineering for cost-effective code generation with large language models”, 2017, p. 3](zotero://select/library/items/VKR2ARTC)) ([pdf](zotero://open-pdf/library/items/9RXNTKG3?page=3&annotation=AZYSURPA)) #ffd400
|
||
|
||
**Here we have "sufficient" and "correct" that need to be concretize. They refer crucial aspects that need to be materialized otherwise it's not clear how they can be achieved in practice.**
|
||
- “In” ([“Evolutionary prompt engineering for cost-effective code generation with large language models”, 2017, p. 3](zotero://select/library/items/VKR2ARTC)) ([pdf](zotero://open-pdf/library/items/9RXNTKG3?page=3&annotation=ZFMN6A3N)) #ffd400
|
||
|
||
**Maybe you can start the paragraph by saying, "By referring to Fig. 2 ..."**
|
||
- “muCandidates ← chooseCandidates (candidates, N − 1)” ([“Evolutionary prompt engineering for cost-effective code generation with large language models”, 2017, p. 4](zotero://select/library/items/VKR2ARTC)) ([pdf](zotero://open-pdf/library/items/9RXNTKG3?page=4&annotation=T3YX3SFT)) #5fb236
|
||
- “test cases.” ([“Evolutionary prompt engineering for cost-effective code generation with large language models”, 2017, p. 4](zotero://select/library/items/VKR2ARTC)) ([pdf](zotero://open-pdf/library/items/9RXNTKG3?page=4&annotation=3RS95BKY)) #ffd400
|
||
|
||
**It seems that also test cases are generated. What if generated test cases are semantically wrong? This is a possible "point of failure" of the whole process.**
|
||
- “If any test case fails on the code, we continue with the EPE phase. If all test cases pass, we report the generated code as the final answer” ([“Evolutionary prompt engineering for cost-effective code generation with large language models”, 2017, p. 4](zotero://select/library/items/VKR2ARTC)) ([pdf](zotero://open-pdf/library/items/9RXNTKG3?page=4&annotation=BCCR5QPJ)) #ffd400
|
||
|
||
**See my previous comment. Test cases might pass because tests are wrong if those are not given as input and instead also part of the generation.**
|
||
- “Note that test cases can be provided in various ways. One approach is to use developer-provided test cases for evaluation, while another is to generate test cases using the LLM.” ([“Evolutionary prompt engineering for cost-effective code generation with large language models”, 2017, p. 4](zotero://select/library/items/VKR2ARTC)) ([pdf](zotero://open-pdf/library/items/9RXNTKG3?page=4&annotation=UJ8IPJQT)) #ffd400
|
||
|
||
**See my previous point. This can be an important point of failure of the process.**
|
||
- “we also opted to use LLMs for test case generation to ensure a fully automated approach, assuming no developer-provided test cases. To ensure the functional correctness of these test cases, we validated them by parsing their Abstract Syntax Trees (AST)” ([“Evolutionary prompt engineering for cost-effective code generation with large language models”, 2017, p. 4](zotero://select/library/items/VKR2ARTC)) ([pdf](zotero://open-pdf/library/items/9RXNTKG3?page=4&annotation=4TIAUVYT)) #ffd400
|
||
|
||
**This is a critical point. It is not clear how it works. When test cases are generated, starting from what? From the same prompt used to generate code? **#**question**
|
||
- “prompts for code and test case generation” ([“Evolutionary prompt engineering for cost-effective code generation with large language models”, 2017, p. 4](zotero://select/library/items/VKR2ARTC)) ([pdf](zotero://open-pdf/library/items/9RXNTKG3?page=4&annotation=VBXKF5CP)) #ffd400
|
||
|
||
**These look like completely disconnected. How can you be sure that test cases are semantically connected to the wanted generated code? **#**question**
|
||
- “assert add3Numbers(1, 2, 3) == 6 assert add3Numbers(-1, 2, 3) == 4 assert add3Numbers(1, -2, 3) == 2 assert add3Numbers(1, 2, -3) == 0 assert add3Numbers(-3, -2, -1) == -6 assert add3Numbers(0, 0, 0) == 0” ([“Evolutionary prompt engineering for cost-effective code generation with large language models”, 2017, p. 4](zotero://select/library/items/VKR2ARTC)) ([pdf](zotero://open-pdf/library/items/9RXNTKG3?page=4&annotation=KB79CV58)) #ffd400
|
||
|
||
**I'm skeptical about the semantic correctness of the generated test cases.**
|
||
- “we generate multiple prompts by modifying the initial prompt using an LLM agent. Generating this prompt population forms an important part of our evolutionary algorithm” ([“Evolutionary prompt engineering for cost-effective code generation with large language models”, 2017, p. 5](zotero://select/library/items/VKR2ARTC)) ([pdf](zotero://open-pdf/library/items/9RXNTKG3?page=5&annotation=MD39LDYQ)) #ffd400
|
||
|
||
**What are the mutation operators to generate multiple prompts?**
|
||
- “We define a fitness function based on the ratio of test cases passed.” ([“Evolutionary prompt engineering for cost-effective code generation with large language models”, 2017, p. 5](zotero://select/library/items/VKR2ARTC)) ([pdf](zotero://open-pdf/library/items/9RXNTKG3?page=5&annotation=9N7RIBH3)) #ffd400
|
||
|
||
**See my previous content.**
|
||
- “randomly mutates” ([“Evolutionary prompt engineering for cost-effective code generation with large language models”, 2017, p. 5](zotero://select/library/items/VKR2ARTC)) ([pdf](zotero://open-pdf/library/items/9RXNTKG3?page=5&annotation=GUZI3ZI2)) #ffd400
|
||
|
||
**This triggers many questions related to the way mutants are generated.**
|
||
- “adding the elite prompt to the pool of mutated prompts.” ([“Evolutionary prompt engineering for cost-effective code generation with large language models”, 2017, p. 5](zotero://select/library/items/VKR2ARTC)) ([pdf](zotero://open-pdf/library/items/9RXNTKG3?page=5&annotation=CC5M7KIP)) #5fb236
|
||
- “LLM_as_mutator , we provide LLMs with predefined instructions on how to implement the prompt mutation.” ([“Evolutionary prompt engineering for cost-effective code generation with large language models”, 2017, p. 6](zotero://select/library/items/VKR2ARTC)) ([pdf](zotero://open-pdf/library/items/9RXNTKG3?page=6&annotation=XHUWHY3H)) #2ea8e5
|
||
- “alter the prompts by substituting words with their synonyms (sim_words_as_mutator )” ([“Evolutionary prompt engineering for cost-effective code generation with large language models”, 2017, p. 6](zotero://select/library/items/VKR2ARTC)) ([pdf](zotero://open-pdf/library/items/9RXNTKG3?page=6&annotation=I6CJ6AMA)) #2ea8e5
|
||
- “You are a mutation tool. This is a Python function and its description. Please change the description by enhancing its clarity and comprehensibility for sophisticated language models. Please put the changed description between #Explanation and #End. Use at most 600 words.” ([“Evolutionary prompt engineering for cost-effective code generation with large language models”, 2017, p. 6](zotero://select/library/items/VKR2ARTC)) ([pdf](zotero://open-pdf/library/items/9RXNTKG3?page=6&annotation=D7UV7IFM)) #ffd400
|
||
|
||
**Sometimes LLMs even fail in producing outputs respecting the given number of words limit!?!?!?**
|
||
- “selected words” ([“Evolutionary prompt engineering for cost-effective code generation with large language models”, 2017, p. 6](zotero://select/library/items/VKR2ARTC)) ([pdf](zotero://open-pdf/library/items/9RXNTKG3?page=6&annotation=V7KQ3QWD)) #ffd400
|
||
|
||
**What's the criteria that is used to select words to changed for mutating prompts? **#**question**
|
||
- “[21],” ([“Evolutionary prompt engineering for cost-effective code generation with large language models”, 2017, p. 6](zotero://select/library/items/VKR2ARTC)) ([pdf](zotero://open-pdf/library/items/9RXNTKG3?page=6&annotation=IUE8MAX7)) #ff6666
|
||
|
||
**It should be **[**3**]**, isn't it?**
|
||
- “For a single problem, this metric estimates the probability that at least one of the top k samples is correct” ([“Evolutionary prompt engineering for cost-effective code generation with large language models”, 2017, p. 7](zotero://select/library/items/VKR2ARTC)) ([pdf](zotero://open-pdf/library/items/9RXNTKG3?page=7&annotation=WEYYH3PP)) #2ea8e5
|
||
- “(Pm − Pb ) × N” ([“Evolutionary prompt engineering for cost-effective code generation with large language models”, 2017, p. 7](zotero://select/library/items/VKR2ARTC)) ([pdf](zotero://open-pdf/library/items/9RXNTKG3?page=7&annotation=N64455W4)) #ffd400
|
||
|
||
**The denominator of the ATSP equation can be 0, isn't it?**
|
||
- “Total token usage for method m and baseline b” ([“Evolutionary prompt engineering for cost-effective code generation with large language models”, 2017, p. 7](zotero://select/library/items/VKR2ARTC)) ([pdf](zotero://open-pdf/library/items/9RXNTKG3?page=7&annotation=NLKB4H2S)) #5fb236
|
||
- “pass@1” ([“Evolutionary prompt engineering for cost-effective code generation with large language models”, 2017, p. 7](zotero://select/library/items/VKR2ARTC)) ([pdf](zotero://open-pdf/library/items/9RXNTKG3?page=7&annotation=3WMIIF8E)) #5fb236
|
||
|
||
**So this is the probability that the first generated sample is correct.**
|
||
- “The goal of the EPiC framework is to optimize the fitness function F based on the provided test cases T by identifying the optimal input x within the prompt space X.” ([“Evolutionary prompt engineering for cost-effective code generation with large language models”, 2017, p. 7](zotero://select/library/items/VKR2ARTC)) ([pdf](zotero://open-pdf/library/items/9RXNTKG3?page=7&annotation=RN8E7V2S)) #5fb236
|
||
- “their cost is not disproportionately high.” ([“Evolutionary prompt engineering for cost-effective code generation with large language models”, 2017, p. 7](zotero://select/library/items/VKR2ARTC)) ([pdf](zotero://open-pdf/library/items/9RXNTKG3?page=7&annotation=MG99SGNT)) #ffd400
|
||
|
||
**What do you mean? What's the threshold that you considered for this? **#**question**
|
||
- “baseline configurations” ([“Evolutionary prompt engineering for cost-effective code generation with large language models”, 2017, p. 7](zotero://select/library/items/VKR2ARTC)) ([pdf](zotero://open-pdf/library/items/9RXNTKG3?page=7&annotation=4GPSA7KT)) #ffd400
|
||
|
||
**Can you say something more about about the used baseline configurations? Are they coming from the original works and presented as those that permit the corresponding approaches at their best? **#**question**
|
||
- “To enable a fairer comparison among the agents, we adopted a consistent test generation approach to produce test cases rather than using the ground-truth tests for internal evaluation.” ([“Evolutionary prompt engineering for cost-effective code generation with large language models”, 2017, p. 8](zotero://select/library/items/VKR2ARTC)) ([pdf](zotero://open-pdf/library/items/9RXNTKG3?page=8&annotation=2PCDFAFH)) #5fb236
|
||
- “For the ablation study in Section 6,” ([“Evolutionary prompt engineering for cost-effective code generation with large language models”, 2017, p. 8](zotero://select/library/items/VKR2ARTC)) ([pdf](zotero://open-pdf/library/items/9RXNTKG3?page=8&annotation=69EA5VTS)) #ffd400
|
||
|
||
**The reference to the performed ablation study suddenly appears without a proper motivation and introduction.**
|
||
- “temperature of 0.0” ([“Evolutionary prompt engineering for cost-effective code generation with large language models”, 2017, p. 8](zotero://select/library/items/VKR2ARTC)) ([pdf](zotero://open-pdf/library/items/9RXNTKG3?page=8&annotation=ZGLK2MX4)) #ffd400
|
||
|
||
**How this temperature setting is connected to the temperature value given as 0.6 for the initial population builder phase described in Sec 3.2.1? ** #question
|
||
- “RQ1: How does EPiC perform across different SOTA LLMs?” ([“Evolutionary prompt engineering for cost-effective code generation with large language models”, 2017, p. 8](zotero://select/library/items/VKR2ARTC)) ([pdf](zotero://open-pdf/library/items/9RXNTKG3?page=8&annotation=MJ3SNBHU)) #ffd400
|
||
|
||
**Do you have any estimation of the costs of the considered baselines? **#**question**
|
||
- “achieves higher functional correctness at the cost of increased token usage.” ([“Evolutionary prompt engineering for cost-effective code generation with large language models”, 2017, p. 8](zotero://select/library/items/VKR2ARTC)) ([pdf](zotero://open-pdf/library/items/9RXNTKG3?page=8&annotation=WPNHLWNL)) #5fb236
|
||
- “RQ2: How does EPiC compare to other iterative-based agents?” ([“Evolutionary prompt engineering for cost-effective code generation with large language models”, 2017, p. 9](zotero://select/library/items/VKR2ARTC)) ([pdf](zotero://open-pdf/library/items/9RXNTKG3?page=9&annotation=5AMDISQA)) #ffd400
|
||
|
||
**What is the LLM that is used to get the results shown in Table 3?**
|
||
- “To alleviate the threat, we have shown that the LLM-generated tests are not that far off from the results using the original (developer-written) tests.” ([“Evolutionary prompt engineering for cost-effective code generation with large language models”, 2017, p. 10](zotero://select/library/items/VKR2ARTC)) ([pdf](zotero://open-pdf/library/items/9RXNTKG3?page=10&annotation=PKNPJ3QV)) #ffd400
|
||
|
||
**How can you be sure about that? **#**question**
|
||
- ### [[Comments]]
|
||
collapsed:: true
|
||
- #.tabular
|
||
- ### Paper summary
|
||
- The paper presents the EPiC framework to support code generation by means of LLM. In particular, the approach aims to automatically generate prompts by means of evolutionary strategies. Initial prompts are automatically mutated to lead to refined prompts that optimize code generation accuracy and cost-effectiveness. The approach has been evaluated on three datasets and has been shown to outperform existing baselines.
|
||
- ### Strengths
|
||
- + The paper is about an important and timely topic
|
||
- + The proposed evolutionary approach is interesting
|
||
- ### Weaknesses
|
||
- - Important assumptions are weakly justified (e.g., correctness of generated test cases)
|
||
- - key technical components (e.g., mutation strategies, test case generation) lack critical analysis
|
||
- ### Detailed comments for authors
|
||
- Novelty: I find the idea of using an evolutionary approach to find optimal prompts promising. However, related approaches like EvoPrompt are not discussed in sufficient depth to delineate clear strengths of the proposed mutation techniques, which represent a key component of the EPiC framework.
|
||
|
||
Rigor: This is one of the main critical aspects of the paper. The proposed process starts from the generation of initial prompts and test cases from an input textual description. The whole process relies on the assumed correctness of the generated test cases. This is a strong assumption that might invalidate the whole work. How do the authors ensure that generated test cases are semantically coherent with the intended behaviour of the generated code? The use of ASTs to validate text case correctness is stated but not properly explained. However, ASTs might give some hints on the syntactic correctness of test cases. I do not see how they can help with the semantic correctness.
|
||
|
||
Relevance: The paper touches on a relevant problem. I consider the focus on cost-effective LLM usage for code generation of high importance for the software engineering community.
|
||
|
||
Verifiability & transparency: The authors provide a detailed online appendix available at [Anonymized Repository - Anonymous GitHub](https://anonymous.4open.science/r/EPiC-F816/README.md)
|
||
|
||
Presentation: Overall, the paper is well structured, even though some parts would benefit from clarification. For instance, the sudden introduction of the ablation study mentioned for the first time on page 8 should be given earlier.
|
||
|
||
|
||
Detailed comments:
|
||
- Page 1 - “Cost-effective”: The term is central to the paper but remains vague at this point. I suggest clarifying even at this stage what kind of cost is being considered (e.g., token usage, financial cost, latency).
|
||
- Page 1 - “Feedback from code execution into prompts”: It is unclear what kind of feedback is extracted from code execution and how this is fed back into prompt refinement. By reading later in the paper, it seems no explicit feedback is put into the prompts. The fitness function is used to assess whether prompts have to be mutated or not. No additional inputs are given as feedback to instruct the mutation phase.
|
||
- Page 1 - “MBPP dataset”: A brief explanation of the MBPP dataset should be provided for clarity and to support the relevance of the experimental setup.
|
||
- Page 1 - “LATS”: The acronym "LATS" is introduced without any explanation. Please define it.
|
||
- Page 2 - “Evaluating fitness is relatively straightforward”: This statement is questionable, as evaluating prompt fitness involves executing code and verifying outputs. More justification is needed, especially considering the overhead of test execution.
|
||
- Page 2 - “Cost-effectiveness perspective”: The paper should better define how cost-effectiveness is operationalized and measured within this context. E.g., what is the threshold for considering a method cost-effective?
|
||
- Page 3 - “Optimized” prompts and improvement over iterations: Does each iteration actually lead to better prompts? Is there a risk of degradation? Please clarify how improvements are measured and if convergence is guaranteed.
|
||
- Page 3 - “Sufficient to generate the correct implementation”: Terms like "sufficient" and "correct" need to be clearly defined and grounded in empirical criteria. How are they measured in practice?
|
||
- Page 3 - Paragraph starting with “In”: Consider rephrasing the paragraph opening to make it clearer, e.g., "By referring to Fig. 2..."
|
||
- Page 4 - “If all test cases pass, we report the generated code as the final answer”: If the test cases are flawed, passing them does not guarantee correct code. Consider acknowledging this limitation.
|
||
- Page 4 - “We used LLMs for test case generation ... and validated via AST parsing”: Please explain how parsing ASTs ensures functional correctness.
|
||
- Page 4 - “Prompts for code and test case generation”: These appear disconnected. How is semantic consistency maintained between code and tests?
|
||
- Page 4 - Example: add3Numbers test cases: The semantic validity of the test cases is questionable. It’s unclear whether they reflect real functional requirements.
|
||
- Page 5 - “Mutation operators”: The process for generating multiple prompts via mutation lacks detail. What specific mutation strategies are used?
|
||
- Page 5 - “Fitness function based on test case pass ratio”: This raises the same concern about test case reliability. See previous comments on test validity.
|
||
- Page 5 - “Randomly mutates”: More detail is needed: How is randomness controlled? Are there guiding heuristics?
|
||
- Page 6 - “600-word limit” in prompt instructions: In practice, LLMs sometimes fail to follow strict constraints. Is there any verification that these limits are respected?
|
||
- Page 6 - “Selected words” for mutation: The criteria for choosing which words to mutate should be clarified.
|
||
- Page 6 - Reference [21]: This seems to be a citation error and should likely be [3].
|
||
- Page 7 - “ATSP equation”: What happens if the denominator is zero? This should be addressed explicitly.
|
||
- Page 7 - “Cost not disproportionately high”: This is vague. What threshold or baseline is used to make this claim?
|
||
- Page 7 - “Baseline configurations”: Please clarify if these were taken from prior work or re-implemented. Are they optimized?
|
||
- Page 8 - “Ablation study”: The mention of the ablation study feels abrupt. Introduce it earlier in the experimental setup.
|
||
- Page 8 - “Temperature of 0.0” vs 0.6 in Sec. 3.2.1: This inconsistency should be addressed. Why are different temperatures used in different phases?
|
||
- Page 8 - RQ1 and Cost Estimation: Are there any estimates for the computational cost of the baseline methods used for comparison?
|
||
- Page 9 - RQ2 / Table 3: What LLM is used to produce the results shown in Table 3? Consistency in experimental conditions must be ensured.
|
||
- Questions
|
||
- 1. How is the correctness of the generated test cases ensured, especially when they are produced automatically by LLMs?
|
||
- 2. What are the mutation operators and selection strategies used to evolve prompts, and how are they guided?
|
||
- 3. How is the concept of cost-effectiveness defined, measured, and validated in the EPiC framework?
|
||
- [[Metareview]]
|
||
- This paper introduces EPiC, a framework for improving LLM-based code generation via evolutionary prompt refinement. While the topic is interesting, all reviewers raised serious concerns about the experimental assumptions, particularly the reliance on automatically generated test cases whose semantic correctness is not validated. The authors' rebuttal acknowledges limitations but fails to sufficiently address how these assumptions impact result validity. Additional concerns include unclear novelty over prior work, weak justification of core design choices, and insufficient evaluation of prompt mutation strategies. Given these limitations, the paper is not ready for acceptance in its current form. However, the reviewers have provided suggestions that we think will be valuable in improving the paper for potential publication in the future.
|
||
- ### [[REVIEWS/Notes]]
|
||
- ### YELLOW CONCERNS
|
||
background-color:: yellow
|
||
- {{query (and [[ffd400]] [[icse2026-paper1640]] )}}
|
||
collapsed:: true
|
||
- ### ❓️Questions
|
||
- {{query (and [[question]] [[icse2026-paper1640]] )[[question]]}}
|
||
query-table:: false
|
||
query-properties:: [:block] |