logseq/pages/PAPERS___Journal-Green-Paper.md

type:: JournalPaper date:: 31-07-2025 - 18:12 full-title:: external-links:: todoist:: year:: date-start:: status:: DOING venue:: priority:: P1
parent:: leader:: todoist:: progress:: {{renderer :todomaster}}

• Tasks
  • DONE Finire sezione 3 (riprendere da 3.2.2) id:: 688b9643-5053-49b9-b9ed-b559e7bf653c
  • TODO Sezione 4
  • TODO Sezione 5
  • TODO Sezione 6
  • TODO Fare passaggio tramite chat per fare un check su eventuali ulteriori aspetti da approfondire
  • TODO Gestire la definizione di "prompt customization"
  • TODO Vedere questione statistical tests

• ⚠️ Areas That May Undermine Convincingness

  • Link between energy usage and output size needs stronger evidence.

    The claim that “energy consumption is strictly related to the size of the output” is intuitive but not rigorously validated in the current text. Including a more detailed correlation analysis (scatter plots, regression, or correlation coefficient) would strengthen this conclusion.

  • Codellama’s negative performance is not fully explained.

    The paper observes that Codellama consumes more energy and is verbose, but the reasons for this behavior (architecture differences, tokenization strategy, fine-tuning objectives) are only hinted at.

    🔹 Suggestion: add a discussion grounded in literature or empirical reasoning to explain why code-specific tuning might lead to over-generation.

  • Impact of text format on summarization is under-discussed.

    While results show that text format has “less pronounced” effects, the explanation remains superficial. Providing hypotheses (e.g., tokenizer normalization, model robustness to markup) or a short qualitative analysis would make this claim more convincing.

  • Comparative discussion with related work is weak.

    The related work section mentions several studies but does not explicitly compare your findings to them.

    🔹 Suggestion: add a paragraph discussing how your results confirm, contradict, or extend previous findings (e.g., Adamska et al. on answer length, Husom et al. on correlation between response size and energy use).

  • Lack of statistical analysis.

    While average values are presented, no statistical test (e.g., t-test, ANOVA) is reported to support claims of significant differences between configurations. Even a basic significance analysis would increase persuasiveness.

  • Some methodological choices need more justification.

    For example:

    • Why were exactly 1,000 snippets and 343 README files selected?

    • Why was a 5-second pause deemed sufficient, despite acknowledging standard practice is longer?

      These decisions should be reframed as conscious design choices, not constraints.

• Weaknesses & Concerns

  collapsed:: true

  • Lack of statistical significance analysis:

    Differences in energy consumption and accuracy are reported as percentages or observations, but without statistical testing it is unclear whether they are significant.

  • Partial explanation of key findings:

    • The strong correlation between output size and energy consumption is asserted but not quantitatively validated.
    • Codellama’s verbosity and inefficiency are observed but not fully explained.

  • Under-discussed text summarization results:

    The impact of input format is deemed minor, but no deeper reasoning or hypotheses are offered.

  • Limited comparative discussion with prior work:

    Related work is cited, but the paper does not explicitly position its findings relative to existing literature (e.g., whether your observations about answer length confirming Adamska et al.’s results).

  • Methodological decisions appear ad-hoc:

    The justification for dataset sizes, input limits, and pause intervals could be better framed as design decisions rather than constraints.

    ---

• Overall Evaluation

  collapsed:: true

  • Technical Soundness: ★★★★☆ (solid but needs more statistical rigor and stronger causal explanations)

  • Persuasiveness: ★★★☆☆ (results are interesting but require deeper contextualization and comparison with literature)

  • Contribution: ★★★★☆ (original and significant in the context of Green AI and SE)

    ---

• 2. Concrete Suggestions to Strengthen the Paper

  collapsed:: true

  • A. Strengthen Explanations

    • Provide a quantitative correlation between output length and energy consumption (scatter plot + Pearson correlation).
    • Explain Codellama’s behavior with references to differences in training data, tokenization, or decoding strategies.

  • B. Expand Comparative Discussion

    • Explicitly compare findings to Adamska et al. (on output length) and Husom et al. (on response-energy correlation).
    • Mention whether your findings align or contradict prior observations on prompt engineering’s energy effects.

  • C. Improve the Text Summarization Analysis

    • Discuss why different formats (HTML vs. Markdown vs. Plain Text) had little impact—e.g., tokenization robustness, model pretraining exposure.
    • Add qualitative examples of how outputs differ across formats.

  • D. Add Minimal Statistical Support

    • Perform at least a paired t-test or ANOVA to confirm differences between configurations are significant.
    • Report confidence intervals alongside averages.

  • E. Reframe Methodological Constraints

    • Reword justifications like “forced to opt for” or “for the sake of time” to “we selected a 5-second interval based on empirical trade-offs between experimental feasibility and consistency with prior studies”.

  • F. Strengthen Conclusions

    • End with a clear, strong statement: “Our findings demonstrate that prompt design is a non-trivial factor in the energy efficiency of LLMs. This opens the way for energy-aware prompt engineering strategies, bridging Green AI and software engineering practices.”

• A. Additional Paragraph for the Discussion Section

  • Impact of Output Size, Model Behavior, and Format Robustness

    Our experimental results suggest a strong link between the size of the generated output and the observed energy consumption. This observation aligns with prior findings by Adamska et al. [1] and Husom et al. [15], who also reported a direct relationship between output verbosity and energy demand during inference. However, unlike previous works, our study demonstrates that this effect is modulated by prompt engineering: configurations that constrain generation length (e.g., C_{c2}) not only reduce energy consumption but also improve accuracy for Llama 3.1. Interestingly, Codellama—despite being specialized for code—exhibited systematically higher consumption and verbosity, likely due to its decoding strategies and fine-tuning objectives, which encourage detailed explanations. Further investigation is required to determine whether this behavior is inherent to code-specific LLMs or an artifact of the model architecture. Regarding text summarization, we observed that input format variations (Markdown, HTML, plain text) had negligible influence on energy usage and predictive performance. This may be explained by the tokenizer’s robustness to markup structures and the model’s exposure to diverse formats during pretraining. These findings highlight that prompt engineering can be an effective lever for controlling energy consumption, while input formatting has a more marginal role.

• B. Strengthened Conclusion

  • 6 Conclusion and Future Directions

    This paper empirically investigated how prompt engineering techniques and input text formats influence the energy consumption and predictive performance of large language models in two representative software engineering tasks: code completion and text summarization. By systematically comparing different prompt configurations across Llama 3.1 and Codellama, we demonstrated that prompt design significantly impacts both energy efficiency and output accuracy. In particular, configurations that explicitly guide the model through custom tags while constraining verbosity (e.g., C_{c2}) resulted in up to a twofold reduction in energy usage and substantial accuracy improvements for Llama 3.1. Conversely, Codellama, despite its code specialization, was found to be energy-inefficient and prone to verbose outputs, raising new questions about the trade-offs of domain-specific fine-tuning. For text summarization, our findings indicate that input format variations (plain text, Markdown, HTML) have minimal impact on energy consumption, suggesting that model tokenization and pretraining mitigate format-related differences.

    Overall, our results provide actionable evidence that prompt engineering is a key factor in shaping the energy footprint of LLMs. This insight opens the door to energy-aware prompting strategies that align with the principles of Green AI and sustainable software engineering. Future work will explore the integration of prompt optimization with dynamic energy monitoring, the extension of our analysis to larger and more diverse models, and the design of automated frameworks to balance performance and environmental costs.

• Threats to Validity
  • Our study is subject to several threats to validity that warrant discussion. Internal validity may be affected by the hardware configuration and monitoring tools. To mitigate this, all experiments were executed on the same high-performance server, and energy measurements were consistently collected using CodeCarbon, a widely adopted and validated tool. We also performed five repetitions per configuration to minimize random fluctuations. Construct validity relates to whether our metrics truly capture the energy-performance trade-off. While CodeCarbon provides reliable estimates of GPU energy usage, future work could incorporate direct hardware-level measurements (e.g., using NVIDIA-smi logging) for finer granularity. External validity concerns the generalizability of our findings. We evaluated only two LLMs and two tasks; while they are representative of common SE scenarios, extending the analysis to larger models, diverse tasks, and alternative architectures is necessary to confirm the observed trends. Finally, conclusion validity may be impacted by the absence of formal statistical testing. Although we observed clear differences across configurations, future work should integrate statistical analyses (e.g., t-tests, ANOVA) to rigorously assess the significance of these differences. Despite these limitations, the consistent patterns observed across multiple metrics and repetitions strengthen our confidence in the reported findings.

• ✅ Final Journal-Readiness Checklist

  • DONE 1. Title & Abstract

id:: 688cc320-7b06-4594-943f-f302da78f521 - Change the title (current one duplicates the GREEN paper).

	  👉 Example: *"Energy-Aware Prompt Engineering for Large Language Models: An Empirical Study on Software Engineering Tasks"*
	- **Enrich abstract** with a sentence highlighting the novelty (*“This is the first work systematically linking prompt design to energy consumption in LLM-based software engineering tasks.”*).
	  
	  ---
- ### **2. References**
	- Add **recent citations** for:
		- LLMs in code completion (e.g., Copilot, Codex studies).
		- Energy-aware AI and Green AI.
		- Specific LLMs mentioned (Gemma, Mistral, Vicuna).
	- Complete missing citations flagged as **DAVIDE ▶Reference◀**.
	  
	  ---
- ### **3. Methodology**
	- Reframe **dataset size and pause interval** as design decisions, not constraints.
	- Clarify ambiguous text in the **Prompt Creator** subsection.
	- Add reference to **Table 1** when configurations are introduced.
	  
	  ---
- ### **4. Results & Analysis**
	- Provide **correlation analysis** between **output length** and **energy consumption** (supports main claim).
	- Discuss **why Codellama over-generates** with reference to architecture/training.
	- Expand discussion on **text summarization robustness** to input formats.
	  
	  ---
- ### **5. Threats to Validity**
	- Include the refined paragraph I provided.
	- Explicitly mention potential **bias from hardware configuration** and **use of one monitoring tool**.
	  
	  ---
- ### **6. Discussion & Conclusion**
	- Include the **additional paragraph** and **strengthened conclusion**.
	- Add a sentence on **practical implications** (e.g., guidelines for energy-efficient prompting).
	  
	  ---

• ✅ Draft: Statistical Tests to Include

  To make your analysis statistically rigorous without overcomplicating, here is a minimal set of tests you can integrate:

  ---

  • 1. Paired t-tests

    • Purpose: Compare energy consumption (and accuracy) between two configurations (e.g., C_{c0} vs. C_{c2}).
    • Application:
      • For each task and PET (zero/one/few-shot), run a paired t-test on the five repetitions per configuration.
    • Reporting:

      • Report p-values and effect sizes (Cohen’s d).

      • Example statement:

        “The energy consumption of C_{c2} was significantly lower than C_{c0} for the code completion task (t(4) = 5.67, p < 0.01, d = 1.9).”

        ---

  • 2. One-way ANOVA

    • Purpose: Compare all configurations ($C_{c0}$–C_{c4}) simultaneously to determine if differences are statistically significant.
    • Application:
      • Run separately for Llama 3.1 and Codellama, for both energy and accuracy metrics.
    • Reporting:

      • Include F-statistic and p-value.

      • Example:

        “ANOVA revealed a significant effect of prompt configuration on energy consumption for Llama 3.1 (F(4,20) = 12.4, p < 0.001). Post-hoc Tukey tests showed that C_{c2} and C_{c3} consumed significantly less energy than C_{c0} and C_{c4}.”

        ---

  • 3. Correlation (Pearson’s r)

    • Purpose: Quantify relationship between output length and energy consumption.
  • Application:
    • Compute Pearson correlation per model and per task.
  • Reporting:

    • Example:

      “Output length was strongly correlated with energy consumption for Codellama (r = 0.89, p < 0.001), confirming that verbosity is a primary driver of higher energy usage.”

      ---

  • 4. Optional: Boxplots/Violin Plots

    • Complement statistical tests with visualizations to show distribution differences.