logseq/pages/RecSys_2025_Reproducibility_689.md

type:: REVIEWS tags:: year:: 2025 venue:: RecSys full-title:: Revisiting Prompt Engineering: A Comprehensive Evaluation for LLM-based Personalized Recommendation date-start:: 10-06-2025 - 10:39 date-submitted:: external-links:: status:: DOING deadline-submission:: file:: @RecSys_2025_Reproducibility_689 parent:: todoist:: https://app.todoist.com/app/task/689-revisiting-prompt-engineering-a-comprehensive-evaluation-for-llm-based-perso-6c7x6q67Wh8HQPcc

- ### [[Highlights]]
- ### [[Comments]]
	- **REVIEW**
		- The paper is clearly written and well structured. Overall I liked reading the paper even though there are some improvements that are needed. In particular, even though the experiments are properly conducted and convincing, authors should clarify the use of random rankings to complete output lists that might introduce noise or bias. Additionally, I would improve the definition of "baseline", which is currently blurred.
		- The methodology is clearly presented and the combinations that have been investigated permit to distill and discuss interesting insights about the selection of the right prompt configrations for the task at hand. Nevertheless, authors should discuss more the external validity of the obtained results. To what extent do they apply to situations / domains that are different than those considered in the paper?
	-
	- ### **Overall Presentation:**
		- **Score: 4 – Well-Presented**
	- ### **Technical Soundness:**
		- **Score: 3 – Adequate Technical Soundness**
	- ### **Reproducibility of Methods:**
		- **Score: 4 – Highly Reproducible**
		  
		  All datasets, LLM configurations, and prompt templates are openly accessible and sufficiently documented.
	- ### **Impact:**
		- **Score: 4 – High**
		  
		  This paper addresses an interesting topic, i.e., prompt design for LLM-based recommender systems. The large-scale comparison across prompts, datasets, and models provides valuable insights for both researchers and practitioners. The systematic analysis of prompt effectiveness and cost-efficiency has the potential to guide real-world deployment strategies in resource-constrained settings.
	- ### **Artifacts – Availability and Documentation:**
		- The paper references a well-structured experimental setup, but the availability and documentation of artifacts (code, prompts, data splits, model calls) should be explicitly confirmed. Assuming they are included and well-documented, this aspect is well-addressed.
	- ### **Artifacts – Executability:**
		- Assuming the documentation is in place, the experimental setup should be executable. Still, some edge cases (e.g., prompt failure fallbacks) could make exact replication difficult without detailed scripts or examples. Minor improvements could help ensure smooth reuse.
	- ### **Review:**
		- This paper presents a comprehensive and timely evaluation of prompt engineering strategies for LLM-based recommendation. The authors consider 23 prompt types, 8 real-world datasets, and 12 LLMs, offering a large-scale empirical analysis with both accuracy and inference cost in view.
		  
		  Some concerns remain about certain experimental choices. For instance, fallback to random rankings when LLMs fail to provide outputs introduces potential bias, especially if such failures are not uniformly distributed across prompt types or models. Clarifications are also needed regarding what constitutes the "baseline" prompt and how prompt complexity is defined.
		  
		  Despite these issues, the findings—especially that simpler prompts tend to outperform more complex ones with high-performance LLMs—are practically relevant and well-articulated. The paper would benefit from a stronger discussion on the external validity of results and more guidance on how these insights generalize across tasks or domains.
		  
		  ---
	- ### **Showstopper:**
		- Clarify the definition and configuration of the "baseline" prompt used in comparisons.
		- Justify and analyze the impact of fallback mechanisms such as random completions—do they affect model-level or prompt-level conclusions?
		- Specify how "complex prompt designs" are operationalized or measured.
		  
		  ---
	- ### **Overall Evaluation:**
		- **Score: 1 – Weak Accept**
		  
		  This is a solid empirical contribution on prompt design in LLM-based recommender systems. While the scale and analysis are commendable, the paper would benefit from stronger discussion of experimental assumptions and their implications. With minor clarifications and justifications, it has potential for impactful contribution to the community.