50 lines
3.2 KiB
Markdown
50 lines
3.2 KiB
Markdown
type:: [[REVIEWS]]
|
|
tags::
|
|
year:: 2025
|
|
venue:: [[RecSys]]
|
|
full-title:: Revisiting Prompt Engineering: A Comprehensive Evaluation for LLM-based Personalized Recommendation
|
|
date-start:: [[10-06-2025]] - 10:39
|
|
date-submitted::
|
|
external-links::
|
|
status:: [[DONE]]
|
|
deadline-submission::
|
|
file:: [[@RecSys_2025_Reproducibility_689]]
|
|
parent::
|
|
todoist:: https://app.todoist.com/app/task/689-revisiting-prompt-engineering-a-comprehensive-evaluation-for-llm-based-perso-6c7x6q67Wh8HQPcc
|
|
|
|
- ### [[Highlights]]
|
|
- [[Comments]]
|
|
- | *Overall Presentation* |
|
|
|
|
**4**: (Well-Presented)
|
|
|
|
|
|
|
| *Technical Soundness* |
|
|
|
|
**3**: (Adequate Technical Soundness)
|
|
|
|
|
|
|
| *Reproducibility of Methods* |
|
|
|
|
**5**: (Fully Reproducible)
|
|
|
|
|
|
|
| *Impact* |
|
|
|
|
**4**: (High)
|
|
|
|
This paper is about an interesting and relevant topic, i.e., the selection and composition of prompt patterns for LLM-based recommender systems. The performed analysis investigates different dimensions, i.e., prompt patterns, datasets, and LLMs. The given insights on prompt effectiveness and cost-efficiency are potentially of high impact in guiding real-world adoption of LLMs for developing RSs. |
|
|
| *Artifacts- . Are the paper artifacts (source code, data) available and well documented?”* | The paper references a well-structured experimental setup, including Python code and Jupyter Notebooks. The package made available is organized with respect to the different configurations shown in the paper. |
|
|
| *Artifacts. Are the artifacts executable by following the instructions provided in the documentation?* | The presented pipeline is documented and it is possible to execute it by following the structured README file. |
|
|
| *Review* | This paper comprehensively evaluates prompt engineering strategies for LLM-based recommendation systems. The authors consider different prompt types, real-world datasets, and LLMs.
|
|
|
|
The paper is clearly written and well structured. I liked reading the paper, even though some improvements are needed. In particular, even though the experiments are properly conducted and convincing, authors should clarify the use of random rankings to complete output lists that might introduce noise or bias. Additionally, I would improve the definition of "baseline," which is currently blurred.
|
|
|
|
The methodology is clearly presented, and the combinations that have been investigated permit us to distill and discuss interesting insights about the selection of prompt configurations. Nevertheless, the authors should discuss the results' external validity. To what extent do they apply to situations/domains different from those considered in the paper? |
|
|
| *Showstopper* | - Can you justify and analyze the impact of random completion of output list? Do they affect model-level or prompt-level conclusions?
|
|
- Can you clarify the definition and configuration of the "baseline" in comparisons. |
|
|
| *Overall Evaluation* |
|
|
|
|
**1**: (Weak accept)
|
|
|
|
This interesting and well-conducted investigation gives the reader insights into prompt design in LLM-based recommender systems. However, the paper would benefit from a stronger discussion of experimental assumptions and their implications (see my comments above). | |