[logseq-plugin-git:commit] 2025-06-10T09:35:35.445Z
This commit is contained in:
@@ -6,49 +6,45 @@ full-title:: Revisiting Prompt Engineering: A Comprehensive Evaluation for LLM-b
|
||||
date-start:: [[10-06-2025]] - 10:39
|
||||
date-submitted::
|
||||
external-links::
|
||||
status:: [[DOING]]
|
||||
status:: [[DONE]]
|
||||
deadline-submission::
|
||||
file:: [[@RecSys_2025_Reproducibility_689]]
|
||||
parent::
|
||||
todoist:: https://app.todoist.com/app/task/689-revisiting-prompt-engineering-a-comprehensive-evaluation-for-llm-based-perso-6c7x6q67Wh8HQPcc
|
||||
|
||||
- ### [[Highlights]]
|
||||
- ### [[Comments]]
|
||||
- **REVIEW**
|
||||
- The paper is clearly written and well structured. Overall I liked reading the paper even though there are some improvements that are needed. In particular, even though the experiments are properly conducted and convincing, authors should clarify the use of random rankings to complete output lists that might introduce noise or bias. Additionally, I would improve the definition of "baseline", which is currently blurred.
|
||||
- The methodology is clearly presented and the combinations that have been investigated permit to distill and discuss interesting insights about the selection of prompt configrations. Nevertheless, authors should discuss more the external validity of the obtained results. To what extent do they apply to situations / domains that are different from those considered in the paper?
|
||||
-
|
||||
- ### **Overall Presentation:**
|
||||
- **Score: 4 – Well-Presented**
|
||||
- ### **Technical Soundness:**
|
||||
- **Score: 3 – Adequate Technical Soundness**
|
||||
- ### **Reproducibility of Methods:**
|
||||
- **Score: 4 – Highly Reproducible**
|
||||
|
||||
All datasets, LLM configurations, and prompt templates are openly accessible and sufficiently documented.
|
||||
- ### **Impact:**
|
||||
- **Score: 4 – High**
|
||||
|
||||
This paper addresses an interesting topic, i.e., prompt design for LLM-based recommender systems. The large-scale comparison across prompts, datasets, and models provides valuable insights for both researchers and practitioners. The systematic analysis of prompt effectiveness and cost-efficiency has the potential to guide real-world deployment strategies in resource-constrained settings.
|
||||
- ### **Artifacts – Availability and Documentation:**
|
||||
- The paper references a well-structured experimental setup, but the availability and documentation of artifacts (code, prompts, data splits, model calls) should be explicitly confirmed. Assuming they are included and well-documented, this aspect is well-addressed.
|
||||
- ### **Artifacts – Executability:**
|
||||
- Assuming the documentation is in place, the experimental setup should be executable. Still, some edge cases (e.g., prompt failure fallbacks) could make exact replication difficult without detailed scripts or examples. Minor improvements could help ensure smooth reuse.
|
||||
- ### **Review:**
|
||||
- This paper presents a comprehensive and timely evaluation of prompt engineering strategies for LLM-based recommendation. The authors consider 23 prompt types, 8 real-world datasets, and 12 LLMs, offering a large-scale empirical analysis with both accuracy and inference cost in view.
|
||||
|
||||
Some concerns remain about certain experimental choices. For instance, fallback to random rankings when LLMs fail to provide outputs introduces potential bias, especially if such failures are not uniformly distributed across prompt types or models. Clarifications are also needed regarding what constitutes the "baseline" prompt and how prompt complexity is defined.
|
||||
|
||||
Despite these issues, the findings—especially that simpler prompts tend to outperform more complex ones with high-performance LLMs—are practically relevant and well-articulated. The paper would benefit from a stronger discussion on the external validity of results and more guidance on how these insights generalize across tasks or domains.
|
||||
|
||||
---
|
||||
- ### **Showstopper:**
|
||||
- Clarify the definition and configuration of the "baseline" prompt used in comparisons.
|
||||
- Justify and analyze the impact of fallback mechanisms such as random completions—do they affect model-level or prompt-level conclusions?
|
||||
- Specify how "complex prompt designs" are operationalized or measured.
|
||||
|
||||
---
|
||||
- ### **Overall Evaluation:**
|
||||
- **Score: 1 – Weak Accept**
|
||||
|
||||
This is a solid empirical contribution on prompt design in LLM-based recommender systems. While the scale and analysis are commendable, the paper would benefit from stronger discussion of experimental assumptions and their implications. With minor clarifications and justifications, it has potential for impactful contribution to the community.
|
||||
- [[Comments]]
|
||||
- | *Overall Presentation* |
|
||||
|
||||
**4**: (Well-Presented)
|
||||
|
||||
|
|
||||
| *Technical Soundness* |
|
||||
|
||||
**3**: (Adequate Technical Soundness)
|
||||
|
||||
|
|
||||
| *Reproducibility of Methods* |
|
||||
|
||||
**5**: (Fully Reproducible)
|
||||
|
||||
|
|
||||
| *Impact* |
|
||||
|
||||
**4**: (High)
|
||||
|
||||
This paper is about an interesting and relevant topic, i.e., the selection and composition of prompt patterns for LLM-based recommender systems. The performed analysis investigates different dimensions, i.e., prompt patterns, datasets, and LLMs. The given insights on prompt effectiveness and cost-efficiency are potentially of high impact in guiding real-world adoption of LLMs for developing RSs. |
|
||||
| *Artifacts- . Are the paper artifacts (source code, data) available and well documented?”* | The paper references a well-structured experimental setup, including Python code and Jupyter Notebooks. The package made available is organized with respect to the different configurations shown in the paper. |
|
||||
| *Artifacts. Are the artifacts executable by following the instructions provided in the documentation?* | The presented pipeline is documented and it is possible to execute it by following the structured README file. |
|
||||
| *Review* | This paper comprehensively evaluates prompt engineering strategies for LLM-based recommendation systems. The authors consider different prompt types, real-world datasets, and LLMs.
|
||||
|
||||
The paper is clearly written and well structured. I liked reading the paper, even though some improvements are needed. In particular, even though the experiments are properly conducted and convincing, authors should clarify the use of random rankings to complete output lists that might introduce noise or bias. Additionally, I would improve the definition of "baseline," which is currently blurred.
|
||||
|
||||
The methodology is clearly presented, and the combinations that have been investigated permit us to distill and discuss interesting insights about the selection of prompt configurations. Nevertheless, the authors should discuss the results' external validity. To what extent do they apply to situations/domains different from those considered in the paper? |
|
||||
| *Showstopper* | - Can you justify and analyze the impact of random completion of output list? Do they affect model-level or prompt-level conclusions?
|
||||
- Can you clarify the definition and configuration of the "baseline" in comparisons. |
|
||||
| *Overall Evaluation* |
|
||||
|
||||
**1**: (Weak accept)
|
||||
|
||||
This interesting and well-conducted investigation gives the reader insights into prompt design in LLM-based recommender systems. However, the paper would benefit from a stronger discussion of experimental assumptions and their implications (see my comments above). |
|
||||
Reference in New Issue
Block a user