316 lines
17 KiB
Markdown
316 lines
17 KiB
Markdown
tags:: [[#zotero]]
|
||
title:: @RecSys_2025_Reproducibility_711
|
||
item-type:: [[document]]
|
||
original-title:: RecSys_2025_Reproducibility_711
|
||
links:: [Local library](zotero://select/library/items/PM5PCLML), [Web library](https://www.zotero.org/users/1039502/items/PM5PCLML)
|
||
|
||
- ### Attachments
|
||
- [PDF](zotero://select/library/items/M5TFTVEK) {{zotero-imported-file M5TFTVEK, "RecSys_2025_Reproducibility_711.pdf"}}
|
||
- ### Notes
|
||
- I'm reviewing a research paper and I took the following notes:
|
||
|
||
# Annotazioni
|
||
(9/6/2025, 16:24:56)
|
||
|
||
- “environmental footprint of recommender systems has received growing attention in the research community” (“RecSys_2025_Reproducibility_711”, p. 1) #5fb236
|
||
|
||
- “comprehensive evaluation should also account for the emissions produced during inference time,” (“RecSys_2025_Reproducibility_711”, p. 1) #5fb236
|
||
|
||
- “In this study, we build upon and extend the analysis of Spillo et al. by incorporating the inference phase into the carbon footprint assessment and exploring how variations in training configurations affect emissions.” (“RecSys_2025_Reproducibility_711”, p. 1) #a28ae5
|
||
|
||
- “Our findings reveal that models with higher training emissions can, in some cases, offer lower environmental costs at inference time” (“RecSys_2025_Reproducibility_711”, p. 1) #e56eee
|
||
|
||
- “Moreover, we show that minimizing the number of validation metrics computed during training can lead to significant reductions in overall carbon footprint, highlighting the importance of thoughtful experimental design in sustainable machine learning.” (“RecSys_2025_Reproducibility_711”, p. 1) #e56eee
|
||
|
||
- “Between 2012 and 2018, the compute used in state-of-the-art training runs increased by over 300,000x” (“RecSys_2025_Reproducibility_711”, p. 1) #a28ae5
|
||
|
||
- “global data center electricity consumption could nearly double by 2030” (“RecSys_2025_Reproducibility_711”, p. 1) #a28ae5
|
||
|
||
- “Green AI initiative” (“RecSys_2025_Reproducibility_711”, p. 1) #2ea8e5
|
||
|
||
- “Recommender Systems (RS)” (“RecSys_2025_Reproducibility_711”, p. 1) #5fb236
|
||
|
||
- “have received comparatively less attention in terms of sustainability” (“RecSys_2025_Reproducibility_711”, p. 1) #a28ae5
|
||
|
||
- “Also, given that RS are commonly retrained at daily and weekly intervals, their environmental footprint can accumulate rapidly over time, magnifying even small inefficiencies in model or infrastructure choices.” (“RecSys_2025_Reproducibility_711”, p. 1) #e56eee
|
||
|
||
- “complex deep and graph-based models often provide only marginal accuracy improvements while significantly increasing carbon emissions.” (“RecSys_2025_Reproducibility_711”, p. 1) #a28ae5
|
||
|
||
- “their analysis focused only the training phase, and neglecting the energy cost of routine validation and model inference, both critical components in real-world deployment.” (“RecSys_2025_Reproducibility_711”, p. 1) #a28ae5
|
||
|
||
- “In this work, we revisit and extend Spillo et al. [18]’s study with the goal of broadening the environmental lens through which RS are evaluated.” (“RecSys_2025_Reproducibility_711”, p. 1) #e56eee
|
||
|
||
- “RQ1: Training setup” (“RecSys_2025_Reproducibility_711”, p. 1) #2ea8e5
|
||
|
||
- “How do variations in the training setup affect the reproducibility of performance and energy footprint measurements for RS training?” (“RecSys_2025_Reproducibility_711”, p. 1) #e56eee
|
||
|
||
- “RQ2: Inference.” (“RecSys_2025_Reproducibility_711”, p. 1) #2ea8e5
|
||
|
||
- “Do models that incur in higher energy footprints during training also exhibit higher footprints at inference time?” (“RecSys_2025_Reproducibility_711”, p. 1) #e56eee
|
||
|
||
- “Validation overhead.” (“RecSys_2025_Reproducibility_711”, p. 1) #2ea8e5
|
||
|
||
- “computing validations metrics” (“RecSys_2025_Reproducibility_711”, p. 1) #a28ae5
|
||
|
||
- “Life-cycle perspective” (“RecSys_2025_Reproducibility_711”, p. 1) #2ea8e5
|
||
|
||
- “inference cost,” (“RecSys_2025_Reproducibility_711”, p. 1) #a28ae5
|
||
|
||
- “Reproducibility conditions.” (“RecSys_2025_Reproducibility_711”, p. 1) #2ea8e5
|
||
|
||
- “how observations shift even with modest changes to the execution environment.” (“RecSys_2025_Reproducibility_711”, p. 1) #a28ae5
|
||
|
||
- “sustainable RS research and offers actionable guidance for researchers and practitioners working towards low-carbon RS.” (“RecSys_2025_Reproducibility_711”, p. 1) #f19837
|
||
|
||
- “Carbon Intensity (CI) of 475 gCO2/kWh, based on the 2019 International Energy Agency Global Energy and CO2 Status Report” (“RecSys_2025_Reproducibility_711”, p. 2) #a28ae5
|
||
|
||
- “This simplified approach treats all computational activity as equally carbon-intensive, regardless of location, energy source or infrastructure.” (“RecSys_2025_Reproducibility_711”, p. 2) #5fb236
|
||
|
||
- “direct energy tracking.” (“RecSys_2025_Reproducibility_711”, p. 2) #f19837
|
||
|
||
- “By reporting raw energy rather than emissions” (“RecSys_2025_Reproducibility_711”, p. 2) #a28ae5
|
||
|
||
- “Amazon Books” (“RecSys_2025_Reproducibility_711”, p. 2) #2ea8e5
|
||
|
||
- “MovieLens 1M” (“RecSys_2025_Reproducibility_711”, p. 2) #2ea8e5
|
||
|
||
- “Mind” (“RecSys_2025_Reproducibility_711”, p. 2) #2ea8e5
|
||
|
||
- “we adopt a mobile laptop environment that reflects increasingly common deployment and experimentation scenarios in real-world contexts (such as academic research, early-stage prototyping, and low-budget settings).” (“RecSys_2025_Reproducibility_711”, p. 2) #ffd400
|
||
*It's not clear what they want to do with this configuration. What's the external validity of it?*
|
||
|
||
- “While our CPU is newer and has more cores, it is also a mobile processor with a focus on energy efficiency” (“RecSys_2025_Reproducibility_711”, p. 2) #5fb236
|
||
|
||
- “The original experiments ran on Ubuntu 20.04 with Linux kernel 5.15.0. Our experiments were conducted on Windows 11 24H2. We conducted our experiments on Windows to reflect a broader range of practical usage scenarios, particularly in environments where it remains the dominant OS.” (“RecSys_2025_Reproducibility_711”, p. 2) #ffd400
|
||
*a completely different setup. How can you get a fair comparison? This is another dimension that differentiates the original setup with that of this paper.*
|
||
|
||
- “we anticipate its impact to be secondary compared to hardware/software stack differences” (“RecSys_2025_Reproducibility_711”, p. 2) #e56eee
|
||
|
||
- “Model Training Methodology” (“RecSys_2025_Reproducibility_711”, p. 2) #5fb236
|
||
|
||
- “RecBole implementations and kept most hyper-parameters at their default values.” (“RecSys_2025_Reproducibility_711”, p. 2) #5fb236
|
||
|
||
- “First, we enabled RecBole’s reproducibility mode, which ensures the training, validation and test splits remain consistent across runs” (“RecSys_2025_Reproducibility_711”, p. 2) #5fb236
|
||
|
||
- “epochs, setting it to 50 for MovieLens and Mind,” (“RecSys_2025_Reproducibility_711”, p. 2) #2ea8e5
|
||
|
||
- “20 for Amazon Books.” (“RecSys_2025_Reproducibility_711”, p. 2) #2ea8e5
|
||
|
||
- “Model evaluation was performed after training each epoch” (“RecSys_2025_Reproducibility_711”, p. 2) #5fb236
|
||
|
||
- “This entire process was repeated 10 times per configuration, and we report the mean and standard deviation across these runs.” (“RecSys_2025_Reproducibility_711”, p. 3) #5fb236
|
||
|
||
- “Relative efficiency ranking is largely stable” (“RecSys_2025_Reproducibility_711”, p. 3) #2ea8e5
|
||
|
||
- “Absolute energy use decreases in most, but not all cases” (“RecSys_2025_Reproducibility_711”, p. 3) #2ea8e5
|
||
|
||
- “Our mobile hardware setup reduces training energy consumption in 31 of the 47 model/dataset combinations compared to the original desktop environment” (“RecSys_2025_Reproducibility_711”, p. 3) #ffd400
|
||
*What can we say about this? It seems you are considering modern hardware devices, and consequently this was quite expected, isn't it? This seems to be obvious..*
|
||
|
||
- “They reinforce the point that absolute energy figures are not easily generalizable across environments.” (“RecSys_2025_Reproducibility_711”, p. 3) #ffd400
|
||
*I think this is quite obvious, isn't it?*
|
||
|
||
- “Validation metric computation carries notable overhead.” (“RecSys_2025_Reproducibility_711”, p. 3) #2ea8e5
|
||
|
||
- “1.03% to 71.71%,” (“RecSys_2025_Reproducibility_711”, p. 3) #ffd400
|
||
*Thisi is a high variance!!!!!*
|
||
|
||
- “We observe that quantitative energy values and model rankings are sensitive to changes in hardware, dataset, and evaluation setup.” (“RecSys_2025_Reproducibility_711”, p. 3) #ffd400
|
||
*This seems obvious to me*
|
||
|
||
- “Our results show that training and inference efficiencies are not strongly aligned.” (“RecSys_2025_Reproducibility_711”, p. 3) #5fb236
|
||
*This is interesting instead.*
|
||
|
||
- “By jointly analyzing training and inference phases, and highlighting the interplay between model complexity and hardware efficiency, our study offers a more granular view of sustainable recommendation and supports more informed, context-aware decisions” (“RecSys_2025_Reproducibility_711”, p. 4) #5fb236
|
||
|
||
COnsider that those that are tagget with #5fb236 are just highlights, those that are tagged with #e56eee and #a28ae5 are imporant sentences. Please pay attention instead to the notes that are tagged with #ffd400. Those that are tagged with #ff6666 are typos or errors. Could you please draft a review by organizing it as follows:
|
||
|
||
**Overall Presentation.*** Please reflect on the overall quality of the presentation of this contribution. Poor – The paper is difficult to follow due to unclear writing, lack of structure, or numerous grammatical and typographical errors. Figures, tables, and explanations are inadequate. Below Average – The main ideas are present but not clearly communicated. Writing may be inconsistent, and the structure could be improved. Some figures or tables may be unclear. Adequate – The paper is understandable but could benefit from better clarity, organization, or formatting. Some sections may be overly complex or underdeveloped. Well-Presented – The paper is clearly written, well-structured, and easy to follow. Figures, tables, and explanations support the content. Minor improvements could enhance readability. Excellent – The paper is exceptionally well-written, logically structured, and highly engaging. All figures, tables, and explanations are clear, precise, and enhance comprehension significantly.
|
||
|
||
|
|
||
|
|
||
5: Excellent
|
||
|
|
||
|
|
||
|
||
|
|
||
|
|
||
4: Well-Presented
|
||
|
|
||
|
|
||
|
||
|
|
||
|
|
||
3: Adequate
|
||
|
|
||
|
|
||
|
||
|
|
||
|
|
||
2: Below Average
|
||
|
|
||
|
|
||
|
||
|
|
||
|
|
||
1: Poor
|
||
|
|
||
|
|
||
|
||
---
|
||
|
||
**Technical Soundness.*** How technically sound is the research presented in this contribution? Fundamentally Flawed – The research contains critical errors, invalid assumptions, or methodological weaknesses that undermine its conclusions. Weak Technical Rigor – Some aspects of the methodology are questionable. Adequate Technical Soundness – The approach is generally valid, but there may be minor methodological weaknesses, limited evaluation, or room for improvement in execution. Strong Technical Soundness – The research is well-executed, with a solid methodology, and appropriate evaluation. Any limitations are acknowledged and justified. Excellent – This contribution is methodologically impeccable.
|
||
|
||
|
|
||
|
|
||
5: Excellent
|
||
|
|
||
|
|
||
|
||
|
|
||
|
|
||
4: Strong Technical Soundness
|
||
|
|
||
|
|
||
|
||
|
|
||
|
|
||
3: Adequate Technical Soundness
|
||
|
|
||
|
|
||
|
||
|
|
||
|
|
||
2: Weak Technical Rigor
|
||
|
|
||
|
|
||
|
||
|
|
||
|
|
||
1: Fundamentally Flawed
|
||
|
|
||
|
|
||
|
||
---
|
||
|
||
**Reproducibility of Methods.*** Please reflect on the reproducibility of the work presented in this contribution. Not Reproducible – Key details of the methodology are missing or unclear, making it impossible for others to replicate the work. No code/data/protocols are provided. Weak Reproducibility – Some methodological details are provided, but important steps, parameters, or implementation details are missing. Limited access to code/data/protocols. Moderate Reproducibility – The methodology is described sufficiently for partial replication, but some ambiguity remains. code/data/protocols may be available but not fully documented. Highly Reproducible – The methodology is well-documented, and replication is feasible with the provided details. code/data/protocols are available and reasonably well-organized. Fully Reproducible – The study is meticulously documented with complete details, ensuring full reproducibility. code/data/protocols are openly available and well-structured for easy replication.
|
||
|
||
|
|
||
|
|
||
5: Fully Reproducible
|
||
|
|
||
|
|
||
|
||
|
|
||
|
|
||
4: Highly Reproducible
|
||
|
|
||
|
|
||
|
||
|
|
||
|
|
||
3: Moderate
|
||
|
|
||
|
|
||
|
||
|
|
||
|
|
||
2: Weak
|
||
|
|
||
|
|
||
|
||
|
|
||
|
|
||
1: Not Reproducible
|
||
|
|
||
|
|
||
|
||
---
|
||
|
||
**Impact.*** Please reflect on whether the ideas and/or results presented will have an impact on the community. Minimal Impact – The ideas or results have little to no influence on the field. They may be too incremental, lack practical relevance, or fail to offer meaningful insights. Limited Impact – The work provides some useful findings but is unlikely to influence future research or applications. Moderate Impact – The contribution is valuable and may inspire further research or applications in certain areas, though its broader significance remains limited. High Impact – The ideas or results have strong potential to shape future research, influence practices, or contribute meaningfully to advancements in the field. Transformative Impact – The work introduces groundbreaking ideas or results that could redefine current understanding, open new research directions, or have significant real-world applications.
|
||
|
||
|
|
||
|
|
||
5: Transformative
|
||
|
|
||
|
|
||
|
||
|
|
||
|
|
||
4: High
|
||
|
|
||
|
|
||
|
||
|
|
||
|
|
||
3: Moderate
|
||
|
|
||
|
|
||
|
||
|
|
||
|
|
||
2: Limited
|
||
|
|
||
|
|
||
|
||
|
|
||
|
|
||
1: Minimal
|
||
|
|
||
|
|
||
|
||
---
|
||
|
||
**Artifacts- . Are the paper artifacts (source code, data) available and well documented?”.*** Consider paper artifacts (source code, data) and state whether they are available and well documented
|
||
|
||
---
|
||
|
||
**Artifacts. Are the artifacts executable by following the instructions provided in the documentation?.** Consider whether the artifacts are executable by following the instructions provided in the documentation. Note that we do not expect the reviewers to run the full set of experiments, especially if they are expensive or rely on specialized hardware, rather we ask you to assess if the experimental pipeline can be started and is executable. Minor issues are acceptable if they can be fixed easily.
|
||
|
||
---
|
||
|
||
**Review.*** Please provide a review of the paper, offering general comments to authors, along with justification from the different aspects you have rated this contribution (relevance, originality, technical soundness, impact, overall presentation, coverage of related work, and reproducibility). In other words, when reviewing, consider how well the paper builds on prior work, the soundness of its methodology, the strength of its evidence and claims, and the clarity of its presentation. Also, assess its practical relevance, ethical considerations, and whether it provides enough detail for replication.
|
||
|
||
---
|
||
|
||
**Showstopper.*** To streamline the review process, please list any key issues where a response from the authors during the rebuttal phase would be essential for your evaluation of this contribution.
|
||
|
||
---
|
||
|
||
**Overall Evaluation.*** You have given detailed feedback to the authors. Now, we ask for your final recommendation along with a brief justification. Both the recommendation score and the review text are required. Remember that when making your final recommendation, you should focus primarily on whether this work makes a strong and original contribution to the field of Recommender Systems. You should also consider the impact of the presented outcomes and the rigor of the research methodology.
|
||
|
||
|
|
||
|
|
||
3: Strong accept
|
||
|
|
||
|
|
||
|
||
|
|
||
|
|
||
2: Accept
|
||
|
|
||
|
|
||
|
||
|
|
||
|
|
||
1: Weak accept
|
||
|
|
||
|
|
||
|
||
|
|
||
|
|
||
-1: Weak reject
|
||
|
|
||
|
|
||
|
||
|
|
||
|
|
||
-2: Reject
|
||
|
|
||
|
|
||
|
||
|
|
||
|
|
||
-3: Strong reject
|
||
|
|
||
| |