diff --git a/pages/RecSys_2025_Reproducibility_711.md b/pages/RecSys_2025_Reproducibility_711.md index 2e83dc6f..18101255 100644 --- a/pages/RecSys_2025_Reproducibility_711.md +++ b/pages/RecSys_2025_Reproducibility_711.md @@ -14,8 +14,14 @@ todoist:: https://app.todoist.com/app/task/711-investigating-carbon-footprint-o collapsed:: true - ### [[Comments]] - - Overall presentation: Overall, the paper is well written and structured. Both the goal and the proposed solution are properly described. I have only a few concerns related to some missing justifications or explanations. In particular, I'm not convinced on why a mobile setup is considered and why such a setup has been used to compare the original environments with that presented in this work. - - Technical Soundness: The [[Paper]] defines interesting research questions especially that concerning the inference phase. The methodological pipeline is clearly described in the online material, The insights of the first research questions, while valid, are obvious and expected to me (i[[.]]e., newer hardware being more efficient). It is necessary to stress what's the take away message and why it is [[IMPORTANT]], and especially what's can be built on top of it. - - Reproducibility of methods: The online appendix contains all the artifacts that are needed to reproduce the experiments. I've played a bit with the docker image made available by the authors. Even though I could not execute the whole process until the end, I managed to easly start the pipelines by following the instructions made available by the authors. - - Impact: Carbon footprint in RS is indeed a relevant and interesting topic. While some results are insightful (e[[.]]g., training/inference trafe-offs) other ones are rather expected and may not be impactfull for the community. However, the emphasis on model validation overhead adds useful considerations for future research direction related to sustainable RS. - - Artifacts (Availability and Documentation): An online repository has been made available at https://anonymous.4open.science/r/RecSys-2025-Reproducibility-Investigating-Carbon-Footprint-of-Recommender-Systems it includes both the artifacts and the instructions to replicate the experiments. A docker image is also avilable at \ No newline at end of file + - Overall Presentation: Overall, the paper is well written and structured. Both the goal and the proposed solution are clearly described. I have only a few concerns related to missing justifications or explanations. In particular, I am not convinced by the rationale for using a mobile setup, nor why such a configuration was chosen to compare the original environment with the one presented in this work. + - Technical Soundness: The paper defines interesting research questions, especially the one concerning the inference phase. The methodological pipeline is clearly described in the online material. However, the insights provided for the first research question, while valid, are somewhat obvious and expected (i.e., newer hardware being more efficient). It is necessary to emphasize the takeaway message, why it is important and what can be built on top of it. + - Reproducibility of methods: The online appendix contains all the artifacts required to reproduce the experiments. I briefly tested the Docker image provided by the authors. Although I could not complete the entire process, I was able to easily start the pipelines by following the authors' instructions. + - Impact: Carbon footprint in recommender systems is indeed a relevant and timely topic. While some results are insightful (e.g., training/inference trade-offs), others are rather expected and may not be impactful for the community. Nevertheless, the emphasis on validation overhead introduces valuable considerations for future research directions on sustainable recommender systems. + - Artifacts (Availability and Documentation): An online repository is available at [https://anonymous.4open.science/r/RecSys-2025-Reproducibility-Investigating-Carbon-Footprint-of-Recommender-Systems](https://anonymous[[.]]4open[[.]]science/r/[[RecSys]]-[[2025]]-Reproducibility-Investigating-Carbon-Footprint-of-Recommender-Systems), which includes both the artifacts and the instructions to replicate the experiments. A docker image is also available at [https://hub.docker.com/r/beyondtrainingtime/recsys2025-beyond-training-time-gpu/](https://hub.[[docker]].com/r/beyondtrainingtime/recsys2025-beyond-training-[[time]]-gpu/). + - Artifacts (Executability): The pipeline is clearly described, and the availability of a Docker image is a valuable addition. However, a Docker Compose file would be expected to simplify the creation and execution of the corresponding container. + - Review: This paper investigates the sustainability of recommender systems by analyzing not only training but also inference energy costs. The study shows that higher training emissions can, in some cases, lead to lower inference emissions, and it quantifies the cost of validation steps, suggesting the need for further research on experimental practices. I have a few concerns about the work, particularly regarding the following issues and choices: + - The use of mobile hardware and a Windows setup raises concerns about the fairness of comparisons and the generalizability of the results. + - Many observations (e.g., differences in energy consumption across hardware setups, high variance in validation overheads) are not surprising and risk being perceived as trivial without deeper insights or stronger empirical grounding. + - More critical reflection is needed on the implications of these findings: Are they truly actionable? Could a better experimental design (e.g., controlling more environmental variables) have led to clearer conclusions? + - Despite these limitations, the core idea of the paper, i.e., extending recommender system sustainability beyond training, is interesting and potentially useful to practitioners. \ No newline at end of file