logseq/pages/ICSE2023-REVIEW-685.md

title:: Do Developers Benefit from Recommendations when Repairing Inconsistent Design Models? A Controlled Experiment link:: #685 - ICSE 2023 (hotcrp.com) external-links::

• Paper summary

  • The paper presents a controlled experiment to investigate how Repair Recommendations (RRs) are perceived and used by developers while repairing design models.
  • The experiments were performed by involving 24 developers in eight different tasks, with repair recommendations provided in half of the tasks. The authors compared the results of the tasks performed with and without RRs. According to the completed investigation, developers benefit from RRs while working on complex tasks.

• Strengths

  • Paper about a relevant topic
  • Overall well written and structured

• Weaknesses

  • Not convincing result conclusions

• Comments for the authors

  • Significance: The paper is about a significant problem. The performed controlled experiment permits the assessment of how model repairing tools are considered by developers. The critical point is that given a set of models to be repaired, each developer can fix inconsistencies differently. Thus, it is crucial to understand the developers' preferences when repairing models so that it is possible to understand how to improve currently existing tools.
  • Soundness: My main concerns about the paper are related to its soundness. In particular, the authors performed and reported the controlled experiments in a manner which is independent of the technique used to produce repair recommendations. While reading the paper, since the abstract, the reader asks herself, what is the tool/method/technique that provided the recommended recommendations for the task considered in the study? The performed user study might have a sense if authors want to assess the efficiency and effectiveness of a novel recommendation tool using a user study. However, in that case, the authors should have considered an existing baseline and compared the approach with it.
  • While reading the paper, I raised to myself the following question: "is it possible to do the investigation in an approach-independent manner?" Honestly, I don't think so. Doing the investigation with relevant recommendations or with recommendations that might not have sense can change the outcome of the study! In particular, RRs that are not relevant would affect the EFT, which is currently always 1, as discussed in sec. IV. In other words, we are assuming here that repairing recommendations given by tools are always correct that might not always be the case!
  • Providing repair recommendations is always preferable. Effectiveness and efficiency can be changed depending on how such recommendations are shown to the user. The opinion of developers can change depending on "how good" recommendations are, and consequently, it is not possible to evaluate how RRs are perceived, whatever they are produced!
  • Novelty: The authors present a novel investigation. Typically repair recommendations are evaluated by assessing their performance while discarding the related perception of the final users, i.e., developers. In this respect, the authors presented a novel work because they explicitly investigate how Repair Recommendation tools are perceived by the final users (i.e., developers) while doing their repair modelling activities.
  • Verifiability and Transparency: The artifacts produced during the experiments are in an online appendix, which is well-structured and organized.
  • Presentation: The paper is overall well-written and structured. I would suggest the authors improve the introduction by mentioning what kinds of models the investigation is about and, in particular, at which stage of the development process developers have been interviewed.

• Question for authors

  • Q1: Why do the authors believe that the performed investigation can be done independently from the quality/relevance of the repair recommendations and, thus, independently from the tools that produced them?
  • Q2: Which tool have the authors used to produce the considered recommendations, and how it performs with existing tools?
• NOTES
  • ((635c0dda-2f1c-40e8-81c2-3a5a0e186183))
    • Who provided repair recommendations?
  • ((635c1672-e5d1-4c47-8a8c-ca1860206644))
  • ((635c1695-1b9d-4d93-a931-9e5aa79d2e68))
    • ((635c1872-ea83-4b71-b049-f8ce5c14c01f))
    • ((635c16c1-bed9-4507-8ede-d0456ba202ee))
      • Concerning RQ3, how is provided and managed user feedback?
  • ((635c1e75-dc61-4a8e-b14d-b7623255328a))
    • It is not clear what they are evaluating and thus the goal of the paper. Are your presenting a novel approach to provide repair recommendation and then present the user study you have done to evaluate the approach? So the contribution of the paper is THE APPROACH. If the evaluation if "approach independent", then, what are you evaluating?
      • ==If you are proposing a novel approach for repair recommendations, then you have to consider one of the considered approach as baseline!==
  • ((635c1fff-cda0-492c-9dd6-24c4a0857d9f))
    • In my opinion, this cannot be "approach"-independent. The opinion of developers can change depending on "how good" recommendations are, and consequently, it is not possible to evaluate how RRs are perceived, whatever they are produced!
      • ((635c21f0-a899-4d57-ba56-50412ee9ff65))
        • Again this cannot be assessed independently from the recommendation tool that is used
      • ((635c55a9-61e4-4da2-8312-6f645abb8be4))
        • ==Same problem here, like the previous comment!==
  • ((635c213f-585f-465f-b19b-bb2f60a862c0))
    • Why 60 in formula 2 ???
  • ((635c5fb6-5e69-464b-acb8-1e4727d80780))
    • By what / who?
  • In my opinion the question that should be asked is... is it possible to do the investigation done in the paper in an approach independent manner? Honestly, I don't think so! Doing the investigation with recommendations that are relevant or with recommendations that might not have sense, can change the outcome of the study!
    • RR that are not relevant would affect the EFT, which is always 1 as discussed in sec. IV
    • We are assuming here that repairing recommendations given by tools are always correct that might not be always the case!
  • Providing recommendations in my opinion is always preferrable.... what can change the effectiveness and the efficency is the way such recommendations are shown to the user; in other words, the tool support can make the difference.