9.9 KiB
9.9 KiB
type:: REVIEWS
tags:: EditoringChairing
year:: 2026
venue:: IEEE SOFTWARE
full-title::
date-start:: 23-12-2025 - 16:08
date-submitted::
external-links::
status:: DONE
deadline-submission::
file:: 
- ### [[Comments]]
- FINAL DECISION (MAJOR)
- While all three reviewers recommend a minor revision, my assessment is more critical. In my view, the issues raised, particularly those highlighted in my comments, are substantive and require significant changes to the manuscript. Addressing them goes beyond polishing or clarification and calls for a more thorough revision of the paper. For this reason, I recommend a major revision.
-
- Reviewers find the paper well written and suitable for IEEE Software. However, several issues have been identified that can be summarized as follows:
- The user study is small and should be framed as a proof of concept rather than making strong assumptions about reducing development time or cognitive load.
- Early in the paper, the authors motivate the work by referring to the need for explainability, especially in the context of AI systems. Unfortunately, the paper lacks a specific, practical AI-based application, weakening its novelty and relevance. It is necessary to delineate the coverage of the pattern by improving the evaluation scenarios.
- A clear justification for the use of AOP is also needed. It is necessary to demonstrate how the pattern can be generalized beyond AOP-centric environments and languages.
- To summarize, a serious and major revision is needed to make the paper convincing beyond the application of the proposed idea to toy examples that are not representative of complex systems, as claimed at the beginning of the paper.
-
- Preliminary comment
- The three reviewers are largely aligned in judging the paper as **sound, well written, and suitable for IEEE Software**, but they consistently identify a set of **minor yet substantive weaknesses** that need addressing. The main criticality concerns the **evaluation and claims**: the user study is small, affected by ordering bias, and should be framed strictly as a *proof of concept*, avoiding strong claims about development-time reduction or cognitive load. A second recurring issue is **scope and positioning**: although the motivation targets AI-based systems and explainability, the paper lacks a **concrete, realistic AI-based application**, relying instead on generic or dummy scenarios, which weakens both novelty and relevance. Reviewers also ask for a **clearer justification of Aspect-Oriented Programming**, including why it was chosen and how the pattern could generalize beyond AOP-centric environments and languages. Finally, some **clarifications and minor fixes** are needed: better delineation of what the pattern does and does not cover (e.g., where explanations should be placed and based on which criteria), improved description of evaluation scenarios, and small presentation and reproducibility improvements (e.g., archiving supplementary material in a long-term repository). Overall, the issues are coherent, bounded, and addressable with a focused minor revision.
- > [[gpt3]]
The reviewers find the paper suitable for IEEE Software but point out some minor yet important weaknesses. They highlight concerns about the evaluation and claims made in the paper, suggesting that the user study should be considered a proof of concept rather than making strong assertions about reducing development time or cognitive load. Additionally, they note that the paper lacks a specific, practical AI-based application, weakening its novelty and relevance. Reviewers also seek a clearer justification for the use of Aspect-Oriented Programming and suggest improvements in delineating the pattern's coverage and enhancing evaluation scenarios. Overall, the issues raised are specific and can be addressed through a focused minor revision.
-
- **R1** / Minor
- Paper Summary:
This paper presents a design pattern for the integration of explanations using aspect-oriented programming (AOP) with the goal of standardizing and facilitating the development process of explanations, similar to what has been previously done for logging.
The authors implement a plugin for Visual Studio Code that supports the implementation of the pattern and preliminarily evaluate it by conducting an experiment involving seven participants. Results of the evaluation reveal benefits in terms of both development time and cognitive load.
- Strengths:
- The authors investigate a relevant and timely topic, i.e., providing a structural solution to manage explanations.
- The paper is very well-written, and it is also very easy to follow and understand.
- To present the pattern, the authors rely on the standard design pattern template.
- The manuscript comes with a link to a GitLab repo providing a sample implementation of the Explainer design pattern.
- I appreciate the presence of the threats to the validity section, where the authors admit as threats both the limited sample size and the ordering bias affecting the evaluation results.
- Points to Address:
- Due to the admitted threat of ordering bias, I would suggest stressing less the improvement in terms of development time and cognitive load. The evaluation should be viewed primarily as a proof of concept, without making any strong claims about the practical impact of adopting the pattern in real development scenarios.
- Since the Aspect-Oriented Programming is not standard in all modern development environments or languages, it would be good to motivate the reason why the authors focused on this paradigm, and the authors must discuss more the feasibility of extending this pattern to be used in different environments/languages.
- Overall Summary: This manuscript presents a novel solution to the integration of the explanations issue within complex software systems. While the evaluation is preliminary, the design is sound, and the accompanying tooling makes it a practical contribution for the IEEE Software audience. However, there is a need to work a little bit more on the writing by smoothing the results of the evaluation and by justifying the use of AOP more effectively, as well as explaining how it can be adapted to different environments and languages.
- **R2** / Minor
- Overall merit
- The proposed pattern is well-described and implemented in a Visual Studio Code plugin. However, the presented scenarios are somehow disconnected compared to the original motivation of the paper, i.e., assisting developers for AI-based systems. In this respect, the paper misses a concrete example of an AI-based application and how the proposed design pattern can concretely help developers. In addition, the pilot applications used in the user study are vaguely described. While the proof-of-concept is suitable to showcase a generic usage of the pattern, I strongly recommend authors to present a concrete AI-based application in which the Explainer Pattern can be used.
- Novelty
- While the proposed pattern can be helpful in practice, the paper does not present a concrete example applied to an actual AI-based project. In addition, existing challenges are not mentioned, and the authors should discuss which benefits the AOP pattern can bring to developers.
- Soundness
- Although the user study is small, it is adequate as a proof-of-concept. However, the scenario application is not reported, and, by looking in the provided repo, it seems limited to dummy projects. As mentioned, the authors should present a concrete example in an AI-based project, as it is the target domain of the approach. Concerning the evaluation metrics, only the required time is reported, although it might be useful to report how practitioners perceive the usefulness of the generated explanations.
- Relevance
- This paper proposes a novel design pattern with a minimal tool-based implementation. Therefore, it is relevant to the journal’s aims and scope.
- Verifiability
- The replication package is available on the GitLab repository, thus fostering the reusage of the tool.
- Presentation
- The paper is overall well-written and easy to follow.
- **R3** / Minor
- The paper addresses the scattered nature of code explanations and the need for a unifying structure, a relevant problem. The proposed design pattern is well motivated, follows established pattern description guidelines, and is likely to be actionable for practitioners. The evaluation reports promising initial results.
- The evaluation, however, is limited in scope. It considers only the integration of explanations using the pattern, and does not address the retrieval or use of these explanations. In addition, the study is small-scale (7 participants), with most participants being students or trainees, which restricts the extent to which conclusions can be generalized. This is discussed as part of the threats to validity.
- While generally clear, some aspects of the pattern description would benefit from further clarification. The pattern description states that explanations are stored "at all points that could be unclear (e.g., when executing a complicated method)", but it remains unclear how complexity is defined or measured. Moreover, since the motivation emphasizes the end-user, it is not obvious whether method complexity is the appropriate criterion for deciding where explanations are needed. If this is intentionally out of scope for the pattern, I would suggest to explicitly state this earlier in the paper to better delineate its scope.
- Minor comments:
- The supplementary material should be archived in a long-term repository such as Zenodo rather than GitLab
- For the feasibility evaluation, either cite the bachelor's thesis explicitly or consider omitting mentioning it