11 lines
12 KiB
Markdown
11 lines
12 KiB
Markdown
- ## **Review 1** (Major Revision)
|
|
- | <table class="datatable" border="0" style="border: 0px; border-collapse: collapse; width: 1353.76px; line-height: 14px; vertical-align: text-top;"><tbody><tr style="background-color: rgb(255, 255, 255); border: 0px; border-collapse: collapse;"><th style="vertical-align: bottom; padding: 3px 3px 3px 4px; border: 1px solid rgb(204, 204, 204); background-color: rgb(58, 85, 104); font-weight: bold; text-align: left; color: rgb(255, 255, 255); border-collapse: collapse;">Comments to Editor:</th></tr><tr style="background-color: rgb(255, 255, 255); border: 0px; border-collapse: collapse;"><td style="word-break: normal; padding: 1px; border: 1px solid rgb(204, 204, 204); vertical-align: top; table-layout: fixed; overflow-wrap: anywhere; display: block; min-height: 72px;">Summary of the submission:<br style="clear: left;">(Please summarize what the submission comprises)<br style="clear: left;"><br style="clear: left;">See comments to Author<br style="clear: left;"><br style="clear: left;">Analysis of the submission:<br style="clear: left;">(Please give a detailed analysis of your findings)<br style="clear: left;"><br style="clear: left;">See comments to Author<br style="clear: left;"><br style="clear: left;">Recommendations:<br style="clear: left;">(Based on your findings, please provide your recommendations to the editor<br style="clear: left;">in chief. Do you advise to accept, accept after revision or reject? Why?)<br style="clear: left;"><br style="clear: left;">In its current shape, the article looks more like a user manual of the tooling rather than a scientific work. As the authors seem to have some evaluation results not presented here, I would recommend accept after Major Revision. This revision must add a related work section and an evaluation section. Otherwise I would recommend to reject the paper.<br style="clear: left;"><br style="clear: left;">References used in this review:<br style="clear: left;">(Please summarize references here that were used in the review)</td></tr></tbody></table> | | |
|
|
| --- | --- | --- |
|
|
| <table class="datatable" border="0" style="border: 0px; border-collapse: collapse; width: 1353.76px; line-height: 14px; vertical-align: text-top;"><tbody><tr style="background-color: rgb(255, 255, 255); border: 0px; border-collapse: collapse;"><th style="vertical-align: bottom; padding: 3px 3px 3px 4px; border: 1px solid rgb(204, 204, 204); background-color: rgb(58, 85, 104); font-weight: bold; text-align: left; color: rgb(255, 255, 255); border-collapse: collapse;">Comments to Author:</th></tr><tr style="background-color: rgb(255, 255, 255); border: 0px; border-collapse: collapse;"><td style="word-break: normal; padding: 1px; border: 1px solid rgb(204, 204, 204); vertical-align: top; table-layout: fixed; overflow-wrap: anywhere; display: block; min-height: 72px;">The article presents Wodel-Edu which is a model-driven framework for the generation of exercises in model-based domains. The framework is domain independent and provides capabilities for defining: (1) new modeling languages, (2) mutations for the generation of exercises, (3) automatic grading of the exercises. The framework targets the Moodle platform, the web, Android and iOS.<br style="clear: left;">The presented tooling is obviously useful to the community and the presented approach is sound. The article is well written and pleasant to read. The tooling description is illustrated with many examples giving a good coverage of the supported capabilities.<br style="clear: left;">Weaknesses the article are the following:<br style="clear: left;">- There is no state of the art section to position the big amount of work achieved by the authors and to explain their methodological and design choices. A lot of work has been done in the subject of education of Model-Driven Engineering. The state of the art can for example cover direct approaches such as gamification, serious games or other non-direct ones like product-line engineering that provides ways to manage variability in models which could be another solution to tackle mutations. It would be interesting for the reader to have the positioning the choices made for Wodel-Edu.<br style="clear: left;">- There is no evaluation section. On the one hand, the presented framework provides a good support for professors by automating a lot of steps, but on the other hand, it could be interesting to have a proper evaluation of the tooling by professors in terms of easiness of use of the tool and the associated languages. This is identified as a future work. However, some experiments have already been conducted on students. Results of these experiments are not described in this article. It seems that these results have been submitted in another article under evaluation (reference 7). It would have been more valuable for the article to have them presented in this article, otherwise this article is similar to a kind of user manual of the framework. My recommendation would be to include these results.</td></tr></tbody></table> | | |
|
|
- ## **Review 2 (Reject)**
|
|
- | <table class="datatable" border="0" style="border: 0px; border-collapse: collapse; width: 1353.76px; line-height: 14px; vertical-align: text-top;"><tbody><tr style="background-color: rgb(255, 255, 255); border: 0px; border-collapse: collapse;"><th style="vertical-align: bottom; padding: 3px 3px 3px 4px; border: 1px solid rgb(204, 204, 204); background-color: rgb(58, 85, 104); font-weight: bold; text-align: left; color: rgb(255, 255, 255); border-collapse: collapse;">Comments to Editor:</th></tr><tr style="background-color: rgb(255, 255, 255); border: 0px; border-collapse: collapse;"><td style="word-break: normal; padding: 1px; border: 1px solid rgb(204, 204, 204); vertical-align: top; table-layout: fixed; overflow-wrap: anywhere; display: block; min-height: 72px;">Summary of the submission:<br style="clear: left;">(Please summarize what the submission comprises)<br style="clear: left;">Wodel-Edu supports the generation of exercises for any diagram-based domain making use of techniques of model-driven engineering. The tool allows generating mutants of a solution, which can then be used to generate different types of exercises (True-False, Multiple Choice, …). The exercises are exportable in different types of formats (a.o. Moodle, Android). The paper presents the technical design of the solution, illustrated with an example. There is no evaluation from the perspective of teachers or students.<br style="clear: left;"><br style="clear: left;">Analysis of the submission:<br style="clear: left;">(Please give a detailed analysis of your findings)<br style="clear: left;"><br style="clear: left;">The problem addressed by the work of the authors is real and pertinent. I consider the work to be highly relevant. The general idea behind the proposed solution is good, and the general design is good too.<br style="clear: left;"><br style="clear: left;">I suggest a reject based on the fact that we were not able to install and run the software.<br style="clear: left;"><br style="clear: left;">In case of resubmission, the comments to the authors provide other points for improvement of the paper.</td></tr></tbody></table> | | |
|
|
| --- | --- | --- |
|
|
| <table class="datatable" border="0" style="border: 0px; border-collapse: collapse; width: 1353.76px; line-height: 14px; vertical-align: text-top;"><tbody><tr style="background-color: rgb(255, 255, 255); border: 0px; border-collapse: collapse;"><th style="vertical-align: bottom; padding: 3px 3px 3px 4px; border: 1px solid rgb(204, 204, 204); background-color: rgb(58, 85, 104); font-weight: bold; text-align: left; color: rgb(255, 255, 255); border-collapse: collapse;">Comments to Author:</th></tr><tr style="background-color: rgb(255, 255, 255); border: 0px; border-collapse: collapse;"><td style="word-break: normal; padding: 1px; border: 1px solid rgb(204, 204, 204); vertical-align: top; table-layout: fixed; overflow-wrap: anywhere; display: block; min-height: 72px;">Wodel-Edu supports the generation of exercises for any diagram-based domain making use of techniques of model-driven engineering. The tool allows generating mutants of a solution, which can then be used to generate different types of exercises (True-False, Multiple Choice, …). The exercises are exportable in different types of formats (a.o. Moodle, Android). The paper presents the technical design of the solution, illustrated with an example.<br style="clear: left;"><br style="clear: left;">The problem addressed by the work of the authors is real and pertinent. I consider the work to be highly relevant. The general idea behind the proposed solution is good, and the general design is good too.<br style="clear: left;"><br style="clear: left;">We attempted to install Wodel as Eclipse plugin. Installing the plugin yields the following error:<br style="clear: left;">Cannot complete the install because one or more required items could not be found.<br style="clear: left;">Software being installed: Wodel 1.0.0.202302271122 (wodel.feature.group 1.0.0.202302271122)<br style="clear: left;">Missing requirement: MutStaticPostprocessor 1.0.0.202302271122 (wodel.static 1.0.0.202302271122) requires 'osgi.bundle; org.emfjson.core 0.10.0' but it could not be found<br style="clear: left;"><br style="clear: left;">Hard to find any online help. Furthermore, the following error is thrown as well:<br style="clear: left;"><br style="clear: left;">Cannot satisfy dependency:<br style="clear: left;">From: Wodel 1.0.0.202302271122 (wodel.feature.group 1.0.0.202302271122)<br style="clear: left;">To: org.eclipse.equinox.p2.iu; wodel.static [1.0.0.202302271122,1.0.0.202302271122]<br style="clear: left;">Tried this://stackoverflow.com/a/75475174.<br style="clear: left;">But with no solution.<br style="clear: left;">To ensure the reproducability of the research, the authors should try to make their solution usable with reasonable technical knowledge.<br style="clear: left;"><br style="clear: left;"><br style="clear: left;">The authors are also encouraged to address the following items in their paper:<br style="clear: left;"><br style="clear: left;">1. The solution still requires quite some technical knowledge for its usage. The wiki mentions as prerequisite: "Before proceeding with this tutorial you should have a good understanding of EMF." That's not a light prerequisite … Authors should develop a clear manual and try to make the software as easy to use as possible.<br style="clear: left;">2. In their manual/description, the authors should address the limitations of Wodel-edu: what can it NOT be used for? The running example demonstrates how to use the tool for what I would call "deterministic" exercises. This is, exercises where there is a literal translation from the assignment to the solution. On the website, one can find sample UML exercises. While I see the value of generating mutants for UML class diagrams, as requirements are always slightly fuzzy, it remains to be seen if a mutant is *really* wrong or could be acceptable as well (e.g. varying between mandatory versus optional, or choosing to model something as an attribute versus a class). In the intro, the authors also refer to the fact the " … most works achieve automated grading by means of comparison algorithms [2,11], which may be problematic if the exercise admits multiple solutions." Since they continue with "To fill this gap, …." This raises the impression that their solution would be able to handle exercises with multiple solutions. This seems not the case? And if it does, please clarify.<br style="clear: left;"></td></tr></tbody></table> | | |
|
|
- ## My recommendation
|
|
- The work requires a Major revision before it can be considered for publication. The reviewers have pointed out several issues that need to be addressed in the revised manuscript. Specifically, one of the reviewers encountered difficulties in installing and using the proposed plugin and could not find adequate documentation to guide them. Furthermore, the reviewers suggested that you should clarify the novelty and the scientific contribution of your proposed tool and provide more evidence to support your claims. Detailed comments are given below.
|
|
- |