75 lines
6.5 KiB
Markdown
75 lines
6.5 KiB
Markdown
-
|
|
type:: [[REVIEWS]]
|
|
venue:: [[TOSEM]]
|
|
year:: 2023
|
|
full-title:: Measuring and Clustering Heterogeneous Chatbot Designs
|
|
date:: [[31-03-2023]] - 13:36
|
|
status:: [[DONE]]
|
|
external-link::
|
|
file:: 
|
|
|
|
- [[Highlights]]
|
|
- ((64270c19-5357-4c40-a759-2ce6d8e898ee))
|
|
- - What's the need / goal of creating groups/clusters of chatbot designs? Are talking about model-based specifications?
|
|
- - Why is it important to group chatbots?
|
|
- ((64270cdb-7b9c-4d56-95e1-581469801ba5))
|
|
- - It seems these are the two development platforms that are considered in the paper.
|
|
- - ((64270d0d-3cb8-46db-ba98-f85e8ccdb270))
|
|
- - This seems to be a possible goal/focus of the paper.
|
|
- - ((64270d3e-3609-4143-acb2-2bf21a05aa9c))
|
|
- - ((64270ec1-fa6a-4192-9b1e-b928d0e536a2))
|
|
- #+BEGIN_IMPORTANT
|
|
GOALS of CHATBOT STATIC METRICS**:
|
|
((64307221-6d9a-47af-82e5-54a16fa70021))
|
|
#+END_IMPORTANT
|
|
-
|
|
- Interesting to see what are the characeristics that are peculiar for identifying the similarity and dissimilarity of chatbots
|
|
- - ((643072ad-0c81-45e5-b15e-949da1825196))
|
|
- -
|
|
- - ((64356f75-f0e5-4767-a91d-fa6ca0f3aa68))
|
|
- - Further than this, I think we need to see a fragment of a chatbot specification with one of the existing technologies (e.g., Dialgflow or Rasa that are considered later in the paper). This is necessary because most of the contet after relies on the management and analysis of chatbost specifications. Users that are not aware of such technologies might have difficulties in grasping the meaning, e.g., of uploading of chatbot, or **clustering chatbots**!!!
|
|
- - ((6433bd5f-1c82-4f9a-94f3-a1f31e280fdc))
|
|
- - ((64356f75-f0e5-4767-a91d-fa6ca0f3aa68))
|
|
- - ((6433bf2a-89ee-467a-80a4-3d55bc1628ff))
|
|
- - can the approach assess dynamic quality properties?
|
|
- - ((6433c204-1b23-4b7d-9074-fd6f98b13553))
|
|
- - ((6433c311-1f72-4b52-9c0b-b7bde69046dc))
|
|
- - Add another column to show if the metrics are going to affect the quality positively or negatively
|
|
- - ((6433d3a9-c5be-479e-b8c0-a055bf88ffb2))
|
|
- - Is there any way to model this explicit.
|
|
- - ((6433d40e-0f51-4f8e-85aa-6822d3061732))
|
|
- - ((6433d749-9f82-4aaa-856d-3492003fd1b8))
|
|
- - Why those metrics?
|
|
- - ((643567d9-76d1-4749-b66d-0eb46ea8c2b4))
|
|
- - What's the meaning in the space?
|
|
- - ((6435689b-3e6c-4371-9a76-13a96f421f89))
|
|
- - What's the meaning of the position in the space?
|
|
- - ((64356a38-076f-4fa8-b085-ea63f91f0ca9))
|
|
- - It is important to mention that the work assumes that each chatbot has a self-contained specification that can be consumed / analyzed by the proposed techniques.
|
|
- - ((643575c8-5044-40fe-b3ad-830d3a8b896e))
|
|
- - Why is this interesting to know?
|
|
- - ((6435769c-5592-45d1-883f-8919ffb5423e))
|
|
- - Need to check if the considered quality issues have been manually checked/verified
|
|
- - Same for the created clusters
|
|
- - ((64357724-3fbd-4e25-a199-67b40ec7d292))
|
|
- -
|
|
- - ((643579b1-5765-46ee-a5fb-1f76f2975cd4))
|
|
- - I would extend this answer to clearly say why all of this is relevant for potential users/adopters of the proposed technology. Why decision can benefit the proposed metrics?
|
|
- - ((64357a61-0924-486b-8f07-5a592295d6ad))
|
|
- - Have the identified issues been confirmed by manual investigations from human experts?
|
|
- - ((64357c1f-9bef-4344-8c43-b2f99e5bba28))
|
|
- - There is a bias here. You are assuming that chatbots that do not have low or high metric outliers do not have quality issues. This needs to be proven.
|
|
- - This recalls the possibility of introducing custom definition of quality issues.
|
|
- - This is related to one of the external validity threats.
|
|
- - ((64357d4a-19c0-4004-9039-51a5c1d60a17))
|
|
- -
|
|
- [[Comments]]
|
|
- - This paper presents a method to facilitate the analysis and comparison of chatbots. The proposed approach suggests using a set of metrics and clustering techniques to assess various chatbot designs. The authors implemented the method using the Asymob tool, which provides a web interface and a REST API for programmatic access. The authors evaluated the method's effectiveness by applying it to a dataset of 259 chatbot designs and provided valuable insights into the usefulness of the metrics and clustering.
|
|
- - Overall, the paper is well-written and structured, and the proposed solution is technically sound. However, some concerns need to be addressed:
|
|
- - The need for automated mechanisms supporting the analysis of chatbot design should be better motivated early in the paper. Concrete examples and situations can convince the reader about the necessity of metrics and grouping chatbot designs.
|
|
- - Section 2 should be enhanced by adding an explanatory design using one of the technologies considered later in the paper, such as Dialogflow or Rasa, to introduce the problem better. Such introductory paragraphs are needed because much of the content in the paper relies on or refers to the problem of managing and analyzing chatbot specifications, even though the reader has yet to be introduced to them. Notably, readers unfamiliar with existing technologies for chatbot designs might have difficulty grasping the meaning of "uploading of chatbot" or "clustering chatbots" discussed later in the manuscript.
|
|
- - An important shortcoming of the paper is related to the notion of quality. Several quality models have been introduced in different application domains, including software engineering, over the last few decades. Consequently, authors should clearly define the underlying quality model used throughout the paper, and as future work, they could investigate the possibility of supporting the definition of custom quality models for assessing the quality of chatbot designs.
|
|
- - Regarding the evaluation, there are concerns related to RQ2. In particular, in line 1430, the authors mention that they have "manually checked all chatbots with low or high metric outliers to assess if they have problems." There is a bias here, as authors assume that chatbots that do not have low or high metric outliers do not have quality issues. This needs to be proven, and it highlights the possibility/need to employ custom definitions of quality issues (see my previous point).
|
|
- - As minor comments:
|
|
- - Concerning Fig. 5, the authors should better explain the meaning of the chatbot position in the space defined by the three considered dimensions.
|
|
- - The selection of the chatbot design metrics given in Table 1 should be motivated/supported, e.g., by some reference. |