546 lines
20 KiB
Markdown
546 lines
20 KiB
Markdown
file:: [TOSEM-2023-0032_Proof_hi_1680262636436_0.pdf](../assets/TOSEM-2023-0032_Proof_hi_1680262636436_0.pdf)
|
||
file-path:: ../assets/TOSEM-2023-0032_Proof_hi_1680262636436_0.pdf
|
||
|
||
- Measuring and Clustering Heterogeneous Chatbot Designs
|
||
ls-type:: annotation
|
||
hl-page:: 2
|
||
hl-color:: green
|
||
id:: 6426c651-96af-4297-8073-88830f4ece78
|
||
- suite of metrics for chatbot designs,
|
||
ls-type:: annotation
|
||
hl-page:: 2
|
||
hl-color:: green
|
||
id:: 6426c65b-52e5-4d0e-9bc9-83620aeb6a2c
|
||
- rouping chatbots along their conversation topics and design features
|
||
ls-type:: annotation
|
||
hl-page:: 2
|
||
hl-color:: green
|
||
id:: 6426c663-4983-4bb7-b7c2-3daf1ac4a64e
|
||
- 59 Dialogflow and Rasa chatbots
|
||
ls-type:: annotation
|
||
hl-page:: 2
|
||
hl-color:: green
|
||
id:: 6426c67d-4a4f-484e-960d-bc33397f0495
|
||
- improved
|
||
ls-type:: annotation
|
||
hl-page:: 2
|
||
hl-color:: green
|
||
id:: 6426c7b4-903e-4762-8f4b-faa9f53c7563
|
||
- We have refined our suite of metrics and extended it with a new readability metric, expanding the explanations and rationale of the metrics.
|
||
ls-type:: annotation
|
||
hl-page:: 3
|
||
hl-color:: green
|
||
id:: 64270a94-e163-4aa2-a155-66a778a97e27
|
||
- Francesco Basciani, Juri Di Rocco, Davide Di Ruscio, Ludovico Iovino, and Alfonso Pierantonio. 2016. Automated Clustering of Metamodel Repositories. In Proc. 28th Int. Conf. on Advanced Information Syst. Eng. (LNCS, Vol. 9694). Springer, 342–358.
|
||
ls-type:: annotation
|
||
hl-page:: 38
|
||
hl-color:: green
|
||
id:: 64270ae3-06c3-4eb7-b8c5-4955f5045e74
|
||
- ESTHER GUERRA
|
||
ls-type:: annotation
|
||
hl-page:: 4
|
||
hl-color:: green
|
||
id:: 64270b12-3b66-4f3d-9c79-4b90a54a5a31
|
||
- Measuring and Clustering Heterogeneous Chatbot Designs
|
||
ls-type:: annotation
|
||
hl-page:: 4
|
||
hl-color:: green
|
||
id:: 64270b16-aa4c-44a4-be32-1aa114a999b7
|
||
- JUAN DE LARA
|
||
ls-type:: annotation
|
||
hl-page:: 4
|
||
hl-color:: green
|
||
id:: 64270b1f-e304-4d66-af8c-bdd9b073ee61
|
||
- there are hardly any mechanisms to compare and cluster chatbots developed with heterogeneous technologies
|
||
ls-type:: annotation
|
||
hl-page:: 4
|
||
hl-color:: yellow
|
||
id:: 64270c19-5357-4c40-a759-2ce6d8e898ee
|
||
hl-stamp:: 1680280620522
|
||
- 21 metrics for chatbot designs
|
||
ls-type:: annotation
|
||
hl-page:: 4
|
||
hl-color:: green
|
||
id:: 64270c5e-847d-46e8-b212-52d7e9c52f54
|
||
hl-stamp:: 1680280672632
|
||
- 259 Dialogflow and Rasa chatbots
|
||
ls-type:: annotation
|
||
hl-page:: 4
|
||
hl-color:: green
|
||
id:: 64270cdb-7b9c-4d56-95e1-581469801ba5
|
||
- early detection of quality issues
|
||
ls-type:: annotation
|
||
hl-page:: 4
|
||
hl-color:: yellow
|
||
id:: 64270d0d-3cb8-46db-ba98-f85e8ccdb270
|
||
hl-stamp:: 1680280847796
|
||
- chatbot comprehension, search and comparison
|
||
ls-type:: annotation
|
||
hl-page:: 4
|
||
hl-color:: green
|
||
id:: 64270d3e-3609-4143-acb2-2bf21a05aa9c
|
||
hl-stamp:: 1680280898517
|
||
- Chatbots are also used to assist workers in different domains, like software engineering
|
||
ls-type:: annotation
|
||
hl-page:: 5
|
||
hl-color:: green
|
||
id:: 64270e27-b4aa-4ec8-b175-babbfd85b037
|
||
- artner estimates that there are more than 2000 natural language technology providers, with a significant number of them offering facilities to create chatbots
|
||
ls-type:: annotation
|
||
hl-page:: 5
|
||
hl-color:: green
|
||
id:: 64270e35-49a7-407f-a40e-b02580704e6b
|
||
- tools for testing chatbots
|
||
ls-type:: annotation
|
||
hl-page:: 5
|
||
hl-color:: green
|
||
id:: 64270e85-54c0-4226-9008-ae0331127bab
|
||
- guide and control the quality of chatbots throughout their development and maintenance, becoming a complement to testing
|
||
ls-type:: annotation
|
||
hl-page:: 5
|
||
hl-color:: green
|
||
id:: 64270ec1-fa6a-4192-9b1e-b928d0e536a2
|
||
- there are hardly any metric proposals for chatbot designs
|
||
ls-type:: annotation
|
||
hl-page:: 5
|
||
hl-color:: purple
|
||
id:: 643071f5-9869-46bb-8b0f-aabd7c516732
|
||
- detect potential problems related to user experience (e.g., complex conversation flows, hard-to-read chatbot answers); as indicators of chatbot complexity; to compare properties of heterogeneous chatbots; to discover commonalities and cluster similar chatbots; and to understand how different implementation platforms can impact on the chatbot design
|
||
ls-type:: annotation
|
||
hl-page:: 5
|
||
hl-color:: purple
|
||
id:: 64307221-6d9a-47af-82e5-54a16fa70021
|
||
- This calls for mechanisms to measure the similarity or dissimilarity between chatbots, even if developed with different technologies, to understand their commonalities and enable searching for chatbots akin to a given one, e.g., as a first step towards the reuse of existing chatbots
|
||
ls-type:: annotation
|
||
hl-page:: 5
|
||
hl-color:: yellow
|
||
id:: 643072ad-0c81-45e5-b15e-949da1825196
|
||
- we propose a suite of 21 static metrics for chatbot designs, and two methods for clustering chatbots
|
||
ls-type:: annotation
|
||
hl-page:: 5
|
||
hl-color:: purple
|
||
id:: 6433ba23-b920-474f-ad4d-406cd9fa63da
|
||
- This is useful, e.g., to identify chatbots on the same topic for their reuse; to facilitate the construction of recommender systems for chatbots; to organise large chatbot repositories according to their domain; or to facilitate chatbot search.
|
||
ls-type:: annotation
|
||
hl-page:: 5
|
||
hl-color:: purple
|
||
id:: 6433ba51-985a-4318-a639-f0a7ab14412f
|
||
- metrics and the clustering methods over Dialogflow and Rasa chatbots
|
||
ls-type:: annotation
|
||
hl-page:: 6
|
||
hl-color:: green
|
||
id:: 6433bb84-0020-4fe7-99d8-cd95d5d5bc53
|
||
- 20 chatbot design metrics, proposed their technology-independent definition atop Conga, made their implementation available via an API, and evaluated their suitability on 12 chatbots.
|
||
ls-type:: annotation
|
||
hl-page:: 6
|
||
hl-color:: green
|
||
id:: 6433bc70-d72e-415b-977c-bc7ab83d864a
|
||
- principal component analysis.
|
||
ls-type:: annotation
|
||
hl-page:: 6
|
||
hl-color:: yellow
|
||
id:: 6433bc90-eb00-4b86-a2a7-282da8bdc294
|
||
- we are interested in task-oriented chatbots
|
||
ls-type:: annotation
|
||
hl-page:: 6
|
||
hl-color:: green
|
||
id:: 6433bd3b-1286-4416-8023-8d476c77c813
|
||
- [:span]
|
||
ls-type:: annotation
|
||
hl-page:: 7
|
||
hl-color:: green
|
||
id:: 6433bd5f-1c82-4f9a-94f3-a1f31e280fdc
|
||
hl-type:: area
|
||
hl-stamp:: 1681112412961
|
||
- ystemcentric
|
||
ls-type:: annotation
|
||
hl-page:: 7
|
||
hl-color:: green
|
||
id:: 6433be90-d066-4f2a-a9a0-14a21683fe72
|
||
- Content-centric
|
||
ls-type:: annotation
|
||
hl-page:: 7
|
||
hl-color:: green
|
||
id:: 6433be97-64ff-4dfe-ab62-25f0cb8cbd80
|
||
- Visualcentric
|
||
ls-type:: annotation
|
||
hl-page:: 7
|
||
hl-color:: green
|
||
id:: 6433be9f-6267-42bf-8aa4-4d0e7c7604a6
|
||
- conversation-centric
|
||
ls-type:: annotation
|
||
hl-page:: 7
|
||
hl-color:: green
|
||
id:: 6433bea2-ea0d-4b36-bcee-0c68120b001e
|
||
- conversation patterns [35] and conversation design principles
|
||
ls-type:: annotation
|
||
hl-page:: 8
|
||
hl-color:: green
|
||
id:: 6433bef4-67d3-4e53-b4d2-7a9af9d85ab5
|
||
- dynamic execution of the chatbot
|
||
ls-type:: annotation
|
||
hl-page:: 8
|
||
hl-color:: yellow
|
||
id:: 6433bf2a-89ee-467a-80a4-3d55bc1628ff
|
||
- A means to facilitate finding artefacts of interest (chatbots in our case) is to organise them into meaningful group
|
||
ls-type:: annotation
|
||
hl-page:: 8
|
||
hl-color:: green
|
||
id:: 6433c0e8-418f-4a75-9fc1-d0967df26e44
|
||
- reusing existing artefacts in a cluster of interest
|
||
ls-type:: annotation
|
||
hl-page:: 8
|
||
hl-color:: green
|
||
id:: 6433c0ef-d481-420e-a21a-1f3432ed1dfc
|
||
- K-means and hierarchical clustering require setting the number of clusters as hyperparameters, while DBSCAN has other parameters like the neighbourhood distance and the minimum number of points required to form a dense region
|
||
ls-type:: annotation
|
||
hl-page:: 8
|
||
hl-color:: green
|
||
id:: 6433c14d-36ef-4daf-a8f0-49d9e92b1a60
|
||
- In our proposal to chatbot clustering, we use similar encodings and techniques, but focus on chatbot designs instead of meta-models
|
||
ls-type:: annotation
|
||
hl-page:: 9
|
||
hl-color:: purple
|
||
id:: 6433c174-0930-4fac-a5f4-8b96565043c8
|
||
- clustering for chatbot designs has not been studied yet in the literature
|
||
ls-type:: annotation
|
||
hl-page:: 9
|
||
hl-color:: purple
|
||
id:: 6433c18f-fbe8-411f-85fc-ea07febb53ce
|
||
- Section 4.1 starts by introducing the chatbot design notation
|
||
ls-type:: annotation
|
||
hl-page:: 9
|
||
hl-color:: purple
|
||
id:: 6433c1ae-2826-4eee-b30d-ea8685bfb55c
|
||
- Section 4.2 details our proposed metric
|
||
ls-type:: annotation
|
||
hl-page:: 9
|
||
hl-color:: purple
|
||
id:: 6433c1b5-9ee7-4f3c-9e18-97b25c1c4943
|
||
- Conga
|
||
ls-type:: annotation
|
||
hl-page:: 9
|
||
hl-color:: purple
|
||
id:: 6433c1df-9e2f-451e-a94a-9ef997a9aebe
|
||
- [:span]
|
||
ls-type:: annotation
|
||
hl-page:: 10
|
||
hl-color:: green
|
||
id:: 6433c204-1b23-4b7d-9074-fd6f98b13553
|
||
hl-type:: area
|
||
hl-stamp:: 1681113602953
|
||
- Intents
|
||
ls-type:: annotation
|
||
hl-page:: 9
|
||
hl-color:: green
|
||
id:: 6433c22a-309c-49d9-b724-9fb588e852bf
|
||
- Entities
|
||
ls-type:: annotation
|
||
hl-page:: 9
|
||
hl-color:: green
|
||
id:: 6433c22d-c52c-46f7-81db-17b8f1176a99
|
||
- t Actions
|
||
ls-type:: annotation
|
||
hl-page:: 9
|
||
hl-color:: green
|
||
id:: 6433c233-5eb0-4e84-851d-023d0744ae3a
|
||
- Flow
|
||
ls-type:: annotation
|
||
hl-page:: 9
|
||
hl-color:: green
|
||
id:: 6433c235-d772-47d5-8225-7c15d957b235
|
||
- For example, a chatbot can declare a simple entity for drink sizes with literals small, medium and large in English, and additionally define synonyms regular for medium and big for large.
|
||
ls-type:: annotation
|
||
hl-page:: 9
|
||
hl-color:: green
|
||
id:: 6433c2c2-9143-4196-9872-e8190587bc6e
|
||
- [:span]
|
||
ls-type:: annotation
|
||
hl-page:: 11
|
||
hl-color:: green
|
||
id:: 6433c311-1f72-4b52-9c0b-b7bde69046dc
|
||
hl-type:: area
|
||
hl-stamp:: 1681113870406
|
||
- Global metrics. We start explaining global metrics. These measure the number of intents(INT), entities (ENT) and conversation flows (FLOW, PATH), and also include understanding and user experience metrics (CNF, SNT)
|
||
ls-type:: annotation
|
||
hl-page:: 11
|
||
hl-color:: green
|
||
id:: 6433c350-559c-4c7e-a70a-21105ef239a9
|
||
- PATH measures the number of paths a conversation may take, which is an indicator of conversation complexity
|
||
ls-type:: annotation
|
||
hl-page:: 11
|
||
hl-color:: green
|
||
id:: 6433c399-7b88-4bdf-98a6-fac791bb6df9
|
||
- The combined use of FLOW and PATH can help detecting deviations of some design principles
|
||
ls-type:: annotation
|
||
hl-page:: 11
|
||
hl-color:: green
|
||
id:: 6433c409-b375-49ff-bc86-3021d87424a9
|
||
- ntent metrics measure quality properties of each intent with respect to the expected user utterances and the bot output phrases
|
||
ls-type:: annotation
|
||
hl-page:: 12
|
||
hl-color:: green
|
||
id:: 6433c49c-3c5b-441a-a622-b2d1032cef34
|
||
- interaction complexity,
|
||
ls-type:: annotation
|
||
hl-page:: 12
|
||
hl-color:: green
|
||
id:: 6433c4c9-9554-44c9-8d26-e7185c6ee7d5
|
||
hl-stamp:: 1681114321426
|
||
- user cognitive load and speed up the completion of the intended tas
|
||
ls-type:: annotation
|
||
hl-page:: 13
|
||
hl-color:: green
|
||
id:: 6433c5e6-33cc-4b9f-bca3-b0d2de20c997
|
||
- chatbot vocabulary
|
||
ls-type:: annotation
|
||
hl-page:: 13
|
||
hl-color:: green
|
||
id:: 6433d39e-4d96-447e-af37-700174196584
|
||
- internal quality features (e.g., complex/simple conversations, large/succinct outputs). The latter groups chatbots by conversation top
|
||
ls-type:: annotation
|
||
hl-page:: 13
|
||
hl-color:: yellow
|
||
id:: 6433d3a9-c5be-479e-b8c0-a055bf88ffb2
|
||
- metric INT (i.e., number of intents) would create groups of chatbots with similar size complexit
|
||
ls-type:: annotation
|
||
hl-page:: 13
|
||
hl-color:: green
|
||
id:: 6433d3f9-5203-47b0-950d-a59c625eaebe
|
||
- [:span]
|
||
ls-type:: annotation
|
||
hl-page:: 14
|
||
hl-color:: green
|
||
id:: 6433d40e-0f51-4f8e-85aa-6822d3061732
|
||
hl-type:: area
|
||
hl-stamp:: 1681118220789
|
||
- principal component analysis (PCA) o
|
||
ls-type:: annotation
|
||
hl-page:: 14
|
||
hl-color:: green
|
||
id:: 6433d722-2ee5-40ec-a9d6-94714b27e880
|
||
- he latter is possible since PCA permits explaining how each metric contributes to each component, by providing the loading factor of the metric for the component. This effectively groups the metrics that contribute the most to each component.
|
||
ls-type:: annotation
|
||
hl-page:: 14
|
||
hl-color:: green
|
||
id:: 6433d739-6293-4cc8-a657-23c4145a4490
|
||
- PATH, INT, FLOW, WL, PPTP and FPATH.
|
||
ls-type:: annotation
|
||
hl-page:: 14
|
||
hl-color:: yellow
|
||
id:: 6433d749-9f82-4aaa-856d-3492003fd1b8
|
||
hl-stamp:: 1681119074022
|
||
- Vocabulary-based clustering
|
||
ls-type:: annotation
|
||
hl-page:: 15
|
||
hl-color:: green
|
||
id:: 64348546-40a4-4e18-9283-55d312c65b9f
|
||
- we provide another clustering method based on the chatbot conversation topics
|
||
ls-type:: annotation
|
||
hl-page:: 15
|
||
hl-color:: purple
|
||
id:: 64348554-62cf-489c-838f-d80ab3ccaf72
|
||
- training phrases of the intents, the chatbot output phrases, and the literals and synonyms in entities.
|
||
ls-type:: annotation
|
||
hl-page:: 15
|
||
hl-color:: green
|
||
id:: 64348598-0791-4c13-b203-55fe7b82ec0b
|
||
- Cluster projection into three dimensions,
|
||
ls-type:: annotation
|
||
hl-page:: 14
|
||
hl-color:: yellow
|
||
id:: 643567d9-76d1-4749-b66d-0eb46ea8c2b4
|
||
- In the graphic, each dot represents a chatbot, and the shape of the dot identifies the cluster where the chatbot belongs.
|
||
ls-type:: annotation
|
||
hl-page:: 14
|
||
hl-color:: yellow
|
||
id:: 6435689b-3e6c-4371-9a76-13a96f421f89
|
||
- Latent Semantic Analysis
|
||
ls-type:: annotation
|
||
hl-page:: 15
|
||
hl-color:: purple
|
||
id:: 64356a14-052b-48c8-b7e6-c804d35f246e
|
||
- set of chatbots
|
||
ls-type:: annotation
|
||
hl-page:: 15
|
||
hl-color:: yellow
|
||
id:: 64356a38-076f-4fa8-b085-ea63f91f0ca9
|
||
- relevance of a word to a chatbot,
|
||
ls-type:: annotation
|
||
hl-page:: 16
|
||
hl-color:: green
|
||
id:: 64356ab2-9bc1-4ee9-807d-2a6cb170f649
|
||
- TF-IDF weight of a word for a chatbot
|
||
ls-type:: annotation
|
||
hl-page:: 16
|
||
hl-color:: green
|
||
id:: 64356b1b-d19d-4f5b-acc7-8fa7e4995558
|
||
- Hence, the weights tend to filter out common terms.
|
||
ls-type:: annotation
|
||
hl-page:: 16
|
||
hl-color:: purple
|
||
id:: 64356b2c-d02c-46f9-b646-3d6f913092eb
|
||
- osine similarit
|
||
ls-type:: annotation
|
||
hl-page:: 16
|
||
hl-color:: green
|
||
id:: 64356b47-da7b-44f5-a50c-bc86f1d4ce01
|
||
- agglomerative clustering
|
||
ls-type:: annotation
|
||
hl-page:: 16
|
||
hl-color:: green
|
||
id:: 64356b61-884f-4f24-a58c-f986e290c0cf
|
||
- Clustered similarity matrix of chatbots.
|
||
ls-type:: annotation
|
||
hl-page:: 18
|
||
hl-color:: green
|
||
id:: 64356bec-b05f-4ec3-8a8e-b61e511d7d88
|
||
- Chatbot working schema
|
||
ls-type:: annotation
|
||
hl-page:: 7
|
||
hl-color:: yellow
|
||
id:: 64356f75-f0e5-4767-a91d-fa6ca0f3aa68
|
||
- across all chatbots in the repository.
|
||
ls-type:: annotation
|
||
hl-page:: 18
|
||
hl-color:: green
|
||
id:: 643570a6-f407-4bd3-aeb0-a45c198f270e
|
||
- clustering chatbots
|
||
ls-type:: annotation
|
||
hl-page:: 18
|
||
hl-color:: green
|
||
id:: 643570b9-1a4b-41f3-8d44-90156ad6c9ff
|
||
- his core has an extensible design, which makes it easy to add new types of metrics, clustering criteria and chatbot technologie
|
||
ls-type:: annotation
|
||
hl-page:: 19
|
||
hl-color:: green
|
||
id:: 643571d5-ad05-41a6-b3a5-82b342c89b9b
|
||
- This way, Asymob computes the metrics on Conga models, independently of any chatbot implementation platform.
|
||
ls-type:: annotation
|
||
hl-page:: 19
|
||
hl-color:: green
|
||
id:: 6435738e-3ebc-4e8c-b966-6562e955bf50
|
||
- Asymob supports some third-party technologies to simplify the implementation of new metrics
|
||
ls-type:: annotation
|
||
hl-page:: 19
|
||
hl-color:: green
|
||
id:: 643573a6-12d4-4dec-885e-9883de5c1c6c
|
||
- Dialogflow and Rasa) into Conga.
|
||
ls-type:: annotation
|
||
hl-page:: 20
|
||
hl-color:: green
|
||
id:: 643573c8-c6f3-4642-98dd-99927d75eb44
|
||
- Agent.json describes global chatbot features, like its name, definition languages, or connection data to external services (the webhook).
|
||
ls-type:: annotation
|
||
hl-page:: 20
|
||
hl-color:: green
|
||
id:: 6435746d-1727-46d2-99a9-591ec23c2e19
|
||
- Intents
|
||
ls-type:: annotation
|
||
hl-page:: 20
|
||
hl-color:: purple
|
||
id:: 64357482-5893-4b2a-a454-36f065ea7160
|
||
hl-stamp:: 1681224841238
|
||
- usefulness of our metrics
|
||
ls-type:: annotation
|
||
hl-page:: 23
|
||
hl-color:: green
|
||
id:: 64357500-0e59-4814-a243-eb231c28aa52
|
||
- efficacy of our vocabulary-based clustering
|
||
ls-type:: annotation
|
||
hl-page:: 23
|
||
hl-color:: green
|
||
id:: 64357505-9559-477a-844a-1424ed2318c3
|
||
- chatbots in the wild compare with respect to their size, conversation style, outputs, expected inputs and vocabulary?
|
||
ls-type:: annotation
|
||
hl-page:: 23
|
||
hl-color:: yellow
|
||
id:: 643575c8-5044-40fe-b3ad-830d3a8b896e
|
||
- quality issues in real chatbots
|
||
ls-type:: annotation
|
||
hl-page:: 23
|
||
hl-color:: yellow
|
||
id:: 6435769c-5592-45d1-883f-8919ffb5423e
|
||
- meaningful groups of semantically related chatbots?
|
||
ls-type:: annotation
|
||
hl-page:: 24
|
||
hl-color:: yellow
|
||
id:: 64357724-3fbd-4e25-a199-67b40ec7d292
|
||
- we compared different dimensions of the chatbots (size, conversation style, responses, expected user utterances and vocabulary) based on the chatbots’ metric values
|
||
ls-type:: annotation
|
||
hl-page:: 24
|
||
hl-color:: green
|
||
id:: 6435778a-fe77-422a-a711-10d5b2f8ff15
|
||
- We do not expect every discordant metric value to signal a real problem, but instead, we are interested in assessing the usefulness of metrics as an inexpensive chatbot quality assurance mechanism (e.g., compared to testing) that can be used early in the development process
|
||
ls-type:: annotation
|
||
hl-page:: 24
|
||
hl-color:: purple
|
||
id:: 643577ad-fb25-4a39-a817-ece0ecbc6e48
|
||
hl-stamp:: 1681225647635
|
||
- size
|
||
ls-type:: annotation
|
||
hl-page:: 25
|
||
hl-color:: green
|
||
id:: 643578f1-018b-461d-b6af-828dac934a0f
|
||
- conversation style
|
||
ls-type:: annotation
|
||
hl-page:: 25
|
||
hl-color:: green
|
||
id:: 643578f6-95b3-4109-8a69-26b255baa946
|
||
- outputs
|
||
ls-type:: annotation
|
||
hl-page:: 25
|
||
hl-color:: green
|
||
id:: 643578f8-f070-4e8d-aa8c-2b802ebdf903
|
||
- expected inputs, vocabulary (pertinent for RQ1) and implementation platform (RQ1.1)
|
||
ls-type:: annotation
|
||
hl-page:: 25
|
||
hl-color:: green
|
||
id:: 64357900-5693-45d4-9ba5-e6c62c6342c1
|
||
- Naturally, chatbots with more intents tend to have more flows, and therefore, more paths.
|
||
ls-type:: annotation
|
||
hl-page:: 26
|
||
hl-color:: purple
|
||
id:: 64357937-6d96-4dbd-9fe7-9e6c1be0cf5e
|
||
- 7.3.7 Answering RQ1 and RQ1.1. With respect to RQ1, as detailed in Sections 7.3.1–7.3.5, our metrics permit comparing chatbot size using INT; conversation style along Moore and Arar’s taxonomy [ 33] using FLOW, PATH, FPATH and CL; chatbot outputs using WPO, CPO, VPOP, OPRE and SNT; expected user inputs using WPTP, VPTP and PPTP; and chatbot vocabulary size using ENT, LPE, SPL and WL. Moreover, since metrics are defined over Conga, they are applicable to different technologies and enable the comparison of heterogeneous chatbots. Regarding RQ1.1, as described in Section 7.3.6, we found that, for our dataset of chatbots, all metrics but PATH and VPTP follow different distributions for Dialogflow and Rasa.
|
||
ls-type:: annotation
|
||
hl-page:: 29
|
||
hl-color:: yellow
|
||
id:: 643579b1-5765-46ee-a5fb-1f76f2975cd4
|
||
hl-stamp:: 1681226391481
|
||
- The extreme cases reveal complex chatbots and may signal redundant intents
|
||
ls-type:: annotation
|
||
hl-page:: 30
|
||
hl-color:: yellow
|
||
id:: 64357a61-0924-486b-8f07-5a592295d6ad
|
||
- low or high metric outliers to assess if they have problems.
|
||
ls-type:: annotation
|
||
hl-page:: 33
|
||
hl-color:: yellow
|
||
id:: 64357c1f-9bef-4344-8c43-b2f99e5bba28
|
||
hl-stamp:: 1681226801614
|
||
- 3 clusters
|
||
ls-type:: annotation
|
||
hl-page:: 34
|
||
hl-color:: green
|
||
id:: 64357cd5-91cb-41e8-9e03-88dcc182ef9c
|
||
- Answering RQ3. Overall, we can answer RQ3 positively: vocabulary-based clustering can create meaningful groups of semantically related chatbots (e.g., groups of chatbots about hotels, restaurants, or weather, among others). To have a quantitative assessment, we measured the fraction of correctly clustered chatbot pairs using balanced accuracy, and compared against three baselines obtaining a substantially higher balanced accuracy (0.664 vs 0.5)
|
||
ls-type:: annotation
|
||
hl-page:: 36
|
||
hl-color:: purple
|
||
id:: 64357d16-08e9-4663-b27e-8b62599e9b81
|
||
- quality assurance during chatbot development, and so, we want our metrics to detect problems related to incomplete designs.
|
||
ls-type:: annotation
|
||
hl-page:: 37
|
||
hl-color:: green
|
||
id:: 64357d38-5ab0-4fd4-b9a8-5065a3021f3f
|
||
- Related to RQ2, while metrics can detect problems in chatbots, problems need to be confirmed by inspection
|
||
ls-type:: annotation
|
||
hl-page:: 37
|
||
hl-color:: purple
|
||
id:: 64357d4a-19c0-4004-9039-51a5c1d60a17
|
||
- Chatbots are increasingly relevant nowadays, so techniques for assessing, comparing and clustering chatbots before their deployment are required.
|
||
ls-type:: annotation
|
||
hl-page:: 37
|
||
hl-color:: green
|
||
id:: 64357d7c-870f-456a-b317-9258fdf1479d |