logseq/pages/hls__TOSEM-2023-0032_Proof_hi_1680262636436_0.md

file:: TOSEM-2023-0032_Proof_hi_1680262636436_0.pdf file-path:: ../assets/TOSEM-2023-0032_Proof_hi_1680262636436_0.pdf

• Measuring and Clustering Heterogeneous Chatbot Designs ls-type:: annotation hl-page:: 2 hl-color:: green id:: 6426c651-96af-4297-8073-88830f4ece78
• suite of metrics for chatbot designs, ls-type:: annotation hl-page:: 2 hl-color:: green id:: 6426c65b-52e5-4d0e-9bc9-83620aeb6a2c
• rouping chatbots along their conversation topics and design features ls-type:: annotation hl-page:: 2 hl-color:: green id:: 6426c663-4983-4bb7-b7c2-3daf1ac4a64e
• 59 Dialogflow and Rasa chatbots ls-type:: annotation hl-page:: 2 hl-color:: green id:: 6426c67d-4a4f-484e-960d-bc33397f0495
• improved ls-type:: annotation hl-page:: 2 hl-color:: green id:: 6426c7b4-903e-4762-8f4b-faa9f53c7563
• We have refined our suite of metrics and extended it with a new readability metric, expanding the explanations and rationale of the metrics. ls-type:: annotation hl-page:: 3 hl-color:: green id:: 64270a94-e163-4aa2-a155-66a778a97e27
• Francesco Basciani, Juri Di Rocco, Davide Di Ruscio, Ludovico Iovino, and Alfonso Pierantonio. 2016. Automated Clustering of Metamodel Repositories. In Proc. 28th Int. Conf. on Advanced Information Syst. Eng. (LNCS, Vol. 9694). Springer, 342–358. ls-type:: annotation hl-page:: 38 hl-color:: green id:: 64270ae3-06c3-4eb7-b8c5-4955f5045e74
• ESTHER GUERRA ls-type:: annotation hl-page:: 4 hl-color:: green id:: 64270b12-3b66-4f3d-9c79-4b90a54a5a31
• Measuring and Clustering Heterogeneous Chatbot Designs ls-type:: annotation hl-page:: 4 hl-color:: green id:: 64270b16-aa4c-44a4-be32-1aa114a999b7
• JUAN DE LARA ls-type:: annotation hl-page:: 4 hl-color:: green id:: 64270b1f-e304-4d66-af8c-bdd9b073ee61
• there are hardly any mechanisms to compare and cluster chatbots developed with heterogeneous technologies ls-type:: annotation hl-page:: 4 hl-color:: yellow id:: 64270c19-5357-4c40-a759-2ce6d8e898ee hl-stamp:: 1680280620522
• 21 metrics for chatbot designs ls-type:: annotation hl-page:: 4 hl-color:: green id:: 64270c5e-847d-46e8-b212-52d7e9c52f54 hl-stamp:: 1680280672632
• 259 Dialogflow and Rasa chatbots ls-type:: annotation hl-page:: 4 hl-color:: green id:: 64270cdb-7b9c-4d56-95e1-581469801ba5
• early detection of quality issues ls-type:: annotation hl-page:: 4 hl-color:: yellow id:: 64270d0d-3cb8-46db-ba98-f85e8ccdb270 hl-stamp:: 1680280847796
• chatbot comprehension, search and comparison ls-type:: annotation hl-page:: 4 hl-color:: green id:: 64270d3e-3609-4143-acb2-2bf21a05aa9c hl-stamp:: 1680280898517
• Chatbots are also used to assist workers in different domains, like software engineering ls-type:: annotation hl-page:: 5 hl-color:: green id:: 64270e27-b4aa-4ec8-b175-babbfd85b037
• artner estimates that there are more than 2000 natural language technology providers, with a significant number of them offering facilities to create chatbots ls-type:: annotation hl-page:: 5 hl-color:: green id:: 64270e35-49a7-407f-a40e-b02580704e6b
• tools for testing chatbots ls-type:: annotation hl-page:: 5 hl-color:: green id:: 64270e85-54c0-4226-9008-ae0331127bab
• guide and control the quality of chatbots throughout their development and maintenance, becoming a complement to testing ls-type:: annotation hl-page:: 5 hl-color:: green id:: 64270ec1-fa6a-4192-9b1e-b928d0e536a2
• there are hardly any metric proposals for chatbot designs ls-type:: annotation hl-page:: 5 hl-color:: purple id:: 643071f5-9869-46bb-8b0f-aabd7c516732
• detect potential problems related to user experience (e.g., complex conversation flows, hard-to-read chatbot answers); as indicators of chatbot complexity; to compare properties of heterogeneous chatbots; to discover commonalities and cluster similar chatbots; and to understand how different implementation platforms can impact on the chatbot design ls-type:: annotation hl-page:: 5 hl-color:: purple id:: 64307221-6d9a-47af-82e5-54a16fa70021
• This calls for mechanisms to measure the similarity or dissimilarity between chatbots, even if developed with different technologies, to understand their commonalities and enable searching for chatbots akin to a given one, e.g., as a first step towards the reuse of existing chatbots ls-type:: annotation hl-page:: 5 hl-color:: yellow id:: 643072ad-0c81-45e5-b15e-949da1825196
• we propose a suite of 21 static metrics for chatbot designs, and two methods for clustering chatbots ls-type:: annotation hl-page:: 5 hl-color:: purple id:: 6433ba23-b920-474f-ad4d-406cd9fa63da
• This is useful, e.g., to identify chatbots on the same topic for their reuse; to facilitate the construction of recommender systems for chatbots; to organise large chatbot repositories according to their domain; or to facilitate chatbot search. ls-type:: annotation hl-page:: 5 hl-color:: purple id:: 6433ba51-985a-4318-a639-f0a7ab14412f
• metrics and the clustering methods over Dialogflow and Rasa chatbots ls-type:: annotation hl-page:: 6 hl-color:: green id:: 6433bb84-0020-4fe7-99d8-cd95d5d5bc53
• 20 chatbot design metrics, proposed their technology-independent definition atop Conga, made their implementation available via an API, and evaluated their suitability on 12 chatbots. ls-type:: annotation hl-page:: 6 hl-color:: green id:: 6433bc70-d72e-415b-977c-bc7ab83d864a
• principal component analysis. ls-type:: annotation hl-page:: 6 hl-color:: yellow id:: 6433bc90-eb00-4b86-a2a7-282da8bdc294
• we are interested in task-oriented chatbots ls-type:: annotation hl-page:: 6 hl-color:: green id:: 6433bd3b-1286-4416-8023-8d476c77c813
• [:span] ls-type:: annotation hl-page:: 7 hl-color:: green id:: 6433bd5f-1c82-4f9a-94f3-a1f31e280fdc hl-type:: area hl-stamp:: 1681112412961
• ystemcentric ls-type:: annotation hl-page:: 7 hl-color:: green id:: 6433be90-d066-4f2a-a9a0-14a21683fe72
• Content-centric ls-type:: annotation hl-page:: 7 hl-color:: green id:: 6433be97-64ff-4dfe-ab62-25f0cb8cbd80
• Visualcentric ls-type:: annotation hl-page:: 7 hl-color:: green id:: 6433be9f-6267-42bf-8aa4-4d0e7c7604a6
• conversation-centric ls-type:: annotation hl-page:: 7 hl-color:: green id:: 6433bea2-ea0d-4b36-bcee-0c68120b001e
• conversation patterns [35] and conversation design principles ls-type:: annotation hl-page:: 8 hl-color:: green id:: 6433bef4-67d3-4e53-b4d2-7a9af9d85ab5
• dynamic execution of the chatbot ls-type:: annotation hl-page:: 8 hl-color:: yellow id:: 6433bf2a-89ee-467a-80a4-3d55bc1628ff
• A means to facilitate finding artefacts of interest (chatbots in our case) is to organise them into meaningful group ls-type:: annotation hl-page:: 8 hl-color:: green id:: 6433c0e8-418f-4a75-9fc1-d0967df26e44
• reusing existing artefacts in a cluster of interest ls-type:: annotation hl-page:: 8 hl-color:: green id:: 6433c0ef-d481-420e-a21a-1f3432ed1dfc
• K-means and hierarchical clustering require setting the number of clusters as hyperparameters, while DBSCAN has other parameters like the neighbourhood distance and the minimum number of points required to form a dense region ls-type:: annotation hl-page:: 8 hl-color:: green id:: 6433c14d-36ef-4daf-a8f0-49d9e92b1a60
• In our proposal to chatbot clustering, we use similar encodings and techniques, but focus on chatbot designs instead of meta-models ls-type:: annotation hl-page:: 9 hl-color:: purple id:: 6433c174-0930-4fac-a5f4-8b96565043c8
• clustering for chatbot designs has not been studied yet in the literature ls-type:: annotation hl-page:: 9 hl-color:: purple id:: 6433c18f-fbe8-411f-85fc-ea07febb53ce
• Section 4.1 starts by introducing the chatbot design notation ls-type:: annotation hl-page:: 9 hl-color:: purple id:: 6433c1ae-2826-4eee-b30d-ea8685bfb55c
• Section 4.2 details our proposed metric ls-type:: annotation hl-page:: 9 hl-color:: purple id:: 6433c1b5-9ee7-4f3c-9e18-97b25c1c4943
• Conga ls-type:: annotation hl-page:: 9 hl-color:: purple id:: 6433c1df-9e2f-451e-a94a-9ef997a9aebe
• [:span] ls-type:: annotation hl-page:: 10 hl-color:: green id:: 6433c204-1b23-4b7d-9074-fd6f98b13553 hl-type:: area hl-stamp:: 1681113602953
• Intents ls-type:: annotation hl-page:: 9 hl-color:: green id:: 6433c22a-309c-49d9-b724-9fb588e852bf
• Entities ls-type:: annotation hl-page:: 9 hl-color:: green id:: 6433c22d-c52c-46f7-81db-17b8f1176a99
• t Actions ls-type:: annotation hl-page:: 9 hl-color:: green id:: 6433c233-5eb0-4e84-851d-023d0744ae3a
• Flow ls-type:: annotation hl-page:: 9 hl-color:: green id:: 6433c235-d772-47d5-8225-7c15d957b235
• For example, a chatbot can declare a simple entity for drink sizes with literals small, medium and large in English, and additionally define synonyms regular for medium and big for large. ls-type:: annotation hl-page:: 9 hl-color:: green id:: 6433c2c2-9143-4196-9872-e8190587bc6e
• [:span] ls-type:: annotation hl-page:: 11 hl-color:: green id:: 6433c311-1f72-4b52-9c0b-b7bde69046dc hl-type:: area hl-stamp:: 1681113870406
• Global metrics. We start explaining global metrics. These measure the number of intents(INT), entities (ENT) and conversation flows (FLOW, PATH), and also include understanding and user experience metrics (CNF, SNT) ls-type:: annotation hl-page:: 11 hl-color:: green id:: 6433c350-559c-4c7e-a70a-21105ef239a9
• PATH measures the number of paths a conversation may take, which is an indicator of conversation complexity ls-type:: annotation hl-page:: 11 hl-color:: green id:: 6433c399-7b88-4bdf-98a6-fac791bb6df9
• The combined use of FLOW and PATH can help detecting deviations of some design principles ls-type:: annotation hl-page:: 11 hl-color:: green id:: 6433c409-b375-49ff-bc86-3021d87424a9
• ntent metrics measure quality properties of each intent with respect to the expected user utterances and the bot output phrases ls-type:: annotation hl-page:: 12 hl-color:: green id:: 6433c49c-3c5b-441a-a622-b2d1032cef34
• interaction complexity, ls-type:: annotation hl-page:: 12 hl-color:: green id:: 6433c4c9-9554-44c9-8d26-e7185c6ee7d5 hl-stamp:: 1681114321426
• user cognitive load and speed up the completion of the intended tas ls-type:: annotation hl-page:: 13 hl-color:: green id:: 6433c5e6-33cc-4b9f-bca3-b0d2de20c997
• chatbot vocabulary ls-type:: annotation hl-page:: 13 hl-color:: green id:: 6433d39e-4d96-447e-af37-700174196584
• internal quality features (e.g., complex/simple conversations, large/succinct outputs). The latter groups chatbots by conversation top ls-type:: annotation hl-page:: 13 hl-color:: yellow id:: 6433d3a9-c5be-479e-b8c0-a055bf88ffb2
• metric INT (i.e., number of intents) would create groups of chatbots with similar size complexit ls-type:: annotation hl-page:: 13 hl-color:: green id:: 6433d3f9-5203-47b0-950d-a59c625eaebe
• [:span] ls-type:: annotation hl-page:: 14 hl-color:: green id:: 6433d40e-0f51-4f8e-85aa-6822d3061732 hl-type:: area hl-stamp:: 1681118220789
• principal component analysis (PCA) o ls-type:: annotation hl-page:: 14 hl-color:: green id:: 6433d722-2ee5-40ec-a9d6-94714b27e880
• he latter is possible since PCA permits explaining how each metric contributes to each component, by providing the loading factor of the metric for the component. This effectively groups the metrics that contribute the most to each component. ls-type:: annotation hl-page:: 14 hl-color:: green id:: 6433d739-6293-4cc8-a657-23c4145a4490
• PATH, INT, FLOW, WL, PPTP and FPATH. ls-type:: annotation hl-page:: 14 hl-color:: yellow id:: 6433d749-9f82-4aaa-856d-3492003fd1b8 hl-stamp:: 1681119074022
• Vocabulary-based clustering ls-type:: annotation hl-page:: 15 hl-color:: green id:: 64348546-40a4-4e18-9283-55d312c65b9f
• we provide another clustering method based on the chatbot conversation topics ls-type:: annotation hl-page:: 15 hl-color:: purple id:: 64348554-62cf-489c-838f-d80ab3ccaf72
• training phrases of the intents, the chatbot output phrases, and the literals and synonyms in entities. ls-type:: annotation hl-page:: 15 hl-color:: green id:: 64348598-0791-4c13-b203-55fe7b82ec0b
• Cluster projection into three dimensions, ls-type:: annotation hl-page:: 14 hl-color:: yellow id:: 643567d9-76d1-4749-b66d-0eb46ea8c2b4
• In the graphic, each dot represents a chatbot, and the shape of the dot identifies the cluster where the chatbot belongs. ls-type:: annotation hl-page:: 14 hl-color:: yellow id:: 6435689b-3e6c-4371-9a76-13a96f421f89
• Latent Semantic Analysis ls-type:: annotation hl-page:: 15 hl-color:: purple id:: 64356a14-052b-48c8-b7e6-c804d35f246e
• set of chatbots ls-type:: annotation hl-page:: 15 hl-color:: yellow id:: 64356a38-076f-4fa8-b085-ea63f91f0ca9
• relevance of a word to a chatbot, ls-type:: annotation hl-page:: 16 hl-color:: green id:: 64356ab2-9bc1-4ee9-807d-2a6cb170f649
• TF-IDF weight of a word for a chatbot ls-type:: annotation hl-page:: 16 hl-color:: green id:: 64356b1b-d19d-4f5b-acc7-8fa7e4995558
• Hence, the weights tend to filter out common terms. ls-type:: annotation hl-page:: 16 hl-color:: purple id:: 64356b2c-d02c-46f9-b646-3d6f913092eb
• osine similarit ls-type:: annotation hl-page:: 16 hl-color:: green id:: 64356b47-da7b-44f5-a50c-bc86f1d4ce01
• agglomerative clustering ls-type:: annotation hl-page:: 16 hl-color:: green id:: 64356b61-884f-4f24-a58c-f986e290c0cf
• Clustered similarity matrix of chatbots. ls-type:: annotation hl-page:: 18 hl-color:: green id:: 64356bec-b05f-4ec3-8a8e-b61e511d7d88
• Chatbot working schema ls-type:: annotation hl-page:: 7 hl-color:: yellow id:: 64356f75-f0e5-4767-a91d-fa6ca0f3aa68
• across all chatbots in the repository. ls-type:: annotation hl-page:: 18 hl-color:: green id:: 643570a6-f407-4bd3-aeb0-a45c198f270e
• clustering chatbots ls-type:: annotation hl-page:: 18 hl-color:: green id:: 643570b9-1a4b-41f3-8d44-90156ad6c9ff
• his core has an extensible design, which makes it easy to add new types of metrics, clustering criteria and chatbot technologie ls-type:: annotation hl-page:: 19 hl-color:: green id:: 643571d5-ad05-41a6-b3a5-82b342c89b9b
• This way, Asymob computes the metrics on Conga models, independently of any chatbot implementation platform. ls-type:: annotation hl-page:: 19 hl-color:: green id:: 6435738e-3ebc-4e8c-b966-6562e955bf50
• Asymob supports some third-party technologies to simplify the implementation of new metrics ls-type:: annotation hl-page:: 19 hl-color:: green id:: 643573a6-12d4-4dec-885e-9883de5c1c6c
• Dialogflow and Rasa) into Conga. ls-type:: annotation hl-page:: 20 hl-color:: green id:: 643573c8-c6f3-4642-98dd-99927d75eb44
• Agent.json describes global chatbot features, like its name, definition languages, or connection data to external services (the webhook). ls-type:: annotation hl-page:: 20 hl-color:: green id:: 6435746d-1727-46d2-99a9-591ec23c2e19
• Intents ls-type:: annotation hl-page:: 20 hl-color:: purple id:: 64357482-5893-4b2a-a454-36f065ea7160 hl-stamp:: 1681224841238
• usefulness of our metrics ls-type:: annotation hl-page:: 23 hl-color:: green id:: 64357500-0e59-4814-a243-eb231c28aa52
• efficacy of our vocabulary-based clustering ls-type:: annotation hl-page:: 23 hl-color:: green id:: 64357505-9559-477a-844a-1424ed2318c3
• chatbots in the wild compare with respect to their size, conversation style, outputs, expected inputs and vocabulary? ls-type:: annotation hl-page:: 23 hl-color:: yellow id:: 643575c8-5044-40fe-b3ad-830d3a8b896e
• quality issues in real chatbots ls-type:: annotation hl-page:: 23 hl-color:: yellow id:: 6435769c-5592-45d1-883f-8919ffb5423e
• meaningful groups of semantically related chatbots? ls-type:: annotation hl-page:: 24 hl-color:: yellow id:: 64357724-3fbd-4e25-a199-67b40ec7d292
• we compared different dimensions of the chatbots (size, conversation style, responses, expected user utterances and vocabulary) based on the chatbots’ metric values ls-type:: annotation hl-page:: 24 hl-color:: green id:: 6435778a-fe77-422a-a711-10d5b2f8ff15
• We do not expect every discordant metric value to signal a real problem, but instead, we are interested in assessing the usefulness of metrics as an inexpensive chatbot quality assurance mechanism (e.g., compared to testing) that can be used early in the development process ls-type:: annotation hl-page:: 24 hl-color:: purple id:: 643577ad-fb25-4a39-a817-ece0ecbc6e48 hl-stamp:: 1681225647635
• size ls-type:: annotation hl-page:: 25 hl-color:: green id:: 643578f1-018b-461d-b6af-828dac934a0f
• conversation style ls-type:: annotation hl-page:: 25 hl-color:: green id:: 643578f6-95b3-4109-8a69-26b255baa946
• outputs ls-type:: annotation hl-page:: 25 hl-color:: green id:: 643578f8-f070-4e8d-aa8c-2b802ebdf903
• expected inputs, vocabulary (pertinent for RQ1) and implementation platform (RQ1.1) ls-type:: annotation hl-page:: 25 hl-color:: green id:: 64357900-5693-45d4-9ba5-e6c62c6342c1
• Naturally, chatbots with more intents tend to have more flows, and therefore, more paths. ls-type:: annotation hl-page:: 26 hl-color:: purple id:: 64357937-6d96-4dbd-9fe7-9e6c1be0cf5e
• 7.3.7 Answering RQ1 and RQ1.1. With respect to RQ1, as detailed in Sections 7.3.1–7.3.5, our metrics permit comparing chatbot size using INT; conversation style along Moore and Arar’s taxonomy [ 33] using FLOW, PATH, FPATH and CL; chatbot outputs using WPO, CPO, VPOP, OPRE and SNT; expected user inputs using WPTP, VPTP and PPTP; and chatbot vocabulary size using ENT, LPE, SPL and WL. Moreover, since metrics are defined over Conga, they are applicable to different technologies and enable the comparison of heterogeneous chatbots. Regarding RQ1.1, as described in Section 7.3.6, we found that, for our dataset of chatbots, all metrics but PATH and VPTP follow different distributions for Dialogflow and Rasa. ls-type:: annotation hl-page:: 29 hl-color:: yellow id:: 643579b1-5765-46ee-a5fb-1f76f2975cd4 hl-stamp:: 1681226391481
• The extreme cases reveal complex chatbots and may signal redundant intents ls-type:: annotation hl-page:: 30 hl-color:: yellow id:: 64357a61-0924-486b-8f07-5a592295d6ad
• low or high metric outliers to assess if they have problems. ls-type:: annotation hl-page:: 33 hl-color:: yellow id:: 64357c1f-9bef-4344-8c43-b2f99e5bba28 hl-stamp:: 1681226801614
• 3 clusters ls-type:: annotation hl-page:: 34 hl-color:: green id:: 64357cd5-91cb-41e8-9e03-88dcc182ef9c
• Answering RQ3. Overall, we can answer RQ3 positively: vocabulary-based clustering can create meaningful groups of semantically related chatbots (e.g., groups of chatbots about hotels, restaurants, or weather, among others). To have a quantitative assessment, we measured the fraction of correctly clustered chatbot pairs using balanced accuracy, and compared against three baselines obtaining a substantially higher balanced accuracy (0.664 vs 0.5) ls-type:: annotation hl-page:: 36 hl-color:: purple id:: 64357d16-08e9-4663-b27e-8b62599e9b81
• quality assurance during chatbot development, and so, we want our metrics to detect problems related to incomplete designs. ls-type:: annotation hl-page:: 37 hl-color:: green id:: 64357d38-5ab0-4fd4-b9a8-5065a3021f3f
• Related to RQ2, while metrics can detect problems in chatbots, problems need to be confirmed by inspection ls-type:: annotation hl-page:: 37 hl-color:: purple id:: 64357d4a-19c0-4004-9039-51a5c1d60a17
• Chatbots are increasingly relevant nowadays, so techniques for assessing, comparing and clustering chatbots before their deployment are required. ls-type:: annotation hl-page:: 37 hl-color:: green id:: 64357d7c-870f-456a-b317-9258fdf1479d