logseq/pages/hls__icse2023-paper1528_1666905345862_0.md

file:: icse2023-paper1528_1666905345862_0.pdf file-path:: ../assets/icse2023-paper1528_1666905345862_0.pdf

• quickly compare pipeline variants by allowing them to swap out code blocks ls-type:: annotation hl-page:: 1 hl-color:: green id:: 636290e3-ab3a-4ea5-974d-a2dc85d1ab00
• AI-driven natural language interface ls-type:: annotation hl-page:: 1 hl-color:: green id:: 63629139-1857-4eb4-afc7-c8a878447534
• CodeT5 model ls-type:: annotation hl-page:: 1 hl-color:: green id:: 6362914f-491a-40a2-8c5c-a2c11ad70422
• generate partial operator invocations depending on available information ls-type:: annotation hl-page:: 1 hl-color:: green id:: 636291e4-d596-45d4-a4d3-05a8db2d1bc5
• Discoverability ls-type:: annotation hl-page:: 1 hl-color:: green id:: 636291fa-bfc2-431a-aafc-2c66172198bb
• terative Composition ls-type:: annotation hl-page:: 1 hl-color:: green id:: 63629201-caf6-4a16-bcfc-43d2b8167eb4
• Overall, our work demonstrates the promise of combining both an AIpowered natural language interface and a visual interface for helping developers create AI pipelines without writing code. ls-type:: annotation hl-page:: 1 hl-color:: green id:: 63629212-d7c2-4f82-afb9-e5daf637cd00
• citizen developers are people without formal training or experience as AI software developers. Furthermore, these libraries are often large. Needing to remember hundreds of AI operators and their hyper-parameters slows down even professional developers. ls-type:: annotation hl-page:: 1 hl-color:: yellow id:: 63629241-cab5-4ad3-b3bf-f84db2144e87
• to democratize software development and increase productivity. ls-type:: annotation hl-page:: 1 hl-color:: green id:: 636292be-8716-408e-af74-5bcb4aa13209
• discoverability ls-type:: annotation hl-page:: 1 hl-color:: green id:: 636292ff-cf62-41d3-beff-10d77e2d8db8
• LOWCODER to overcome this problem by combining visual programming with PBNL ls-type:: annotation hl-page:: 1 hl-color:: green id:: 63629306-a399-4dc8-ae3a-eb4c9c01adc9
• programming by natural language (PBNL) ls-type:: annotation hl-page:: 1 hl-color:: purple id:: 6362930f-5a5c-49ac-a0d6-39e256d55fc4
• This does not always return correct programs, necessitating a way to help users understand and fix generated programs. ls-type:: annotation hl-page:: 1 hl-color:: purple id:: 636293b6-f2ce-483b-b953-9a1b46a4bab7
• The visual programming component of LOWCODER, LOWCODERVP, lets users snap together visual blocks for AI operators into well-structured AI pipelines ls-type:: annotation hl-page:: 1 hl-color:: purple id:: 636293c2-d030-4841-b388-a1f86642abf9
• We further noticed that it is common in this domain for queries to mention at most a subset of hyper-parameters for each pipeline step, so we developed a novel task formulation tailored to this use case that improved learning outcomes ls-type:: annotation hl-page:: 1 hl-color:: yellow id:: 63629405-9cd5-4e1c-a87c-5f3f78b4fb53
• showing the effect of pipelines on data ls-type:: annotation hl-page:: 1 hl-color:: purple id:: 63629415-ad85-448b-b83f-7b213aae4cdd
• We find that LOWCODERNL helped users discover previouslyunknown operators more easily than other methods, such as web search ls-type:: annotation hl-page:: 1 hl-color:: purple id:: 63629431-4ebc-47a2-9775-cd3301a0ae32
• e explore AI models for PBNL and integrate those models with the visual tool. ls-type:: annotation hl-page:: 2 hl-color:: green id:: 6362944a-22e9-43dd-b948-99f59646234a
• ow-Code for AI: ls-type:: annotation hl-page:: 1 hl-color:: blue id:: 6362944e-fde5-4af4-acfa-e211049f2f0a
• AI for Low-Code: ls-type:: annotation hl-page:: 2 hl-color:: blue id:: 63629452-f8ad-4ece-aecb-7f3b671b2db7
• User Study: ls-type:: annotation hl-page:: 2 hl-color:: blue id:: 63629455-c670-4bb5-ba5d-ec1b768bb43c
• G. Baudart, M. Hirzel, K. Kate, P. Ram, A. Shinnar, and J. Tsay, “Pipeline combinators for gradual AutoML,” in Advances in Neural Information Processing Systems (NeurIPS), Dec.2021. [Online]. Available: https://proceedings.neurips.cc/paper/2021/ file/a3b36cb25e2e0b93b5f334ffb4e4064e-Paper.pd ls-type:: annotation hl-page:: 11 hl-color:: green id:: 63629484-ce26-44f9-ac7e-4732196bff4e
• AI for Low-code for AI ls-type:: annotation hl-page:: 2 hl-color:: blue id:: 6362948f-8546-4d5e-9a46-c9765f7ed69a
• AI for Low-code for AI ls-type:: annotation hl-page:: 2 hl-color:: blue id:: 6362949c-6600-4908-8742-c277680df426
• we add a natural language interface to help users discover and configure operators. ls-type:: annotation hl-page:: 2 hl-color:: purple id:: 636294b4-06c4-4dee-b69e-c66d60ee1c5b
• AI for Low-code for AI: ls-type:: annotation hl-page:: 2 hl-color:: blue id:: 636294d0-40b5-4642-8a2c-e20ff58d940e
• Combining low-code techniques ls-type:: annotation hl-page:: 2 hl-color:: blue id:: 636294ef-3e7c-497f-8748-6f69ff516113
• CodeT5 ls-type:: annotation hl-page:: 2 hl-color:: green id:: 63629516-298f-490b-b17f-029c15cd5793
• PBNL as a write-only view, ls-type:: annotation hl-page:: 2 hl-color:: green id:: 63629784-e4b1-4626-a1ba-e14902883f04
• Python Flask back-end serve ls-type:: annotation hl-page:: 2 hl-color:: green id:: 636297a1-b7bd-4f95-af19-d289485255db
• front-end based on Blockl ls-type:: annotation hl-page:: 2 hl-color:: green id:: 636297a5-5617-4a22-a339-53a3b3a42fa2
• LOWCODER supports 143 sklearn operators. Sklearn distinguishes between operators, which are steps of machine learning pipelines and usually have learned coefficients, and functions, which are used separately and cannot be a step in a pipeline. ls-type:: annotation hl-page:: 2 hl-color:: green id:: 636297d4-ccc1-4477-9616-39ef5093f309
• LOWCODER focuses on pipelines, it only has explicit blocks for operators but not functions ls-type:: annotation hl-page:: 3 hl-color:: purple id:: 636297eb-c4db-48f6-a996-ac1d9d3595f6
• Lale which is then sent to the back-end ls-type:: annotation hl-page:: 3 hl-color:: green id:: 636297f5-34e5-40a9-80e7-b8583012ed8b
• One goal that this tool shares with other block-based visual tools such as Scratch [9] is to encourage a more “tinkerable” experience. ls-type:: annotation hl-page:: 3 hl-color:: green id:: 6362983b-ff09-40ac-910c-c90b5bc4fd9d
• For ease of execution, our tool only allows for one valid pipeline at a time, so blocks must be attached downstream of the pre-defined Start block to be considered part of the active pipeline. ls-type:: annotation hl-page:: 3 hl-color:: green id:: 63629871-3f13-406c-912e-708d9f9b5f34
• hyper-parameter for an operator along with a description (when hovering over the hyper-parameter name) and default values along with input boxes to modify each hyper-parameter. ls-type:: annotation hl-page:: 3 hl-color:: yellow id:: 63629949-0e6c-4881-b329-df61e0399d17
• the palette of LOWCODERVP contains more than a hundred operator blocks ls-type:: annotation hl-page:: 3 hl-color:: green id:: 63629a5e-e217-4e86-bc85-474d40cf9eb4
• The tool then infers relevant operator(s) and any applicable hyper-parameters using an underlying natural language to code translation model and automatically adds the most relevant operator to the end of the pipeline. ls-type:: annotation hl-page:: 3 hl-color:: green id:: 63629ab7-08fc-4f04-86f8-d2df98fe699b
• PCA with 2 components” ls-type:: annotation hl-page:: 3 hl-color:: green id:: 63629ad1-2257-4739-8d7b-b5acd3bf37d8
• A potential weakness of natural language low-code tools is that the generated programs can be incorrect, due to a lack of clarity, or ambiguity, in the query, or a lack of context for the model providing inferences [16] ls-type:: annotation hl-page:: 3 hl-color:: green id:: 63629ae4-2308-488d-a9d0-9329d8af893b
• isual inputs and representations are unambiguous [25], requiring no probabilistic interpretation, so users can easily understand and manipulate the results returned by LOWCODERNL ls-type:: annotation hl-page:: 3 hl-color:: green id:: 63629aee-5414-4071-a9dd-587424b6e03e
• RandomForestClassifier ls-type:: annotation hl-page:: 3 hl-color:: blue id:: 63629b26-6075-450b-bc74-8a36514a0599
• Data Collection ls-type:: annotation hl-page:: 4 hl-color:: blue id:: 63629b68-a90e-4591-969d-63f740bc5ba0
• escribe what they want to do when they do not know how ls-type:: annotation hl-page:: 4 hl-color:: green id:: 6362d2a3-f95c-4b21-99c9-59368b0e4336
• sklearn pipelines in a low-code setting, using a natural language interface that can be used as an intelligent search tool ls-type:: annotation hl-page:: 4 hl-color:: purple id:: 6362d2ad-b334-481a-99cf-ecb14265e255
• proxy dataset using 140K Python Kaggle notebooks that were collected as part of the Google AI4Code challenge ls-type:: annotation hl-page:: 4 hl-color:: green id:: 6362d2d9-8a3a-40d3-b868-3b5470eaaca8
• B. Data Preprocessing ls-type:: annotation hl-page:: 4 hl-color:: blue id:: 6362d33e-c15d-4554-8104-de8a0ee885ce
• 211,916 aligned NL-code pairs ls-type:: annotation hl-page:: 4 hl-color:: green id:: 6362d35c-e6d2-4382-8f5c-c3da398d7d66
• We discard any code cells that do not include sklearn operation invocation statements but include other sklearn code leaving a final total of 79,372 NL-Code pairs. ls-type:: annotation hl-page:: 4 hl-color:: green id:: 6362d387-bcbf-4e72-a894-4695cb2cedd3
• 64,779 train samples ls-type:: annotation hl-page:: 4 hl-color:: green id:: 6362d393-43c8-46b7-8dfe-df57b26cd500
• 7,242 validation samples ls-type:: annotation hl-page:: 4 hl-color:: green id:: 6362d397-8731-40f7-a25e-2f0af5e19b68
• 7,351 test samples ls-type:: annotation hl-page:: 4 hl-color:: green id:: 6362d39c-6085-48ff-8590-6f5cf3213087
• Given the NL query, our model aims to generate a line of sklearn code corresponding to an operation invocation that can be used to build the next step of the pipeline. ls-type:: annotation hl-page:: 4 hl-color:: green id:: 6362d3e1-a371-499f-b3e9-d0307cd94fe4
• C. Tasks ls-type:: annotation hl-page:: 4 hl-color:: blue id:: 6362d3e5-10af-46de-9ac3-3fa262309cb2
• 1. Operator Name Generation: ls-type:: annotation hl-page:: 4 hl-color:: blue id:: 6362d457-02f9-4083-b161-60fe697265fd
• 2. Complete Operator Invocation Generation ls-type:: annotation hl-page:: 4 hl-color:: blue id:: 6362d470-f746-45ba-8239-a421b6bf92a1
• often predicts arbitrary hyperparameter values, resulting in lines of code that can rarely be used directly by developers. ls-type:: annotation hl-page:: 4 hl-color:: yellow id:: 6362d48b-528b-4b11-9553-ad2b24829f57
• 3. Masked Operator Invocation Generation ls-type:: annotation hl-page:: 4 hl-color:: blue id:: 6362d619-8a7f-4fb1-a437-a026634b1dca
• ensure that the model learns to predict the specific invocation signature, even if it is unaware of the values to provide for the hyper-parameters. ls-type:: annotation hl-page:: 4 hl-color:: green id:: 6362d62e-e3d0-4cc8-b989-92806337015e
• 4. Hybrid Operator Invocation Generation (HOI) ls-type:: annotation hl-page:: 4 hl-color:: blue id:: 6362d638-115e-4aec-aea5-16eb3099995d
• This gives the model an opportunity to learn the hyper-parameter names and values if they are explicitly stated in the description, ls-type:: annotation hl-page:: 4 hl-color:: blue id:: 6362d8c6-342c-40f6-a399-54287972425c
• equence-tosequence model ls-type:: annotation hl-page:: 4 hl-color:: green id:: 6362d8d1-9473-482a-9996-49758c21316e
• 1. Transformer (from scratch): ls-type:: annotation hl-page:: 5 hl-color:: blue id:: 6362d93e-b55a-4797-9151-210584316923
• trifecta ls-type:: annotation hl-page:: 5 hl-color:: green id:: 6362d95f-7e8e-45fe-a7b4-1dbb4656a48b
• medium sized models (100M-1B parameters) are pretrained with a generic training signal and then fine-tuned on task data ls-type:: annotation hl-page:: 5 hl-color:: green id:: 6362d973-97c8-489e-b527-03d06773be58
• arge models(>1B parameters) are only pretrained on very large datasets and are prompted with examples from the training data as demonstration followed by the query ls-type:: annotation hl-page:: 5 hl-color:: green id:: 6362d97d-124b-4fcd-a62b-93062c4e98ce
• 2. Fine-tuning CodeT5 ls-type:: annotation hl-page:: 5 hl-color:: blue id:: 6362d98e-5800-4f3d-987a-1c641a5953ac
• We fine-tune CodeT5 on the HOI generation task by adding the ‘Generate Python’ prefix to all NL queries. ls-type:: annotation hl-page:: 5 hl-color:: green id:: 6362d9ba-9322-49b4-964e-0394fb429605
• 3. Few-Shot Learning With CodeGen ls-type:: annotation hl-page:: 5 hl-color:: blue id:: 6362d9c2-b8f1-41bd-939f-97d10f94b925
• Models at this scale are expensive to fine-tune and are instead commonly used for inference by means of “few-shot prompting” ls-type:: annotation hl-page:: 5 hl-color:: green id:: 6362d9ea-c5cd-48de-a0d4-f768837d82bc
• 48 GB NVIDIA Quadro RTX 8000 GPUs ls-type:: annotation hl-page:: 6 hl-color:: green id:: 6362da2d-5d45-466d-90fe-a968b9ac7ecf
• ection 1 of the supplementary material ls-type:: annotation hl-page:: 4 hl-color:: yellow id:: 636435bc-49c5-4fe5-b420-49598f00fb93
• Section IV-D1) ls-type:: annotation hl-page:: 4 hl-color:: red id:: 63658ea4-c041-4794-a0b9-a9052f8073d2
• such models heavily rely on data to learn such behaviour and would need to be trained on an aligned dataset of natural language queries and the corresponding sklearn line(s) of code demonstrating how a user would want to use such an intelligent search too ls-type:: annotation hl-page:: 4 hl-color:: green id:: 63666434-5bdf-4402-829c-46fb6cdee121
• V-A5 ls-type:: annotation hl-page:: 4 hl-color:: yellow id:: 63666457-cadf-413b-ac9f-e1d054cabfc2
• We further filter out notebooks with non-English descriptions in all of the markdown cells, resulting in 59,569 notebooks ls-type:: annotation hl-page:: 4 hl-color:: green id:: 63666481-c796-4e5a-9c50-df4976b73b23
• ne(s) of code corresponding to an sklearn operation invocation statement ls-type:: annotation hl-page:: 4 hl-color:: green id:: 6366651a-0597-44c5-9367-72d2341c29f4
• train/validation/test split ls-type:: annotation hl-page:: 4 hl-color:: yellow id:: 63666544-d8bf-45ec-ad10-7b06d50d8ac2
• . A common alternative is to pretrain a larger model on a large dataset based on a generic task, such as masked token infilling, and/or a combination of objectives. Such models can then be fine-tuned (calibrated) on a smaller dataset to high accuracy. For the second paradigm, we fine-tune one such“medium” sized model, CodeT5 [10]. Finally, Large Language Models (LLMs), with billions of parameters, are often able to generalize to new tasks from just a few examples. These models are typically expensive to train from scratch or fine-tune; we adopt such a model, named CodeGen [11], and query it by means of few-shot prompting. We elaborate on these models below. Note that we use top-k sampling for our top-5 results.(A comparison of results with other decoding strategies can be found in Section 3 and 4 in the supplementary material ls-type:: annotation hl-page:: 5 hl-color:: yellow id:: 63666896-e137-411e-9645-854a1139dba2
• ) Experimental Setup ls-type:: annotation hl-page:: 6 hl-color:: blue id:: 6366734a-6404-4e70-a368-0c8e8c944faf
• Test Datasets ls-type:: annotation hl-page:: 6 hl-color:: blue id:: 6366736e-4753-441c-8b27-19e57585a47b
• We log all the NL queries that users searched for in LOWCODER during the user studie ls-type:: annotation hl-page:: 6 hl-color:: yellow id:: 636673a0-4717-4898-b049-1c7ce2b2aad3
• We evaluate the models’ ability to generate just the operator name as well as the entire operator invocation (including all the hyper-parameter names and values) based on the hybrid formulation ls-type:: annotation hl-page:: 6 hl-color:: green id:: 636674a8-8026-4029-ac03-c91e187d032b
• Test Metrics ls-type:: annotation hl-page:: 6 hl-color:: blue id:: 636674ab-08ea-4b85-9608-982d03f1c5c8
• ask Comparison: ls-type:: annotation hl-page:: 6 hl-color:: blue id:: 636674af-203b-460a-a3f7-e5d52559b846
• e compare the model’s ability to correctly generate the operator name and the operator invocation based on the formulation corresponding to the training task using top-1 and top-5 accuracy as shown in Figure 4 ls-type:: annotation hl-page:: 6 hl-color:: green id:: 63667575-2986-470b-9e8d-f9f5d8219f6d
• Model Comparison ls-type:: annotation hl-page:: 6 hl-color:: blue id:: 636675cd-7391-4915-ab87-d5cf1a842fdc
• (See Section 4 in the supplementary material for additional results and ablation studies.) ls-type:: annotation hl-page:: 6 hl-color:: yellow id:: 636675f4-f71b-47eb-b382-c362848f5384
• Performance in Practice ls-type:: annotation hl-page:: 7 hl-color:: blue id:: 63667694-e6d3-40f8-aa41-db308a4224f2
• The distribution of NL queries collected from the user studies represents the “true” distribution of queries that can be expected from users in a low-code settin ls-type:: annotation hl-page:: 7 hl-color:: green id:: 63667746-e018-466e-98a7-4bf021715343
• operation name ls-type:: annotation hl-page:: 7 hl-color:: green id:: 63667791-f92d-437e-9a0b-ffe8fcb38f59
• 141 out of185 queries correctly for an overall accuracy of 76.2%. ls-type:: annotation hl-page:: 7 hl-color:: green id:: 636677bd-3a9c-4579-877e-5c374c3e8cbd
• B. User Study ls-type:: annotation hl-page:: 7 hl-color:: blue id:: 636677ef-19c0-441e-843e-1a4d7a6d37be
• 20 participants using two versions of our too ls-type:: annotation hl-page:: 7 hl-color:: yellow id:: 636677ff-5f77-4431-9dfe-66d3478f430a
• RQ1 ls-type:: annotation hl-page:: 7 hl-color:: green id:: 63667849-2d45-45f5-842f-7e7769ea2397
• RQ2 ls-type:: annotation hl-page:: 7 hl-color:: green id:: 6366784b-be60-4056-babd-97bb73aca835
• RQ3 ls-type:: annotation hl-page:: 7 hl-color:: green id:: 6366784e-035f-4da9-8e05-2d31253327a4
• How do LOWCODERNL and other features help participants discover previously-unknown operators? ls-type:: annotation hl-page:: 7 hl-color:: green id:: 63667a77-288c-4cb2-9a30-2c1862f05d06
• Are participants able to compose and then iteratively refine AI pipelines in our tool ls-type:: annotation hl-page:: 7 hl-color:: green id:: 63667a7e-cbd5-4500-abf0-52fbcf577fe0
• What are the benefits and challenges of using low-code for AI? ls-type:: annotation hl-page:: 7 hl-color:: green id:: 63667a85-300e-4352-9a07-26460b44b62e
• 1. Study Methodology ls-type:: annotation hl-page:: 7 hl-color:: blue id:: 63667a96-466e-42ae-9a85-667e2b505350
• using LOWCODER with (NL condition) and without (keyword condition) the natural language (NL) interface powered by LOWCODERNL ls-type:: annotation hl-page:: 7 hl-color:: green id:: 63667b06-2100-4ee5-abc4-cccf5c529210
• participants were instructed to create AI pipelines with data preprocessing and classifier steps on a sample dataset with as high a score (accuracy on the test set) as possible during a time period of five to ten minutes ls-type:: annotation hl-page:: 7 hl-color:: purple id:: 63667b4f-b600-417e-b0bb-e866b20fe5ec hl-stamp:: 1667660627064
• he order of the conditions and the order of the tasks were shuffled such that there is a uniform distribution of the order of conditions and tasks. ls-type:: annotation hl-page:: 7 hl-color:: green id:: 63667b83-d211-4881-9def-b046fdeac968
• o avoid operators that were potentially useful in user tasks, the overview used both a non-sklearn operator that was not available in the study versions of the tool as well as sklearn’s DummyClassifier that generates predictions without considering input features ls-type:: annotation hl-page:: 8 hl-color:: green id:: 63667ba6-6c35-4afd-babf-382f86709bb4
• 2. Data Collection and Analysis ls-type:: annotation hl-page:: 8 hl-color:: blue id:: 63667c1e-1c5c-411a-a26f-954429efe242
• previously-unknow ls-type:: annotation hl-page:: 8 hl-color:: green id:: 63667c39-6156-4d4d-a024-361575f79954
• a preprocessing operator block is added or swapped, ls-type:: annotation hl-page:: 8 hl-color:: green id:: 63667c88-0dce-4bc7-81c9-aef93c74fe32
• a classifier block is swapped ls-type:: annotation hl-page:: 8 hl-color:: green id:: 63667c8f-cd80-48aa-b25c-c450ea5496aa
• yper-parameters are tuned. ls-type:: annotation hl-page:: 8 hl-color:: green id:: 63667c96-cd01-4e5d-b1b6-c68cd55f606d
• Both sets of quantitative metrics are counted per task (80 tasks total for 20 participants). ls-type:: annotation hl-page:: 8 hl-color:: green id:: 63667ca4-4d19-4803-aaf5-b7f02b9c5a81
• qualitative data to answer RQ3. ls-type:: annotation hl-page:: 8 hl-color:: green id:: 63667cad-35ef-4f74-bce8-1df8c6e00fe8
• 3. Study Results ls-type:: annotation hl-page:: 8 hl-color:: blue id:: 63667cf0-81bf-474c-ba1b-25287034a930
• We note that the participants were not able to use the keyword search to discover unknown operators due to needing at least part of the exact name ls-type:: annotation hl-page:: 8 hl-color:: green id:: 63667efe-87d1-4966-bc82-cd5f425df4fe
• These results suggest that LOWCODERNL is particularly helpful in discovering previously-unknown operators for non-novices but novices face some challenges. We discuss these challenges in RQ3. ls-type:: annotation hl-page:: 8 hl-color:: purple id:: 63667f2c-0552-4c2a-bf68-1deb50881101 hl-stamp:: 1667661842337
• sks participants commplete and iterate on preprocessors, classifier, and hyper-parameters. ls-type:: annotation hl-page:: 9 hl-color:: green id:: 63667f66-5d48-460a-ae4c-515854f6d33c
• compose ls-type:: annotation hl-page:: 8 hl-color:: green id:: 63667fb0-a06a-47e0-b960-12a7d47460c7
• iter ls-type:: annotation hl-page:: 8 hl-color:: green id:: 63667fb2-10aa-4d29-a6d2-817672207732
• atively ls-type:: annotation hl-page:: 8 hl-color:: green id:: 63667fb5-57de-44e2-b183-33bb8883b9d4
• refine ls-type:: annotation hl-page:: 8 hl-color:: green id:: 63667fb8-025c-45b3-8993-b0fbfe062488
• Regardless of experience, both novices and non-novices are able to iteratively refine their pipelines, but novices face some challenges compared to non-novices regarding actually completing the task. These challenges are discussed in the next research question ls-type:: annotation hl-page:: 9 hl-color:: purple id:: 6366800a-f9f1-4a58-a5c4-1d6bcb6f77e8 hl-stamp:: 1667661837854
• Discovery ls-type:: annotation hl-page:: 9 hl-color:: green id:: 63668041-69b5-433e-85b9-c694611743d8
• Iterative Composition ls-type:: annotation hl-page:: 9 hl-color:: green id:: 63668045-929d-4887-a8b5-75cb2b07cbca
• Challenges ls-type:: annotation hl-page:: 9 hl-color:: green id:: 63668047-d43f-48a8-942d-3eae49ef8cd1
• Feedback ls-type:: annotation hl-page:: 9 hl-color:: green id:: 63668050-7ebe-423e-b0fe-daa7782238eb
• We found in RQ1 that LOWCODERNL was helpful in finding unknown operators compared to other methods. ls-type:: annotation hl-page:: 9 hl-color:: green id:: 6366807d-a2f7-439f-ba21-506ba51db0b3
• We note that challenges regarding general web search is also an axial code ls-type:: annotation hl-page:: 9 hl-color:: purple id:: 636680b0-e004-4009-b158-daef33b2260c
• General challenges (10/20 ls-type:: annotation hl-page:: 10 hl-color:: green id:: 636681d9-a8bf-4610-8ad4-5bea6f37976d
• Our results show that both the LOWCODERVP and LOWCODERNL components were helpful with aspects like discovering operators (RQ1) or iteratively composing pipelines (RQ2), especially for novice participants ls-type:: annotation hl-page:: 10 hl-color:: purple id:: 6366821b-95ca-4f8b-96fd-10dcfab1330b hl-stamp:: 1667662403295
• have an idea of what they would like to do but do not fully know how to accomplish tha ls-type:: annotation hl-page:: 10 hl-color:: purple id:: 6366826d-0e1d-48fc-b81b-0636009586b3
• a number of our participants (including all novices who participated) struggled with knowing what to do ls-type:: annotation hl-page:: 10 hl-color:: purple id:: 6366828d-3c5e-424b-b9d7-114879f68cd7
• particularly difficult in this regard due to its highly experimental nature where progress has a high degree of uncertainty ls-type:: annotation hl-page:: 10 hl-color:: purple id:: 63668299-5375-4b67-ae9a-fa322c4d547c
• Some participants in our studies echo this, identifying that some ML knowledge is necessary to use our tool ls-type:: annotation hl-page:: 10 hl-color:: purple id:: 636682ad-32f7-4482-85e5-54e22cbf12cb
• to provide suggestions in the form of templates or recipes for pipelines ls-type:: annotation hl-page:: 10 hl-color:: purple id:: 636682c6-aa49-4d29-9ac7-fdb870c1b53e
• contextual to the given dataset or active pipeline ls-type:: annotation hl-page:: 10 hl-color:: purple id:: 636682d2-0c6c-469f-83ee-be4c5f27edbc
• hese contextual suggestions may also help in guiding developers in what to do, making for a more generally useful low-code tool for citizen developers and pro-developers alike. ls-type:: annotation hl-page:: 10 hl-color:: purple id:: 6366831f-1038-491f-aea0-0878e3134251
• Threats to Validity ls-type:: annotation hl-page:: 10 hl-color:: blue id:: 63668327-2deb-4dcc-9956-8e3b2322a07e
• small, public tabular datasets and scikit-learn operators and may not be indicative of other machine learning tasks such as deep learning on large datasets ls-type:: annotation hl-page:: 10 hl-color:: purple id:: 63668336-a4e5-4d06-8295-a7cfa82d2aef
• Participants also all come from the same large technology company and may not be representative of general user ls-type:: annotation hl-page:: 10 hl-color:: purple id:: 63668345-5921-452e-b3e8-477253c44c15
• This paper describes LOWCODER, an AI-driven low-code tool combining a visual programming (VP) interface with programming by natural language (NL) that assists users with a variety of skill levels in creating AI pipelines. ls-type:: annotation hl-page:: 10 hl-color:: purple id:: 6366837b-120c-49ec-aa87-853ff19b9ba4
• the tool is particularly useful for helping users, even machine learning novices, discover previouslyunknown operators and compose and iterate AI pipelines ls-type:: annotation hl-page:: 10 hl-color:: purple id:: 63668387-99f8-4259-94bc-f1de1e00dc90
• lack of guidance for “what” actions should be performed for a given dataset. ls-type:: annotation hl-page:: 10 hl-color:: purple id:: 63668391-d7a6-4468-b0b3-cae28c3f87b7