722 lines
26 KiB
Markdown
722 lines
26 KiB
Markdown
file:: [icse2023-paper1528_1666905345862_0.pdf](../assets/icse2023-paper1528_1666905345862_0.pdf)
|
||
file-path:: ../assets/icse2023-paper1528_1666905345862_0.pdf
|
||
|
||
- quickly compare pipeline variants by allowing them to swap out code blocks
|
||
ls-type:: annotation
|
||
hl-page:: 1
|
||
hl-color:: green
|
||
id:: 636290e3-ab3a-4ea5-974d-a2dc85d1ab00
|
||
- AI-driven natural language interface
|
||
ls-type:: annotation
|
||
hl-page:: 1
|
||
hl-color:: green
|
||
id:: 63629139-1857-4eb4-afc7-c8a878447534
|
||
- CodeT5 model
|
||
ls-type:: annotation
|
||
hl-page:: 1
|
||
hl-color:: green
|
||
id:: 6362914f-491a-40a2-8c5c-a2c11ad70422
|
||
- generate partial operator invocations depending on available information
|
||
ls-type:: annotation
|
||
hl-page:: 1
|
||
hl-color:: green
|
||
id:: 636291e4-d596-45d4-a4d3-05a8db2d1bc5
|
||
- Discoverability
|
||
ls-type:: annotation
|
||
hl-page:: 1
|
||
hl-color:: green
|
||
id:: 636291fa-bfc2-431a-aafc-2c66172198bb
|
||
- terative Composition
|
||
ls-type:: annotation
|
||
hl-page:: 1
|
||
hl-color:: green
|
||
id:: 63629201-caf6-4a16-bcfc-43d2b8167eb4
|
||
- Overall, our work demonstrates the promise of combining both an AIpowered natural language interface and a visual interface for helping developers create AI pipelines without writing code.
|
||
ls-type:: annotation
|
||
hl-page:: 1
|
||
hl-color:: green
|
||
id:: 63629212-d7c2-4f82-afb9-e5daf637cd00
|
||
- citizen developers are people without formal training or experience as AI software developers. Furthermore, these libraries are often large. Needing to remember hundreds of AI operators and their hyper-parameters slows down even professional developers.
|
||
ls-type:: annotation
|
||
hl-page:: 1
|
||
hl-color:: yellow
|
||
id:: 63629241-cab5-4ad3-b3bf-f84db2144e87
|
||
- to democratize software development and increase productivity.
|
||
ls-type:: annotation
|
||
hl-page:: 1
|
||
hl-color:: green
|
||
id:: 636292be-8716-408e-af74-5bcb4aa13209
|
||
- discoverability
|
||
ls-type:: annotation
|
||
hl-page:: 1
|
||
hl-color:: green
|
||
id:: 636292ff-cf62-41d3-beff-10d77e2d8db8
|
||
- LOWCODER to overcome this problem by combining visual programming with PBNL
|
||
ls-type:: annotation
|
||
hl-page:: 1
|
||
hl-color:: green
|
||
id:: 63629306-a399-4dc8-ae3a-eb4c9c01adc9
|
||
- programming by natural language (PBNL)
|
||
ls-type:: annotation
|
||
hl-page:: 1
|
||
hl-color:: purple
|
||
id:: 6362930f-5a5c-49ac-a0d6-39e256d55fc4
|
||
- This does not always return correct programs, necessitating a way to help users understand and fix generated programs.
|
||
ls-type:: annotation
|
||
hl-page:: 1
|
||
hl-color:: purple
|
||
id:: 636293b6-f2ce-483b-b953-9a1b46a4bab7
|
||
- The visual programming component of LOWCODER, LOWCODERVP, lets users snap together visual blocks for AI operators into well-structured AI pipelines
|
||
ls-type:: annotation
|
||
hl-page:: 1
|
||
hl-color:: purple
|
||
id:: 636293c2-d030-4841-b388-a1f86642abf9
|
||
- We further noticed that it is common in this domain for queries to mention at most a subset of hyper-parameters for each pipeline step, so we developed a novel task formulation tailored to this use case that improved learning outcomes
|
||
ls-type:: annotation
|
||
hl-page:: 1
|
||
hl-color:: yellow
|
||
id:: 63629405-9cd5-4e1c-a87c-5f3f78b4fb53
|
||
- showing the effect of pipelines on data
|
||
ls-type:: annotation
|
||
hl-page:: 1
|
||
hl-color:: purple
|
||
id:: 63629415-ad85-448b-b83f-7b213aae4cdd
|
||
- We find that LOWCODERNL helped users discover previouslyunknown operators more easily than other methods, such as web search
|
||
ls-type:: annotation
|
||
hl-page:: 1
|
||
hl-color:: purple
|
||
id:: 63629431-4ebc-47a2-9775-cd3301a0ae32
|
||
- e explore AI models for PBNL and integrate those models with the visual tool.
|
||
ls-type:: annotation
|
||
hl-page:: 2
|
||
hl-color:: green
|
||
id:: 6362944a-22e9-43dd-b948-99f59646234a
|
||
- ow-Code for AI:
|
||
ls-type:: annotation
|
||
hl-page:: 1
|
||
hl-color:: blue
|
||
id:: 6362944e-fde5-4af4-acfa-e211049f2f0a
|
||
- AI for Low-Code:
|
||
ls-type:: annotation
|
||
hl-page:: 2
|
||
hl-color:: blue
|
||
id:: 63629452-f8ad-4ece-aecb-7f3b671b2db7
|
||
- User Study:
|
||
ls-type:: annotation
|
||
hl-page:: 2
|
||
hl-color:: blue
|
||
id:: 63629455-c670-4bb5-ba5d-ec1b768bb43c
|
||
- G. Baudart, M. Hirzel, K. Kate, P. Ram, A. Shinnar, and J. Tsay, “Pipeline combinators for gradual AutoML,” in Advances in Neural Information Processing Systems (NeurIPS), Dec.2021. [Online]. Available: https://proceedings.neurips.cc/paper/2021/ file/a3b36cb25e2e0b93b5f334ffb4e4064e-Paper.pd
|
||
ls-type:: annotation
|
||
hl-page:: 11
|
||
hl-color:: green
|
||
id:: 63629484-ce26-44f9-ac7e-4732196bff4e
|
||
- AI for Low-code for AI
|
||
ls-type:: annotation
|
||
hl-page:: 2
|
||
hl-color:: blue
|
||
id:: 6362948f-8546-4d5e-9a46-c9765f7ed69a
|
||
- AI for Low-code for AI
|
||
ls-type:: annotation
|
||
hl-page:: 2
|
||
hl-color:: blue
|
||
id:: 6362949c-6600-4908-8742-c277680df426
|
||
- we add a natural language interface to help users discover and configure operators.
|
||
ls-type:: annotation
|
||
hl-page:: 2
|
||
hl-color:: purple
|
||
id:: 636294b4-06c4-4dee-b69e-c66d60ee1c5b
|
||
- AI for Low-code for AI:
|
||
ls-type:: annotation
|
||
hl-page:: 2
|
||
hl-color:: blue
|
||
id:: 636294d0-40b5-4642-8a2c-e20ff58d940e
|
||
- Combining low-code techniques
|
||
ls-type:: annotation
|
||
hl-page:: 2
|
||
hl-color:: blue
|
||
id:: 636294ef-3e7c-497f-8748-6f69ff516113
|
||
- CodeT5
|
||
ls-type:: annotation
|
||
hl-page:: 2
|
||
hl-color:: green
|
||
id:: 63629516-298f-490b-b17f-029c15cd5793
|
||
- PBNL as a write-only view,
|
||
ls-type:: annotation
|
||
hl-page:: 2
|
||
hl-color:: green
|
||
id:: 63629784-e4b1-4626-a1ba-e14902883f04
|
||
- Python Flask back-end serve
|
||
ls-type:: annotation
|
||
hl-page:: 2
|
||
hl-color:: green
|
||
id:: 636297a1-b7bd-4f95-af19-d289485255db
|
||
- front-end based on Blockl
|
||
ls-type:: annotation
|
||
hl-page:: 2
|
||
hl-color:: green
|
||
id:: 636297a5-5617-4a22-a339-53a3b3a42fa2
|
||
- LOWCODER supports 143 sklearn operators. Sklearn distinguishes between operators, which are steps of machine learning pipelines and usually have learned coefficients, and functions, which are used separately and cannot be a step in a pipeline.
|
||
ls-type:: annotation
|
||
hl-page:: 2
|
||
hl-color:: green
|
||
id:: 636297d4-ccc1-4477-9616-39ef5093f309
|
||
- LOWCODER focuses on pipelines, it only has explicit blocks for operators but not functions
|
||
ls-type:: annotation
|
||
hl-page:: 3
|
||
hl-color:: purple
|
||
id:: 636297eb-c4db-48f6-a996-ac1d9d3595f6
|
||
- Lale which is then sent to the back-end
|
||
ls-type:: annotation
|
||
hl-page:: 3
|
||
hl-color:: green
|
||
id:: 636297f5-34e5-40a9-80e7-b8583012ed8b
|
||
- One goal that this tool shares with other block-based visual tools such as Scratch [9] is to encourage a more “tinkerable” experience.
|
||
ls-type:: annotation
|
||
hl-page:: 3
|
||
hl-color:: green
|
||
id:: 6362983b-ff09-40ac-910c-c90b5bc4fd9d
|
||
- For ease of execution, our tool only allows for one valid pipeline at a time, so blocks must be attached downstream of the pre-defined Start block to be considered part of the active pipeline.
|
||
ls-type:: annotation
|
||
hl-page:: 3
|
||
hl-color:: green
|
||
id:: 63629871-3f13-406c-912e-708d9f9b5f34
|
||
- hyper-parameter for an operator along with a description (when hovering over the hyper-parameter name) and default values along with input boxes to modify each hyper-parameter.
|
||
ls-type:: annotation
|
||
hl-page:: 3
|
||
hl-color:: yellow
|
||
id:: 63629949-0e6c-4881-b329-df61e0399d17
|
||
- the palette of LOWCODERVP contains more than a hundred operator blocks
|
||
ls-type:: annotation
|
||
hl-page:: 3
|
||
hl-color:: green
|
||
id:: 63629a5e-e217-4e86-bc85-474d40cf9eb4
|
||
- The tool then infers relevant operator(s) and any applicable hyper-parameters using an underlying natural language to code translation model and automatically adds the most relevant operator to the end of the pipeline.
|
||
ls-type:: annotation
|
||
hl-page:: 3
|
||
hl-color:: green
|
||
id:: 63629ab7-08fc-4f04-86f8-d2df98fe699b
|
||
- PCA with 2 components”
|
||
ls-type:: annotation
|
||
hl-page:: 3
|
||
hl-color:: green
|
||
id:: 63629ad1-2257-4739-8d7b-b5acd3bf37d8
|
||
- A potential weakness of natural language low-code tools is that the generated programs can be incorrect, due to a lack of clarity, or ambiguity, in the query, or a lack of context for the model providing inferences [16]
|
||
ls-type:: annotation
|
||
hl-page:: 3
|
||
hl-color:: green
|
||
id:: 63629ae4-2308-488d-a9d0-9329d8af893b
|
||
- isual inputs and representations are unambiguous [25], requiring no probabilistic interpretation, so users can easily understand and manipulate the results returned by LOWCODERNL
|
||
ls-type:: annotation
|
||
hl-page:: 3
|
||
hl-color:: green
|
||
id:: 63629aee-5414-4071-a9dd-587424b6e03e
|
||
- RandomForestClassifier
|
||
ls-type:: annotation
|
||
hl-page:: 3
|
||
hl-color:: blue
|
||
id:: 63629b26-6075-450b-bc74-8a36514a0599
|
||
- Data Collection
|
||
ls-type:: annotation
|
||
hl-page:: 4
|
||
hl-color:: blue
|
||
id:: 63629b68-a90e-4591-969d-63f740bc5ba0
|
||
- escribe what they want to do when they do not know how
|
||
ls-type:: annotation
|
||
hl-page:: 4
|
||
hl-color:: green
|
||
id:: 6362d2a3-f95c-4b21-99c9-59368b0e4336
|
||
- sklearn pipelines in a low-code setting, using a natural language interface that can be used as an intelligent search tool
|
||
ls-type:: annotation
|
||
hl-page:: 4
|
||
hl-color:: purple
|
||
id:: 6362d2ad-b334-481a-99cf-ecb14265e255
|
||
- proxy dataset using 140K Python Kaggle notebooks that were collected as part of the Google AI4Code challenge
|
||
ls-type:: annotation
|
||
hl-page:: 4
|
||
hl-color:: green
|
||
id:: 6362d2d9-8a3a-40d3-b868-3b5470eaaca8
|
||
- B. Data Preprocessing
|
||
ls-type:: annotation
|
||
hl-page:: 4
|
||
hl-color:: blue
|
||
id:: 6362d33e-c15d-4554-8104-de8a0ee885ce
|
||
- 211,916 aligned NL-code pairs
|
||
ls-type:: annotation
|
||
hl-page:: 4
|
||
hl-color:: green
|
||
id:: 6362d35c-e6d2-4382-8f5c-c3da398d7d66
|
||
- We discard any code cells that do not include sklearn operation invocation statements but include other sklearn code leaving a final total of 79,372 NL-Code pairs.
|
||
ls-type:: annotation
|
||
hl-page:: 4
|
||
hl-color:: green
|
||
id:: 6362d387-bcbf-4e72-a894-4695cb2cedd3
|
||
- 64,779 train samples
|
||
ls-type:: annotation
|
||
hl-page:: 4
|
||
hl-color:: green
|
||
id:: 6362d393-43c8-46b7-8dfe-df57b26cd500
|
||
- 7,242 validation samples
|
||
ls-type:: annotation
|
||
hl-page:: 4
|
||
hl-color:: green
|
||
id:: 6362d397-8731-40f7-a25e-2f0af5e19b68
|
||
- 7,351 test samples
|
||
ls-type:: annotation
|
||
hl-page:: 4
|
||
hl-color:: green
|
||
id:: 6362d39c-6085-48ff-8590-6f5cf3213087
|
||
- Given the NL query, our model aims to generate a line of sklearn code corresponding to an operation invocation that can be used to build the next step of the pipeline.
|
||
ls-type:: annotation
|
||
hl-page:: 4
|
||
hl-color:: green
|
||
id:: 6362d3e1-a371-499f-b3e9-d0307cd94fe4
|
||
- C. Tasks
|
||
ls-type:: annotation
|
||
hl-page:: 4
|
||
hl-color:: blue
|
||
id:: 6362d3e5-10af-46de-9ac3-3fa262309cb2
|
||
- 1) Operator Name Generation:
|
||
ls-type:: annotation
|
||
hl-page:: 4
|
||
hl-color:: blue
|
||
id:: 6362d457-02f9-4083-b161-60fe697265fd
|
||
- 2) Complete Operator Invocation Generation
|
||
ls-type:: annotation
|
||
hl-page:: 4
|
||
hl-color:: blue
|
||
id:: 6362d470-f746-45ba-8239-a421b6bf92a1
|
||
- often predicts arbitrary hyperparameter values, resulting in lines of code that can rarely be used directly by developers.
|
||
ls-type:: annotation
|
||
hl-page:: 4
|
||
hl-color:: yellow
|
||
id:: 6362d48b-528b-4b11-9553-ad2b24829f57
|
||
- 3) Masked Operator Invocation Generation
|
||
ls-type:: annotation
|
||
hl-page:: 4
|
||
hl-color:: blue
|
||
id:: 6362d619-8a7f-4fb1-a437-a026634b1dca
|
||
- ensure that the model learns to predict the specific invocation signature, even if it is unaware of the values to provide for the hyper-parameters.
|
||
ls-type:: annotation
|
||
hl-page:: 4
|
||
hl-color:: green
|
||
id:: 6362d62e-e3d0-4cc8-b989-92806337015e
|
||
- 4) Hybrid Operator Invocation Generation (HOI)
|
||
ls-type:: annotation
|
||
hl-page:: 4
|
||
hl-color:: blue
|
||
id:: 6362d638-115e-4aec-aea5-16eb3099995d
|
||
- This gives the model an opportunity to learn the hyper-parameter names and values if they are explicitly stated in the description,
|
||
ls-type:: annotation
|
||
hl-page:: 4
|
||
hl-color:: blue
|
||
id:: 6362d8c6-342c-40f6-a399-54287972425c
|
||
- equence-tosequence model
|
||
ls-type:: annotation
|
||
hl-page:: 4
|
||
hl-color:: green
|
||
id:: 6362d8d1-9473-482a-9996-49758c21316e
|
||
- 1) Transformer (from scratch):
|
||
ls-type:: annotation
|
||
hl-page:: 5
|
||
hl-color:: blue
|
||
id:: 6362d93e-b55a-4797-9151-210584316923
|
||
- trifecta
|
||
ls-type:: annotation
|
||
hl-page:: 5
|
||
hl-color:: green
|
||
id:: 6362d95f-7e8e-45fe-a7b4-1dbb4656a48b
|
||
- medium sized models (100M-1B parameters) are pretrained with a generic training signal and then fine-tuned on task data
|
||
ls-type:: annotation
|
||
hl-page:: 5
|
||
hl-color:: green
|
||
id:: 6362d973-97c8-489e-b527-03d06773be58
|
||
- arge models(>1B parameters) are only pretrained on very large datasets and are prompted with examples from the training data as demonstration followed by the query
|
||
ls-type:: annotation
|
||
hl-page:: 5
|
||
hl-color:: green
|
||
id:: 6362d97d-124b-4fcd-a62b-93062c4e98ce
|
||
- 2) Fine-tuning CodeT5
|
||
ls-type:: annotation
|
||
hl-page:: 5
|
||
hl-color:: blue
|
||
id:: 6362d98e-5800-4f3d-987a-1c641a5953ac
|
||
- We fine-tune CodeT5 on the HOI generation task by adding the ‘Generate Python’ prefix to all NL queries.
|
||
ls-type:: annotation
|
||
hl-page:: 5
|
||
hl-color:: green
|
||
id:: 6362d9ba-9322-49b4-964e-0394fb429605
|
||
- 3) Few-Shot Learning With CodeGen
|
||
ls-type:: annotation
|
||
hl-page:: 5
|
||
hl-color:: blue
|
||
id:: 6362d9c2-b8f1-41bd-939f-97d10f94b925
|
||
- Models at this scale are expensive to fine-tune and are instead commonly used for inference by means of “few-shot prompting”
|
||
ls-type:: annotation
|
||
hl-page:: 5
|
||
hl-color:: green
|
||
id:: 6362d9ea-c5cd-48de-a0d4-f768837d82bc
|
||
- 48 GB NVIDIA Quadro RTX 8000 GPUs
|
||
ls-type:: annotation
|
||
hl-page:: 6
|
||
hl-color:: green
|
||
id:: 6362da2d-5d45-466d-90fe-a968b9ac7ecf
|
||
- ection 1 of the supplementary material
|
||
ls-type:: annotation
|
||
hl-page:: 4
|
||
hl-color:: yellow
|
||
id:: 636435bc-49c5-4fe5-b420-49598f00fb93
|
||
- Section IV-D1)
|
||
ls-type:: annotation
|
||
hl-page:: 4
|
||
hl-color:: red
|
||
id:: 63658ea4-c041-4794-a0b9-a9052f8073d2
|
||
- such models heavily rely on data to learn such behaviour and would need to be trained on an aligned dataset of natural language queries and the corresponding sklearn line(s) of code demonstrating how a user would want to use such an intelligent search too
|
||
ls-type:: annotation
|
||
hl-page:: 4
|
||
hl-color:: green
|
||
id:: 63666434-5bdf-4402-829c-46fb6cdee121
|
||
- V-A5
|
||
ls-type:: annotation
|
||
hl-page:: 4
|
||
hl-color:: yellow
|
||
id:: 63666457-cadf-413b-ac9f-e1d054cabfc2
|
||
- We further filter out notebooks with non-English descriptions in all of the markdown cells, resulting in 59,569 notebooks
|
||
ls-type:: annotation
|
||
hl-page:: 4
|
||
hl-color:: green
|
||
id:: 63666481-c796-4e5a-9c50-df4976b73b23
|
||
- ne(s) of code corresponding to an sklearn operation invocation statement
|
||
ls-type:: annotation
|
||
hl-page:: 4
|
||
hl-color:: green
|
||
id:: 6366651a-0597-44c5-9367-72d2341c29f4
|
||
- train/validation/test split
|
||
ls-type:: annotation
|
||
hl-page:: 4
|
||
hl-color:: yellow
|
||
id:: 63666544-d8bf-45ec-ad10-7b06d50d8ac2
|
||
- . A common alternative is to pretrain a larger model on a large dataset based on a generic task, such as masked token infilling, and/or a combination of objectives. Such models can then be fine-tuned (calibrated) on a smaller dataset to high accuracy. For the second paradigm, we fine-tune one such“medium” sized model, CodeT5 [10]. Finally, Large Language Models (LLMs), with billions of parameters, are often able to generalize to new tasks from just a few examples. These models are typically expensive to train from scratch or fine-tune; we adopt such a model, named CodeGen [11], and query it by means of few-shot prompting. We elaborate on these models below. Note that we use top-k sampling for our top-5 results.(A comparison of results with other decoding strategies can be found in Section 3 and 4 in the supplementary material
|
||
ls-type:: annotation
|
||
hl-page:: 5
|
||
hl-color:: yellow
|
||
id:: 63666896-e137-411e-9645-854a1139dba2
|
||
- ) Experimental Setup
|
||
ls-type:: annotation
|
||
hl-page:: 6
|
||
hl-color:: blue
|
||
id:: 6366734a-6404-4e70-a368-0c8e8c944faf
|
||
- Test Datasets
|
||
ls-type:: annotation
|
||
hl-page:: 6
|
||
hl-color:: blue
|
||
id:: 6366736e-4753-441c-8b27-19e57585a47b
|
||
- We log all the NL queries that users searched for in LOWCODER during the user studie
|
||
ls-type:: annotation
|
||
hl-page:: 6
|
||
hl-color:: yellow
|
||
id:: 636673a0-4717-4898-b049-1c7ce2b2aad3
|
||
- We evaluate the models’ ability to generate just the operator name as well as the entire operator invocation (including all the hyper-parameter names and values) based on the hybrid formulation
|
||
ls-type:: annotation
|
||
hl-page:: 6
|
||
hl-color:: green
|
||
id:: 636674a8-8026-4029-ac03-c91e187d032b
|
||
- Test Metrics
|
||
ls-type:: annotation
|
||
hl-page:: 6
|
||
hl-color:: blue
|
||
id:: 636674ab-08ea-4b85-9608-982d03f1c5c8
|
||
- ask Comparison:
|
||
ls-type:: annotation
|
||
hl-page:: 6
|
||
hl-color:: blue
|
||
id:: 636674af-203b-460a-a3f7-e5d52559b846
|
||
- e compare the model’s ability to correctly generate the operator name and the operator invocation based on the formulation corresponding to the training task using top-1 and top-5 accuracy as shown in Figure 4
|
||
ls-type:: annotation
|
||
hl-page:: 6
|
||
hl-color:: green
|
||
id:: 63667575-2986-470b-9e8d-f9f5d8219f6d
|
||
- Model Comparison
|
||
ls-type:: annotation
|
||
hl-page:: 6
|
||
hl-color:: blue
|
||
id:: 636675cd-7391-4915-ab87-d5cf1a842fdc
|
||
- (See Section 4 in the supplementary material for additional results and ablation studies.)
|
||
ls-type:: annotation
|
||
hl-page:: 6
|
||
hl-color:: yellow
|
||
id:: 636675f4-f71b-47eb-b382-c362848f5384
|
||
- Performance in Practice
|
||
ls-type:: annotation
|
||
hl-page:: 7
|
||
hl-color:: blue
|
||
id:: 63667694-e6d3-40f8-aa41-db308a4224f2
|
||
- The distribution of NL queries collected from the user studies represents the “true” distribution of queries that can be expected from users in a low-code settin
|
||
ls-type:: annotation
|
||
hl-page:: 7
|
||
hl-color:: green
|
||
id:: 63667746-e018-466e-98a7-4bf021715343
|
||
- operation name
|
||
ls-type:: annotation
|
||
hl-page:: 7
|
||
hl-color:: green
|
||
id:: 63667791-f92d-437e-9a0b-ffe8fcb38f59
|
||
- 141 out of185 queries correctly for an overall accuracy of 76.2%.
|
||
ls-type:: annotation
|
||
hl-page:: 7
|
||
hl-color:: green
|
||
id:: 636677bd-3a9c-4579-877e-5c374c3e8cbd
|
||
- B. User Study
|
||
ls-type:: annotation
|
||
hl-page:: 7
|
||
hl-color:: blue
|
||
id:: 636677ef-19c0-441e-843e-1a4d7a6d37be
|
||
- 20 participants using two versions of our too
|
||
ls-type:: annotation
|
||
hl-page:: 7
|
||
hl-color:: yellow
|
||
id:: 636677ff-5f77-4431-9dfe-66d3478f430a
|
||
- RQ1
|
||
ls-type:: annotation
|
||
hl-page:: 7
|
||
hl-color:: green
|
||
id:: 63667849-2d45-45f5-842f-7e7769ea2397
|
||
- RQ2
|
||
ls-type:: annotation
|
||
hl-page:: 7
|
||
hl-color:: green
|
||
id:: 6366784b-be60-4056-babd-97bb73aca835
|
||
- RQ3
|
||
ls-type:: annotation
|
||
hl-page:: 7
|
||
hl-color:: green
|
||
id:: 6366784e-035f-4da9-8e05-2d31253327a4
|
||
- How do LOWCODERNL and other features help participants discover previously-unknown operators?
|
||
ls-type:: annotation
|
||
hl-page:: 7
|
||
hl-color:: green
|
||
id:: 63667a77-288c-4cb2-9a30-2c1862f05d06
|
||
- Are participants able to compose and then iteratively refine AI pipelines in our tool
|
||
ls-type:: annotation
|
||
hl-page:: 7
|
||
hl-color:: green
|
||
id:: 63667a7e-cbd5-4500-abf0-52fbcf577fe0
|
||
- What are the benefits and challenges of using low-code for AI?
|
||
ls-type:: annotation
|
||
hl-page:: 7
|
||
hl-color:: green
|
||
id:: 63667a85-300e-4352-9a07-26460b44b62e
|
||
- 1) Study Methodology
|
||
ls-type:: annotation
|
||
hl-page:: 7
|
||
hl-color:: blue
|
||
id:: 63667a96-466e-42ae-9a85-667e2b505350
|
||
- using LOWCODER with (NL condition) and without (keyword condition) the natural language (NL) interface powered by LOWCODERNL
|
||
ls-type:: annotation
|
||
hl-page:: 7
|
||
hl-color:: green
|
||
id:: 63667b06-2100-4ee5-abc4-cccf5c529210
|
||
- participants were instructed to create AI pipelines with data preprocessing and classifier steps on a sample dataset with as high a score (accuracy on the test set) as possible during a time period of five to ten minutes
|
||
ls-type:: annotation
|
||
hl-page:: 7
|
||
hl-color:: purple
|
||
id:: 63667b4f-b600-417e-b0bb-e866b20fe5ec
|
||
hl-stamp:: 1667660627064
|
||
- he order of the conditions and the order of the tasks were shuffled such that there is a uniform distribution of the order of conditions and tasks.
|
||
ls-type:: annotation
|
||
hl-page:: 7
|
||
hl-color:: green
|
||
id:: 63667b83-d211-4881-9def-b046fdeac968
|
||
- o avoid operators that were potentially useful in user tasks, the overview used both a non-sklearn operator that was not available in the study versions of the tool as well as sklearn’s DummyClassifier that generates predictions without considering input features
|
||
ls-type:: annotation
|
||
hl-page:: 8
|
||
hl-color:: green
|
||
id:: 63667ba6-6c35-4afd-babf-382f86709bb4
|
||
- 2) Data Collection and Analysis
|
||
ls-type:: annotation
|
||
hl-page:: 8
|
||
hl-color:: blue
|
||
id:: 63667c1e-1c5c-411a-a26f-954429efe242
|
||
- previously-unknow
|
||
ls-type:: annotation
|
||
hl-page:: 8
|
||
hl-color:: green
|
||
id:: 63667c39-6156-4d4d-a024-361575f79954
|
||
- a preprocessing operator block is added or swapped,
|
||
ls-type:: annotation
|
||
hl-page:: 8
|
||
hl-color:: green
|
||
id:: 63667c88-0dce-4bc7-81c9-aef93c74fe32
|
||
- a classifier block is swapped
|
||
ls-type:: annotation
|
||
hl-page:: 8
|
||
hl-color:: green
|
||
id:: 63667c8f-cd80-48aa-b25c-c450ea5496aa
|
||
- yper-parameters are tuned.
|
||
ls-type:: annotation
|
||
hl-page:: 8
|
||
hl-color:: green
|
||
id:: 63667c96-cd01-4e5d-b1b6-c68cd55f606d
|
||
- Both sets of quantitative metrics are counted per task (80 tasks total for 20 participants).
|
||
ls-type:: annotation
|
||
hl-page:: 8
|
||
hl-color:: green
|
||
id:: 63667ca4-4d19-4803-aaf5-b7f02b9c5a81
|
||
- qualitative data to answer RQ3.
|
||
ls-type:: annotation
|
||
hl-page:: 8
|
||
hl-color:: green
|
||
id:: 63667cad-35ef-4f74-bce8-1df8c6e00fe8
|
||
- 3) Study Results
|
||
ls-type:: annotation
|
||
hl-page:: 8
|
||
hl-color:: blue
|
||
id:: 63667cf0-81bf-474c-ba1b-25287034a930
|
||
- We note that the participants were not able to use the keyword search to discover unknown operators due to needing at least part of the exact name
|
||
ls-type:: annotation
|
||
hl-page:: 8
|
||
hl-color:: green
|
||
id:: 63667efe-87d1-4966-bc82-cd5f425df4fe
|
||
- These results suggest that LOWCODERNL is particularly helpful in discovering previously-unknown operators for non-novices but novices face some challenges. We discuss these challenges in RQ3.
|
||
ls-type:: annotation
|
||
hl-page:: 8
|
||
hl-color:: purple
|
||
id:: 63667f2c-0552-4c2a-bf68-1deb50881101
|
||
hl-stamp:: 1667661842337
|
||
- sks participants commplete and iterate on preprocessors, classifier, and hyper-parameters.
|
||
ls-type:: annotation
|
||
hl-page:: 9
|
||
hl-color:: green
|
||
id:: 63667f66-5d48-460a-ae4c-515854f6d33c
|
||
- compose
|
||
ls-type:: annotation
|
||
hl-page:: 8
|
||
hl-color:: green
|
||
id:: 63667fb0-a06a-47e0-b960-12a7d47460c7
|
||
- iter
|
||
ls-type:: annotation
|
||
hl-page:: 8
|
||
hl-color:: green
|
||
id:: 63667fb2-10aa-4d29-a6d2-817672207732
|
||
- atively
|
||
ls-type:: annotation
|
||
hl-page:: 8
|
||
hl-color:: green
|
||
id:: 63667fb5-57de-44e2-b183-33bb8883b9d4
|
||
- refine
|
||
ls-type:: annotation
|
||
hl-page:: 8
|
||
hl-color:: green
|
||
id:: 63667fb8-025c-45b3-8993-b0fbfe062488
|
||
- Regardless of experience, both novices and non-novices are able to iteratively refine their pipelines, but novices face some challenges compared to non-novices regarding actually completing the task. These challenges are discussed in the next research question
|
||
ls-type:: annotation
|
||
hl-page:: 9
|
||
hl-color:: purple
|
||
id:: 6366800a-f9f1-4a58-a5c4-1d6bcb6f77e8
|
||
hl-stamp:: 1667661837854
|
||
- Discovery
|
||
ls-type:: annotation
|
||
hl-page:: 9
|
||
hl-color:: green
|
||
id:: 63668041-69b5-433e-85b9-c694611743d8
|
||
- Iterative Composition
|
||
ls-type:: annotation
|
||
hl-page:: 9
|
||
hl-color:: green
|
||
id:: 63668045-929d-4887-a8b5-75cb2b07cbca
|
||
- Challenges
|
||
ls-type:: annotation
|
||
hl-page:: 9
|
||
hl-color:: green
|
||
id:: 63668047-d43f-48a8-942d-3eae49ef8cd1
|
||
- Feedback
|
||
ls-type:: annotation
|
||
hl-page:: 9
|
||
hl-color:: green
|
||
id:: 63668050-7ebe-423e-b0fe-daa7782238eb
|
||
- We found in RQ1 that LOWCODERNL was helpful in finding unknown operators compared to other methods.
|
||
ls-type:: annotation
|
||
hl-page:: 9
|
||
hl-color:: green
|
||
id:: 6366807d-a2f7-439f-ba21-506ba51db0b3
|
||
- We note that challenges regarding general web search is also an axial code
|
||
ls-type:: annotation
|
||
hl-page:: 9
|
||
hl-color:: purple
|
||
id:: 636680b0-e004-4009-b158-daef33b2260c
|
||
- General challenges (10/20
|
||
ls-type:: annotation
|
||
hl-page:: 10
|
||
hl-color:: green
|
||
id:: 636681d9-a8bf-4610-8ad4-5bea6f37976d
|
||
- Our results show that both the LOWCODERVP and LOWCODERNL components were helpful with aspects like discovering operators (RQ1) or iteratively composing pipelines (RQ2), especially for novice participants
|
||
ls-type:: annotation
|
||
hl-page:: 10
|
||
hl-color:: purple
|
||
id:: 6366821b-95ca-4f8b-96fd-10dcfab1330b
|
||
hl-stamp:: 1667662403295
|
||
- have an idea of what they would like to do but do not fully know how to accomplish tha
|
||
ls-type:: annotation
|
||
hl-page:: 10
|
||
hl-color:: purple
|
||
id:: 6366826d-0e1d-48fc-b81b-0636009586b3
|
||
- a number of our participants (including all novices who participated) struggled with knowing what to do
|
||
ls-type:: annotation
|
||
hl-page:: 10
|
||
hl-color:: purple
|
||
id:: 6366828d-3c5e-424b-b9d7-114879f68cd7
|
||
- particularly difficult in this regard due to its highly experimental nature where progress has a high degree of uncertainty
|
||
ls-type:: annotation
|
||
hl-page:: 10
|
||
hl-color:: purple
|
||
id:: 63668299-5375-4b67-ae9a-fa322c4d547c
|
||
- Some participants in our studies echo this, identifying that some ML knowledge is necessary to use our tool
|
||
ls-type:: annotation
|
||
hl-page:: 10
|
||
hl-color:: purple
|
||
id:: 636682ad-32f7-4482-85e5-54e22cbf12cb
|
||
- to provide suggestions in the form of templates or recipes for pipelines
|
||
ls-type:: annotation
|
||
hl-page:: 10
|
||
hl-color:: purple
|
||
id:: 636682c6-aa49-4d29-9ac7-fdb870c1b53e
|
||
- contextual to the given dataset or active pipeline
|
||
ls-type:: annotation
|
||
hl-page:: 10
|
||
hl-color:: purple
|
||
id:: 636682d2-0c6c-469f-83ee-be4c5f27edbc
|
||
- hese contextual suggestions may also help in guiding developers in what to do, making for a more generally useful low-code tool for citizen developers and pro-developers alike.
|
||
ls-type:: annotation
|
||
hl-page:: 10
|
||
hl-color:: purple
|
||
id:: 6366831f-1038-491f-aea0-0878e3134251
|
||
- Threats to Validity
|
||
ls-type:: annotation
|
||
hl-page:: 10
|
||
hl-color:: blue
|
||
id:: 63668327-2deb-4dcc-9956-8e3b2322a07e
|
||
- small, public tabular datasets and scikit-learn operators and may not be indicative of other machine learning tasks such as deep learning on large datasets
|
||
ls-type:: annotation
|
||
hl-page:: 10
|
||
hl-color:: purple
|
||
id:: 63668336-a4e5-4d06-8295-a7cfa82d2aef
|
||
- Participants also all come from the same large technology company and may not be representative of general user
|
||
ls-type:: annotation
|
||
hl-page:: 10
|
||
hl-color:: purple
|
||
id:: 63668345-5921-452e-b3e8-477253c44c15
|
||
- This paper describes LOWCODER, an AI-driven low-code tool combining a visual programming (VP) interface with programming by natural language (NL) that assists users with a variety of skill levels in creating AI pipelines.
|
||
ls-type:: annotation
|
||
hl-page:: 10
|
||
hl-color:: purple
|
||
id:: 6366837b-120c-49ec-aa87-853ff19b9ba4
|
||
- the tool is particularly useful for helping users, even machine learning novices, discover previouslyunknown operators and compose and iterate AI pipelines
|
||
ls-type:: annotation
|
||
hl-page:: 10
|
||
hl-color:: purple
|
||
id:: 63668387-99f8-4259-94bc-f1de1e00dc90
|
||
- lack of guidance for “what” actions should be performed for a given dataset.
|
||
ls-type:: annotation
|
||
hl-page:: 10
|
||
hl-color:: purple
|
||
id:: 63668391-d7a6-4468-b0b3-cae28c3f87b7 |