147 lines
25 KiB
Markdown
147 lines
25 KiB
Markdown
type:: [[REVIEWS]]
|
||
tags::
|
||
year:: 2024
|
||
venue:: [[ICSE]]
|
||
full-title:: Can an LLM find its way around a Spreadsheet?
|
||
date-start:: [[29-09-2024]] - 19:19
|
||
date-submitted:: [[29-09-2024]]
|
||
external-links::
|
||
status:: [[DONE]]
|
||
deadline-submission::
|
||
file:: [[@icse2025-paper1580]]
|
||
parent::
|
||
todoist:: https://app.todoist.com/app/task/1580-can-an-llm-find-its-way-around-a-spreadsheet-6WFCX7RxpHfvRw2c
|
||
|
||
- ### [[Highlights]]
|
||
-
|
||
- # Annotazioni
|
||
- (29/9/2024, 19:18:59)
|
||
- - “data cleaning prior to analysis and evaluation” ([“icse2025-paper1580”, p. 1](zotero://select/library/items/H5PXLPAG)) ([pdf](zotero://open-pdf/library/items/TRUCG6BD?page=1&annotation=M7MR25TL)) #5fb236
|
||
- - “whether an LLM can find its way around a spreadsheet and how to support end-users in taking their free-form data processing requests to fruition.” ([“icse2025-paper1580”, p. 1](zotero://select/library/items/H5PXLPAG)) ([pdf](zotero://open-pdf/library/items/TRUCG6BD?page=1&annotation=ALHPICEX)) #a28ae5
|
||
- - “we can retrieve elements from a code library to compose data preprocessing pipelines” ([“icse2025-paper1580”, p. 1](zotero://select/library/items/H5PXLPAG)) ([pdf](zotero://open-pdf/library/items/TRUCG6BD?page=1&annotation=62LWXUAW)) #a28ae5
|
||
- - “LLMs can function as coding assistants, enabling users with limited proficiency to produce accurate and executable code to accomplish their tasks” ([“icse2025-paper1580”, p. 1](zotero://select/library/items/H5PXLPAG)) ([pdf](zotero://open-pdf/library/items/TRUCG6BD?page=1&annotation=VBC5HPTE)) #5fb236
|
||
- - “processing tabular data, e.g., in the form of spreadsheets” ([“icse2025-paper1580”, p. 1](zotero://select/library/items/H5PXLPAG)) ([pdf](zotero://open-pdf/library/items/TRUCG6BD?page=1&annotation=9WS8N85I)) #a28ae5
|
||
- - “lack of standardization, necessitating the creation of highly specialized pipeline” ([“icse2025-paper1580”, p. 1](zotero://select/library/items/H5PXLPAG)) ([pdf](zotero://open-pdf/library/items/TRUCG6BD?page=1&annotation=HRNV8ID9)) #5fb236
|
||
- - “correcting typographical errors in a dataset may inadvertently lead to over-clustering or under-clustering of the relevant column, while inaccurately resolving entities can distort the true distribution of data points” ([“icse2025-paper1580”, p. 1](zotero://select/library/items/H5PXLPAG)) ([pdf](zotero://open-pdf/library/items/TRUCG6BD?page=1&annotation=M7VN63CA)) #a28ae5
|
||
- - ““human-in-the-loop”” ([“icse2025-paper1580”, p. 1](zotero://select/library/items/H5PXLPAG)) ([pdf](zotero://open-pdf/library/items/TRUCG6BD?page=1&annotation=G2INMR9H)) #5fb236
|
||
- - “LLM can effectively find its way around a spreadsheet and how it can support end-users in fulfilling their free-form data preprocessing requests.” ([“icse2025-paper1580”, p. 1](zotero://select/library/items/H5PXLPAG)) ([pdf](zotero://open-pdf/library/items/TRUCG6BD?page=1&annotation=6VHXRVIA)) #a28ae5
|
||
- - “TradeSweep” ([“icse2025-paper1580”, p. 1](zotero://select/library/items/H5PXLPAG)) ([pdf](zotero://open-pdf/library/items/TRUCG6BD?page=1&annotation=XBJLY9BR)) #5fb236
|
||
- - “generates code proposals that can be composed and applied to targeted datasets, achieving high performance.” ([“icse2025-paper1580”, p. 1](zotero://select/library/items/H5PXLPAG)) ([pdf](zotero://open-pdf/library/items/TRUCG6BD?page=1&annotation=ENZ74PFZ)) #ffd400
|
||
- *How to deal with the specificity of the spreadsheet environment in use?*
|
||
- - “Similar to how Retrieval-Augmented Generation (RAG) retrieves context to answer users’ queries, we demonstrate how elements from a code library can be stored and retrieved to compose complex pipelines.” ([“icse2025-paper1580”, p. 1](zotero://select/library/items/H5PXLPAG)) ([pdf](zotero://open-pdf/library/items/TRUCG6BD?page=1&annotation=TUAV3QSL)) #5fb236
|
||
- - “English conversations to understand and respond to users’ requests with Python code encompassing preprocessing functions, supporting three main capabilities:” ([“icse2025-paper1580”, p. 1](zotero://select/library/items/H5PXLPAG)) ([pdf](zotero://open-pdf/library/items/TRUCG6BD?page=1&annotation=GUXQILBA)) #ffd400
|
||
- *Python? So I assume you are supporting CSV files... details of the environment in use should be given.*
|
||
- - “The proposed code is automatically tailored to the target data, including accurate column names and suitable algorithms.” ([“icse2025-paper1580”, p. 2](zotero://select/library/items/H5PXLPAG)) ([pdf](zotero://open-pdf/library/items/TRUCG6BD?page=2&annotation=SSWXZ5W2)) #ffd400
|
||
- *see my previous comment.*
|
||
- - “efficient” ([“icse2025-paper1580”, p. 2](zotero://select/library/items/H5PXLPAG)) ([pdf](zotero://open-pdf/library/items/TRUCG6BD?page=2&annotation=YDXSH8CZ)) #ffd400
|
||
- *how to support this?*
|
||
- - “This line of work has facilitated the convergence between information retrieval and information generation” ([“icse2025-paper1580”, p. 2](zotero://select/library/items/H5PXLPAG)) ([pdf](zotero://open-pdf/library/items/TRUCG6BD?page=2&annotation=FTPWYQNA)) #5fb236
|
||
- - “support developers rather than end-users” ([“icse2025-paper1580”, p. 2](zotero://select/library/items/H5PXLPAG)) ([pdf](zotero://open-pdf/library/items/TRUCG6BD?page=2&annotation=74LKIQNX)) #ffd400
|
||
- *TO what extent users are different from developers in the context of this work?*
|
||
- - “SKCoder” ([“icse2025-paper1580”, p. 2](zotero://select/library/items/H5PXLPAG)) ([pdf](zotero://open-pdf/library/items/TRUCG6BD?page=2&annotation=LVWJD5XZ)) #5fb236
|
||
- - “they still encounter difficulties in handling algorithmic complexities and often require human verification [42, 43] to ensure correct and/or efficient code” ([“icse2025-paper1580”, p. 2](zotero://select/library/items/H5PXLPAG)) ([pdf](zotero://open-pdf/library/items/TRUCG6BD?page=2&annotation=L7TK3GDQ)) #a28ae5
|
||
- - “either CSV or Excel format and submit a preprocessing request” ([“icse2025-paper1580”, p. 2](zotero://select/library/items/H5PXLPAG)) ([pdf](zotero://open-pdf/library/items/TRUCG6BD?page=2&annotation=5IQS9WMS)) #ffd400
|
||
- *This detail should be also given in the introduction just to make clear such a technical constraint of the proposed approach.*
|
||
- - “column names c1, c2, ..., cm” ([“icse2025-paper1580”, p. 2](zotero://select/library/items/H5PXLPAG)) ([pdf](zotero://open-pdf/library/items/TRUCG6BD?page=2&annotation=4PJTB7ZZ)) #5fb236
|
||
- - “FEATURE COMPARISON” ([“icse2025-paper1580”, p. 3](zotero://select/library/items/H5PXLPAG)) ([pdf](zotero://open-pdf/library/items/TRUCG6BD?page=3&annotation=AFHNTKQG)) #ffd400
|
||
- *How those features have been elicited/selected among those of the analyzed systems?*
|
||
- - “In practical applications, it is often necessary to perform data cleaning (e.g., handling inconsistent formats, outliers, missing values) and preprocessing (e.g., normalization, label encoding) on the dataset D before training a machine learning model.” ([“icse2025-paper1580”, p. 3](zotero://select/library/items/H5PXLPAG)) ([pdf](zotero://open-pdf/library/items/TRUCG6BD?page=3&annotation=YVD2K7DR)) #5fb236
|
||
- - “generate Python code fˆ and a processed dataset D” ([“icse2025-paper1580”, p. 3](zotero://select/library/items/H5PXLPAG)) ([pdf](zotero://open-pdf/library/items/TRUCG6BD?page=3&annotation=7KJKNML8)) #a28ae5
|
||
- - “prompting curriculum” ([“icse2025-paper1580”, p. 3](zotero://select/library/items/H5PXLPAG)) ([pdf](zotero://open-pdf/library/items/TRUCG6BD?page=3&annotation=FDBPPNCB)) #ffd400
|
||
- *What's that?*
|
||
- - “TradeSweep operates without requiring programming expertise from the user, as the specific columns A necessitating modification are automatically determined by the model, and the entire process from input description to execution outcome is managed by the system.” ([“icse2025-paper1580”, p. 3](zotero://select/library/items/H5PXLPAG)) ([pdf](zotero://open-pdf/library/items/TRUCG6BD?page=3&annotation=LBZ8GFAY)) #ffd400
|
||
- *I'm not sure about that.... Python code is generated and I doubt that users without programming expertise can work with it.*
|
||
- - “vulnerability to hallucinations.” ([“icse2025-paper1580”, p. 3](zotero://select/library/items/H5PXLPAG)) ([pdf](zotero://open-pdf/library/items/TRUCG6BD?page=3&annotation=6UAUC23C)) #a28ae5
|
||
- - “library L” ([“icse2025-paper1580”, p. 3](zotero://select/library/items/H5PXLPAG)) ([pdf](zotero://open-pdf/library/items/TRUCG6BD?page=3&annotation=5FIR6AYN)) #ffd400
|
||
- *What are the functions that are part of this library as default? I guess it is not empty, isn't it?*
|
||
- - “This allows the library L to continually learn and provide more accurate and advanced reference functions in the future.” ([“icse2025-paper1580”, p. 3](zotero://select/library/items/H5PXLPAG)) ([pdf](zotero://open-pdf/library/items/TRUCG6BD?page=3&annotation=IA3KAH3A)) #5fb236
|
||
- *That's important.*
|
||
- - “This makes the tool accessible for nonexpert programmers” ([“icse2025-paper1580”, p. 4](zotero://select/library/items/H5PXLPAG)) ([pdf](zotero://open-pdf/library/items/TRUCG6BD?page=4&annotation=MQCZUWYV)) #ffd400
|
||
- *AS previously mentioned, I'm not fully convinced about this aspect.*
|
||
- - “The library comprises code functions commonly applied in various preprocessing tasks. Along with these functions, data information is also provided to the LLM to aid its understanding of the dataset structure” ([“icse2025-paper1580”, p. 4](zotero://select/library/items/H5PXLPAG)) ([pdf](zotero://open-pdf/library/items/TRUCG6BD?page=4&annotation=NIDC58GT)) #ffd400
|
||
- *Have you checked the difference in accuracy with and without the prompt augmentation?*
|
||
- - “Instead of presenting all code functions in the prompt, we utilize information retrieval (IR) techniques to perform a more precise selection on code functions, providing the LLM with the most relevant and useful codes.” ([“icse2025-paper1580”, p. 4](zotero://select/library/items/H5PXLPAG)) ([pdf](zotero://open-pdf/library/items/TRUCG6BD?page=4&annotation=PSA3JIYQ)) #5fb236
|
||
- - “This process not only effectively shortens the prompt length and reduces the LLM’s response time but also helps the LLM focus on functions most pertinent to fulfilling the user’s request” ([“icse2025-paper1580”, p. 4](zotero://select/library/items/H5PXLPAG)) ([pdf](zotero://open-pdf/library/items/TRUCG6BD?page=4&annotation=A4VDCWAV)) #ffd400
|
||
- *It is important to show that such a step is really needed. What happens if top-k relevant codes are not retrieved and given to the next step (prompt construction)?*
|
||
- - “candidate codes” ([“icse2025-paper1580”, p. 4](zotero://select/library/items/H5PXLPAG)) ([pdf](zotero://open-pdf/library/items/TRUCG6BD?page=4&annotation=X5XSDL38)) #ffd400
|
||
- *Can this represent a source of bias, or in general bring the LLM to the wrong path during the inference phase? It is necessary to show what I asked in my previous point!*
|
||
- - “If the LLM does not find any candidate code that aligns with the user’s request, it generates a novel code function” ([“icse2025-paper1580”, p. 4](zotero://select/library/items/H5PXLPAG)) ([pdf](zotero://open-pdf/library/items/TRUCG6BD?page=4&annotation=85GBQYE4)) #a28ae5
|
||
- - “Incorrect Code Proposal Format:” ([“icse2025-paper1580”, p. 4](zotero://select/library/items/H5PXLPAG)) ([pdf](zotero://open-pdf/library/items/TRUCG6BD?page=4&annotation=YZYC3WJQ)) #5fb236
|
||
- - “Execution Error:” ([“icse2025-paper1580”, p. 4](zotero://select/library/items/H5PXLPAG)) ([pdf](zotero://open-pdf/library/items/TRUCG6BD?page=4&annotation=6VB5CN45)) #5fb236
|
||
- - “User Feedback:” ([“icse2025-paper1580”, p. 4](zotero://select/library/items/H5PXLPAG)) ([pdf](zotero://open-pdf/library/items/TRUCG6BD?page=4&annotation=J5QEJ7KA)) #5fb236
|
||
- - “they can request code revisions by providing feedback describing necessary fixes to the LLM” ([“icse2025-paper1580”, p. 4](zotero://select/library/items/H5PXLPAG)) ([pdf](zotero://open-pdf/library/items/TRUCG6BD?page=4&annotation=4LDXCZGN)) #5fb236
|
||
- - “Fig. 4.” ([“icse2025-paper1580”, p. 4](zotero://select/library/items/H5PXLPAG)) ([pdf](zotero://open-pdf/library/items/TRUCG6BD?page=4&annotation=MDG3DJC8)) #ffd400
|
||
- *I agree that the Python code shown in Fig. 4 is easy to understand. However, it is not possible to sustain that users without any programming experience can understand what that source is supposed to do. I suggest to smooth statements supporting such claims. Moreover, I don't see in the code any reference to the "Arrival Date" column as asked by the user.*
|
||
- - “it generates a novel code proposal. Since this newly generated code is not initially included in the library, we incorporate it back into the library to improve efficiency and accuracy for future code generations.” ([“icse2025-paper1580”, p. 5](zotero://select/library/items/H5PXLPAG)) ([pdf](zotero://open-pdf/library/items/TRUCG6BD?page=5&annotation=8XJN5Z6P)) #ffd400
|
||
- - “IV. EXPERIMENTS” ([“icse2025-paper1580”, p. 5](zotero://select/library/items/H5PXLPAG)) ([pdf](zotero://open-pdf/library/items/TRUCG6BD?page=5&annotation=TZ3QCPJU)) #ffd400
|
||
- *The section is missing an explicit description of the research questions that you wanted to answer by means of the performed experiments.*
|
||
- - “effectiveness of identifying shipments potentially circumventing economic sanctions, high tariffs, or engaging in suspicious activities largely depends on the quality of data initially cleaned and preprocessed.” ([“icse2025-paper1580”, p. 5](zotero://select/library/items/H5PXLPAG)) ([pdf](zotero://open-pdf/library/items/TRUCG6BD?page=5&annotation=J6WP8LSX)) #a28ae5
|
||
- - “import prohibitions,” ([“icse2025-paper1580”, p. 5](zotero://select/library/items/H5PXLPAG)) ([pdf](zotero://open-pdf/library/items/TRUCG6BD?page=5&annotation=EJLMX5E5)) #5fb236
|
||
- - “high tariff rates” ([“icse2025-paper1580”, p. 5](zotero://select/library/items/H5PXLPAG)) ([pdf](zotero://open-pdf/library/items/TRUCG6BD?page=5&annotation=G2LCGIA8)) #5fb236
|
||
- - “sanctions” ([“icse2025-paper1580”, p. 5](zotero://select/library/items/H5PXLPAG)) ([pdf](zotero://open-pdf/library/items/TRUCG6BD?page=5&annotation=K85JBZR9)) #5fb236
|
||
- - “Teak” ([“icse2025-paper1580”, p. 5](zotero://select/library/items/H5PXLPAG)) ([pdf](zotero://open-pdf/library/items/TRUCG6BD?page=5&annotation=5P87D6G3)) #5fb236
|
||
- - “Panjiva1” ([“icse2025-paper1580”, p. 5](zotero://select/library/items/H5PXLPAG)) ([pdf](zotero://open-pdf/library/items/TRUCG6BD?page=5&annotation=P8ZBDC2B)) #5fb236
|
||
- - “Grain” ([“icse2025-paper1580”, p. 5](zotero://select/library/items/H5PXLPAG)) ([pdf](zotero://open-pdf/library/items/TRUCG6BD?page=5&annotation=RHJV8WMQ)) #5fb236
|
||
- - “ExportGenius2” ([“icse2025-paper1580”, p. 5](zotero://select/library/items/H5PXLPAG)) ([pdf](zotero://open-pdf/library/items/TRUCG6BD?page=5&annotation=L5WGK8AX)) #5fb236
|
||
- - “Timber” ([“icse2025-paper1580”, p. 5](zotero://select/library/items/H5PXLPAG)) ([pdf](zotero://open-pdf/library/items/TRUCG6BD?page=5&annotation=3FJT8X2V)) #5fb236
|
||
- - “ExportGenius and ImportGenius.” ([“icse2025-paper1580”, p. 5](zotero://select/library/items/H5PXLPAG)) ([pdf](zotero://open-pdf/library/items/TRUCG6BD?page=5&annotation=NIUGGMTV)) #5fb236
|
||
- - “Consequently, data analysis becomes a laborious and time-consuming task for analysts, highlighting the urgency and importance of effective data preprocessing.” ([“icse2025-paper1580”, p. 5](zotero://select/library/items/H5PXLPAG)) ([pdf](zotero://open-pdf/library/items/TRUCG6BD?page=5&annotation=EYYIMWKV)) #5fb236
|
||
- - “Qdrant is employed to efficiently store and retrieve code functions based on its vector embeddings.” ([“icse2025-paper1580”, p. 5](zotero://select/library/items/H5PXLPAG)) ([pdf](zotero://open-pdf/library/items/TRUCG6BD?page=5&annotation=TFH5QJ34)) #5fb236
|
||
- - “CodeLlama-13b-Instruct” ([“icse2025-paper1580”, p. 5](zotero://select/library/items/H5PXLPAG)) ([pdf](zotero://open-pdf/library/items/TRUCG6BD?page=5&annotation=U3594BFC)) #5fb236
|
||
- - “12 widely recognized and commonly used data preprocessing functions.” ([“icse2025-paper1580”, p. 5](zotero://select/library/items/H5PXLPAG)) ([pdf](zotero://open-pdf/library/items/TRUCG6BD?page=5&annotation=S27E9627)) #5fb236
|
||
- - “we developed three baselines for comparison” ([“icse2025-paper1580”, p. 5](zotero://select/library/items/H5PXLPAG)) ([pdf](zotero://open-pdf/library/items/TRUCG6BD?page=5&annotation=4TXRZKA2)) #5fb236
|
||
- - “Baseline 1 (B1): State-of-the-Art (SOTA) Simulation Code Generation Using Only LLM.” ([“icse2025-paper1580”, p. 5](zotero://select/library/items/H5PXLPAG)) ([pdf](zotero://open-pdf/library/items/TRUCG6BD?page=5&annotation=D9CFCYM3)) #ffd400
|
||
- *User request should be added in the title*
|
||
- - “This setup evaluates the LLM’s capability to produce relevant code purely from the textual description provided by the user, which may involve significant effort and may result in less effective data cleaning outcomes.” ([“icse2025-paper1580”, p. 6](zotero://select/library/items/H5PXLPAG)) ([pdf](zotero://open-pdf/library/items/TRUCG6BD?page=6&annotation=UWPXEEXR)) #ffd400
|
||
- *Without giving as input the spreadsheet to clean?*
|
||
- - “we provided the LLM with a set of top-k candidate codes retrieved from the code library, along with the user’s request.” ([“icse2025-paper1580”, p. 6](zotero://select/library/items/H5PXLPAG)) ([pdf](zotero://open-pdf/library/items/TRUCG6BD?page=6&annotation=8RF3VHBR)) #a28ae5
|
||
- - “excludes specific data details” ([“icse2025-paper1580”, p. 6](zotero://select/library/items/H5PXLPAG)) ([pdf](zotero://open-pdf/library/items/TRUCG6BD?page=6&annotation=ZKTBTXRR)) #ffd400
|
||
- *What do you mean? What kind of data details are excluded?*
|
||
- - “Providing the Entire Code Library Without Descriptions.” ([“icse2025-paper1580”, p. 6](zotero://select/library/items/H5PXLPAG)) ([pdf](zotero://open-pdf/library/items/TRUCG6BD?page=6&annotation=VJTD43L9)) #ffd400
|
||
- *Add also the fact that user request is given (if this is the case, I think so).*
|
||
- - “DATA PREPROCESSING TASKS USED IN OUR EXPERIMENTS” ([“icse2025-paper1580”, p. 6](zotero://select/library/items/H5PXLPAG)) ([pdf](zotero://open-pdf/library/items/TRUCG6BD?page=6&annotation=XL4V2S7G)) #ffd400
|
||
- *It's not clear if the tasks to be executed to each dataset has been decided by the authors, or it was TradeSweep that decided them by looking at the datasets.*
|
||
- - “Only show records where trading country is US” ([“icse2025-paper1580”, p. 6](zotero://select/library/items/H5PXLPAG)) ([pdf](zotero://open-pdf/library/items/TRUCG6BD?page=6&annotation=WMLVNMMI)) #5fb236
|
||
- - “preprocessed and cleaned” ([“icse2025-paper1580”, p. 6](zotero://select/library/items/H5PXLPAG)) ([pdf](zotero://open-pdf/library/items/TRUCG6BD?page=6&annotation=Q48V982U)) #ffd400
|
||
- *The proposed approach does not seem to distinguish preprocess vs clean operations. If they are used interchangeably, this must be clarified.*
|
||
- - “During the data preprocessing operation, if TradeSweep is unable to identify a relevant function in the code library, it generates a novel code. This limitation causes TradeSweep to behave similarly to SOTA tools, relying solely on the LLM to generate code.” ([“icse2025-paper1580”, p. 10](zotero://select/library/items/H5PXLPAG)) ([pdf](zotero://open-pdf/library/items/TRUCG6BD?page=10&annotation=8LGWEL54)) #5fb236
|
||
- - “Our results demonstrate TradeSweep ’s effectiveness in practical data transformation scenarios.” ([“icse2025-paper1580”, p. 10](zotero://select/library/items/H5PXLPAG)) ([pdf](zotero://open-pdf/library/items/TRUCG6BD?page=10&annotation=S9NBQ5TD)) #5fb236
|
||
- - “The ablation study highlights that utilizing a code library and a vector database for information retrieval accelerates the code generation process.” ([“icse2025-paper1580”, p. 10](zotero://select/library/items/H5PXLPAG)) ([pdf](zotero://open-pdf/library/items/TRUCG6BD?page=10&annotation=WUCUUNYK)) #a28ae5
|
||
- ### [[Comments]]
|
||
- #.tabular
|
||
- - ### Paper summary
|
||
- The paper presents TradeSweep, an approach based on LLM that supports the cleaning and preprocessing of tabular datasets. The authors investigate how effectively LLMs can interpret natural language requests from users to generate Python code for tasks such as fixing formatting inconsistencies or handling missing values. The approach incorporates a growing code library of reusable Python functions that can be applied to preprocess the dataset at hand. User feedback is integrated into the system to evaluate the quality of the LLM-generated code and determine whether it should be added to the library. The paper demonstrates the system's effectiveness through experiments on trade datasets, illustrating how the approach can simplify data preprocessing tasks.
|
||
- - ### Strengths
|
||
- +
|
||
- + Novel usage of LLMs to preprocess tabular data.
|
||
- + Human-in-the-loop support to evaluate if generated functions are appropriate for the problem at hand and if they can be reused for future requests.
|
||
- + Experiments using real-world trade data.
|
||
- - ### Weaknesses
|
||
- -
|
||
- **-** While the system aims to be user-friendly for non-programmers, there are doubts about whether users without programming skills can effectively engage with the generated Python code.
|
||
- **-** The experimental section lacks clearly defined research questions, making it difficult to follow the specific objectives and outcomes of the experiments.
|
||
- - ### Detailed comments for authors
|
||
-
|
||
- Novelty: The integration of LLMs with a continuously growing code library is a novel approach that adds value to the field of automated data preprocessing. However, while Python is a popular language for data processing, the authors should provide a clear rationale for choosing it rather than leveraging existing tools or frameworks specifically designed to define and execute data workflows and pipelines (e.g., Apache Airflow, Orange, etc.).
|
||
- Rigor: The description of the proposed approach is well-structured. However, the absence of clearly stated research questions in the experimental section weakens the quality of the whole paper.
|
||
- Relevance: The paper addresses a relevant problem in the automation of data preprocessing. Enhancing the approach to target users with low programming skills is an important direction that needs further investigation.
|
||
- Verifiability & Transparency: The properly presents the system's architecture and experimental design. A GitHub repository containing the implementation of the proposed tool available at [anonymousrabbit111222/TradeSweep (github.com)](https://github.com/anonymousrabbit111222/TradeSweep) is also provided.
|
||
- Presentation: The paper is generally well-written, but some claims, especially regarding the system's accessibility for non-programmers, may be exaggerated. Additionally, it is essential to provide early details about the technical requirements of the approach (e.g., it is based on Python, it is designed to handle CSV files, etc.).
|
||
- #### Detailed Comments:
|
||
- 1. The paper frequently discusses data cleaning and preprocessing tasks but does not always clearly distinguish between these two concepts. Clarifying this distinction would improve the presentation of the system's capabilities. If these concepts are used interchangeably, this must be clarified.
|
||
- 2. The claim that non-programmers can use the system effectively might be too optimistic. The paper would benefit from including user studies or more concrete evidence supporting this claim.
|
||
- 3. The feature comparison given in Table 1 provides valuable insights, but it is unclear how the specific features were chosen. Providing more details on the feature selection process would enhance the credibility of the comparison.
|
||
- 4. While the experiments are well-executed, the absence of clearly articulated research questions makes it challenging to fully understand the goals and outcomes of the experiments. Including explicit research questions would provide a stronger foundation for the results.
|
||
- 5. "User's Request" should be included in the title of Baseline 1 and Baseline 3 as done for Baseline 2 (page 5 and page 6)
|
||
- 6. The description of Baseline 1 on page 6 is unclear. This setup only involves producing code from textual descriptions without providing any information about the input data. The experiments should better explain the relevance of such a setup.
|
||
- 7. Concerning Baseline 2, "this baseline excludes specific data details from the prompt." it is necessary to give some information about the data details that are neglected.
|
||
- 8. By looking at Table III, it's unclear if the tasks to be executed for each dataset have been decided by the authors or if TradeSweep decided them by analyzing the datasets or by taking input from some user requests.
|
||
- ### [[REVIEWS/Notes]]
|
||
- ### YELLOW CONCERNS
|
||
background-color:: yellow
|
||
- {{query (and [[ffd400]] [[icse2025-paper1580]] )}}
|
||
collapsed:: true
|
||
- ### ❓️Questions
|
||
- {{query (and [[question]] [[icse2025-paper1580]] )[[question]]}}
|
||
query-table:: true
|
||
query-properties:: [:block] |