logseq/pages/@icse2025-paper1580.md

tags:: #zotero title:: @icse2025-paper1580 item-type:: document original-title:: icse2025-paper1580 links:: Local library, Web library

• Attachments

  • PDF {{zotero-imported-file TRUCG6BD, "icse2025-paper1580.pdf"}}

• Notes

  • Annotazioni

    (29/9/2024, 19:18:59)

    • “data cleaning prior to analysis and evaluation” (“icse2025-paper1580”, p. 1) #5fb236

    • “whether an LLM can find its way around a spreadsheet and how to support end-users in taking their free-form data processing requests to fruition.” (“icse2025-paper1580”, p. 1) #a28ae5

    • “we can retrieve elements from a code library to compose data preprocessing pipelines” (“icse2025-paper1580”, p. 1) #a28ae5

    • “LLMs can function as coding assistants, enabling users with limited proficiency to produce accurate and executable code to accomplish their tasks” (“icse2025-paper1580”, p. 1) #5fb236

    • “processing tabular data, e.g., in the form of spreadsheets” (“icse2025-paper1580”, p. 1) #a28ae5

    • “lack of standardization, necessitating the creation of highly specialized pipeline” (“icse2025-paper1580”, p. 1) #5fb236

    • “correcting typographical errors in a dataset may inadvertently lead to over-clustering or under-clustering of the relevant column, while inaccurately resolving entities can distort the true distribution of data points” (“icse2025-paper1580”, p. 1) #a28ae5

    • ““human-in-the-loop”” (“icse2025-paper1580”, p. 1) #5fb236

    • “LLM can effectively find its way around a spreadsheet and how it can support end-users in fulfilling their free-form data preprocessing requests.” (“icse2025-paper1580”, p. 1) #a28ae5

    • “TradeSweep” (“icse2025-paper1580”, p. 1) #5fb236

    • “generates code proposals that can be composed and applied to targeted datasets, achieving high performance.” (“icse2025-paper1580”, p. 1) #ffd400 How to deal with the specificity of the spreadsheet environment in use?

    • “Similar to how Retrieval-Augmented Generation (RAG) retrieves context to answer users’ queries, we demonstrate how elements from a code library can be stored and retrieved to compose complex pipelines.” (“icse2025-paper1580”, p. 1) #5fb236

    • “English conversations to understand and respond to users’ requests with Python code encompassing preprocessing functions, supporting three main capabilities:” (“icse2025-paper1580”, p. 1) #ffd400 Python? So I assume you are supporting CSV files... details of the environment in use should be given.

    • “The proposed code is automatically tailored to the target data, including accurate column names and suitable algorithms.” (“icse2025-paper1580”, p. 2) #ffd400 see my previous comment.

    • “efficient” (“icse2025-paper1580”, p. 2) #ffd400 how to support this?

    • “This line of work has facilitated the convergence between information retrieval and information generation” (“icse2025-paper1580”, p. 2) #5fb236

    • “support developers rather than end-users” (“icse2025-paper1580”, p. 2) #ffd400 TO what extent users are different from developers in the context of this work?

    • “SKCoder” (“icse2025-paper1580”, p. 2) #5fb236

    • “they still encounter difficulties in handling algorithmic complexities and often require human verification [42, 43] to ensure correct and/or efficient code” (“icse2025-paper1580”, p. 2) #a28ae5

    • “either CSV or Excel format and submit a preprocessing request” (“icse2025-paper1580”, p. 2) #ffd400 This detail should be also given in the introduction just to make clear such a technical constraint of the proposed approach.

    • “column names c1, c2, ..., cm” (“icse2025-paper1580”, p. 2) #5fb236

    • “FEATURE COMPARISON” (“icse2025-paper1580”, p. 3) #ffd400 How those features have been elicited/selected among those of the analyzed systems?

    • “In practical applications, it is often necessary to perform data cleaning (e.g., handling inconsistent formats, outliers, missing values) and preprocessing (e.g., normalization, label encoding) on the dataset D before training a machine learning model.” (“icse2025-paper1580”, p. 3) #5fb236

    • “generate Python code fˆ and a processed dataset D” (“icse2025-paper1580”, p. 3) #a28ae5

    • “prompting curriculum” (“icse2025-paper1580”, p. 3) #ffd400 What's that?

    • “TradeSweep operates without requiring programming expertise from the user, as the specific columns A necessitating modification are automatically determined by the model, and the entire process from input description to execution outcome is managed by the system.” (“icse2025-paper1580”, p. 3) #ffd400 I'm not sure about that.... Python code is generated and I doubt that users without programming expertise can work with it.

    • “vulnerability to hallucinations.” (“icse2025-paper1580”, p. 3) #a28ae5

    • “library L” (“icse2025-paper1580”, p. 3) #ffd400 What are the functions that are part of this library as default? I guess it is not empty, isn't it?

    • “This allows the library L to continually learn and provide more accurate and advanced reference functions in the future.” (“icse2025-paper1580”, p. 3) #5fb236 That's important.

    • “This makes the tool accessible for nonexpert programmers” (“icse2025-paper1580”, p. 4) #ffd400 AS previously mentioned, I'm not fully convinced about this aspect.

    • “The library comprises code functions commonly applied in various preprocessing tasks. Along with these functions, data information is also provided to the LLM to aid its understanding of the dataset structure” (“icse2025-paper1580”, p. 4) #ffd400 Have you checked the difference in accuracy with and without the prompt augmentation?

    • “Instead of presenting all code functions in the prompt, we utilize information retrieval (IR) techniques to perform a more precise selection on code functions, providing the LLM with the most relevant and useful codes.” (“icse2025-paper1580”, p. 4) #5fb236

    • “This process not only effectively shortens the prompt length and reduces the LLM’s response time but also helps the LLM focus on functions most pertinent to fulfilling the user’s request” (“icse2025-paper1580”, p. 4) #ffd400 It is important to show that such a step is really needed. What happens if top-k relevant codes are not retrieved and given to the next step (prompt construction)?

    • “candidate codes” (“icse2025-paper1580”, p. 4) #ffd400 Can this represent a source of bias, or in general bring the LLM to the wrong path during the inference phase? It is necessary to show what I asked in my previous point!

    • “If the LLM does not find any candidate code that aligns with the user’s request, it generates a novel code function” (“icse2025-paper1580”, p. 4) #a28ae5

    • “Incorrect Code Proposal Format:” (“icse2025-paper1580”, p. 4) #5fb236

    • “Execution Error:” (“icse2025-paper1580”, p. 4) #5fb236

    • “User Feedback:” (“icse2025-paper1580”, p. 4) #5fb236

    • “they can request code revisions by providing feedback describing necessary fixes to the LLM” (“icse2025-paper1580”, p. 4) #5fb236

    • “Fig. 4.” (“icse2025-paper1580”, p. 4) #ffd400 I agree that the Python code shown in Fig. 4 is easy to understand. However, it is not possible to sustain that users without any programming experience can understand what that source is supposed to do. I suggest to smooth statements supporting such claims. Moreover, I don't see in the code any reference to the "Arrival Date" column as asked by the user.

    • “it generates a novel code proposal. Since this newly generated code is not initially included in the library, we incorporate it back into the library to improve efficiency and accuracy for future code generations.” (“icse2025-paper1580”, p. 5) #ffd400

    • “IV. EXPERIMENTS” (“icse2025-paper1580”, p. 5) #ffd400 The section is missing an explicit description of the research questions that you wanted to answer by means of the performed experiments.

    • “effectiveness of identifying shipments potentially circumventing economic sanctions, high tariffs, or engaging in suspicious activities largely depends on the quality of data initially cleaned and preprocessed.” (“icse2025-paper1580”, p. 5) #a28ae5

    • “import prohibitions,” (“icse2025-paper1580”, p. 5) #5fb236

    • “high tariff rates” (“icse2025-paper1580”, p. 5) #5fb236

    • “sanctions” (“icse2025-paper1580”, p. 5) #5fb236

    • “Teak” (“icse2025-paper1580”, p. 5) #5fb236

    • “Panjiva1” (“icse2025-paper1580”, p. 5) #5fb236

    • “Grain” (“icse2025-paper1580”, p. 5) #5fb236

    • “ExportGenius2” (“icse2025-paper1580”, p. 5) #5fb236

    • “Timber” (“icse2025-paper1580”, p. 5) #5fb236

    • “ExportGenius and ImportGenius.” (“icse2025-paper1580”, p. 5) #5fb236

    • “Consequently, data analysis becomes a laborious and time-consuming task for analysts, highlighting the urgency and importance of effective data preprocessing.” (“icse2025-paper1580”, p. 5) #5fb236

    • “Qdrant is employed to efficiently store and retrieve code functions based on its vector embeddings.” (“icse2025-paper1580”, p. 5) #5fb236

    • “CodeLlama-13b-Instruct” (“icse2025-paper1580”, p. 5) #5fb236

    • “12 widely recognized and commonly used data preprocessing functions.” (“icse2025-paper1580”, p. 5) #5fb236

    • “we developed three baselines for comparison” (“icse2025-paper1580”, p. 5) #5fb236

    • “Baseline 1 (B1): State-of-the-Art (SOTA) Simulation Code Generation Using Only LLM.” (“icse2025-paper1580”, p. 5) #ffd400 User request should be added in the title

    • “This setup evaluates the LLM’s capability to produce relevant code purely from the textual description provided by the user, which may involve significant effort and may result in less effective data cleaning outcomes.” (“icse2025-paper1580”, p. 6) #ffd400 Without giving as input the spreadsheet to clean?

    • “we provided the LLM with a set of top-k candidate codes retrieved from the code library, along with the user’s request.” (“icse2025-paper1580”, p. 6) #a28ae5

    • “excludes specific data details” (“icse2025-paper1580”, p. 6) #ffd400 What do you mean? What kind of data details are excluded?

    • “Providing the Entire Code Library Without Descriptions.” (“icse2025-paper1580”, p. 6) #ffd400 Add also the fact that user request is given (if this is  the case, I think so).

    • “DATA PREPROCESSING TASKS USED IN OUR EXPERIMENTS” (“icse2025-paper1580”, p. 6) #ffd400 It's not clear if the tasks to be executed to each dataset has been decided by the authors, or it was TradeSweep that decided them by looking at the datasets.

    • “Only show records where trading country is US” (“icse2025-paper1580”, p. 6) #5fb236

    • “preprocessed and cleaned” (“icse2025-paper1580”, p. 6) #ffd400 The proposed approach does not seem to distinguish preprocess vs clean operations. If they are used interchangeably, this must be clarified.

    • “During the data preprocessing operation, if TradeSweep is unable to identify a relevant function in the code library, it generates a novel code. This limitation causes TradeSweep to behave similarly to SOTA tools, relying solely on the LLM to generate code.” (“icse2025-paper1580”, p. 10) #5fb236

    • “Our results demonstrate TradeSweep ’s effectiveness in practical data transformation scenarios.” (“icse2025-paper1580”, p. 10) #5fb236

    • “The ablation study highlights that utilizing a code library and a vector database for information retrieval accelerates the code generation process.” (“icse2025-paper1580”, p. 10) #a28ae5