14 KiB
14 KiB
tags:: #zotero title:: @TSE-2024-08-0441_Proof_hi item-type:: document original-title:: TSE-2024-08-0441_Proof_hi links:: Local library, Web library
-
Attachments
- PDF {{zotero-imported-file 2Q94I35B, "TSE-2024-08-0441_Proof_hi.pdf"}}
-
Notes
collapsed:: true-
Annotazioni
(27/10/2024, 11:15:25)
- “The inclusion of comprehensive setup instructions in ReadMe files is crucial for newcomers to effectively engage with these projects.” (“TSE-2024-08-0441_Proof_hi”, p. 1) #5fb236
-
- “developers might overlook this aspect, posing challenges for newcomers in setting up JWAs.” (“TSE-2024-08-0441_Proof_hi”, p. 1) #a28ae5
* *
- “PAL4JavaWeb, a novel method designed to simplify the process of setting up JWA projects by automatically generating setup steps from the source files.” (“TSE-2024-08-0441_Proof_hi”, p. 1) #a28ae5
*By reading this, it does not seem to be a so disruptive contribution. *
- “utilizing the obtained segments, we leverage the LLM to recognize key configuration steps and generate setup instructions accordingly.” (“TSE-2024-08-0441_Proof_hi”, p. 1) #5fb236
* *
- “baseline method by 48.57%” (“TSE-2024-08-0441_Proof_hi”, p. 1) #ffd400
*Check what is the baseline method that has been considered. *
- “This resulted in positive responses, with 10 PRs being merged, highlighting the practical usefulness of PAL4JavaWeb in simplifying the onboarding process for newcomers” (“TSE-2024-08-0441_Proof_hi”, p. 1) #ffd400
*Check what happened with the remaining 3. *
- “online shopping, Internet banking” (“TSE-2024-08-0441_Proof_hi”, p. 1) #5fb236
* *
- “developers need to install and configure some requirements such as the framework, the dependency, the database, etc, both in front-end and back-end, as shown in the Fig. 1.” (“TSE-2024-08-0441_Proof_hi”, p. 1) #5fb236
* *
- “It is difficult for developers, especially newcomers to set up a project successfully with only the source code because they have to get the exact steps from the unfamiliar source code.” (“TSE-2024-08-0441_Proof_hi”, p. 1) #5fb236
* *
- “We selected 113,000 JWA projects from GitHub for research, among which 82.0% of the projects contain issues related to setup steps, totally 140,900 issues. Among these issues, 31.2% are still in open status, indicating they are not solved.” (“TSE-2024-08-0441_Proof_hi”, p. 2) #5fb236
* *
- “we randomly set up” (“TSE-2024-08-0441_Proof_hi”, p. 2) #ffd400
*what do you mean? *
- “apply Machine Learning (ML) techniques [10] to evaluate the ReadMe file, then send evaluation results to project owners, reminding them to improve the setup section.” (“TSE-2024-08-0441_Proof_hi”, p. 2) #5fb236
* *
- “Some studies [11, 12] constructed relational databases containing information about all the requirements for the entire environment,” (“TSE-2024-08-0441_Proof_hi”, p. 2) #5fb236
* *
- “The LLMs trained on billions of samples from the Internet exhibit promising performance across various tasks and domains, including machine translation [13], information retrieval [14], bug detection [15], mathematical reasoning [16], and so o” (“TSE-2024-08-0441_Proof_hi”, p. 2) #5fb236
* *
- “natural language generation, and roleplaying, we designate the LLMs as expert developers capable of generating setup steps” (“TSE-2024-08-0441_Proof_hi”, p. 2) #5fb236
* *
- “1) How to extract setup-related information efficiently from diverse source codes to address LLMs token limitations” (“TSE-2024-08-0441_Proof_hi”, p. 2) #5fb236
* *
- “In this paper, we introduce PAL4JavaWeb (Program Analysis and LLM for JavaWeb), a method that automatically generates setup steps using program analysis and LLMs.” (“TSE-2024-08-0441_Proof_hi”, p. 2) #a28ae5
* *
- “first challenge,” (“TSE-2024-08-0441_Proof_hi”, p. 2) #2ea8e5
* *
- “next two challenges” (“TSE-2024-08-0441_Proof_hi”, p. 2) #2ea8e5
* *
- “we provide LLMs with the chain-of-thought reasoning from developers, guiding it in recognizing the frameworks, databases and configuration information contained in the data obtained in the first phase.” (“TSE-2024-08-0441_Proof_hi”, p. 2) #5fb236
* *
- “all recognized results are consolidated into a JSON object with key-value pairs.” (“TSE-2024-08-0441_Proof_hi”, p. 2) #5fb236
* *
- “35 JWA projects from GitHu” (“TSE-2024-08-0441_Proof_hi”, p. 2) #5fb236
* *
- “first work that combines program analysis and LLMs for the generation of JWA setup steps” (“TSE-2024-08-0441_Proof_hi”, p. 2) #ffd400
*What does it mean integration of program analysis with LLMs? To be checked. *
- “PAL4JavaWeb, leveraging static program analysis and prompt engineering to harness LLM’s knowledge for setup steps generation.” (“TSE-2024-08-0441_Proof_hi”, p. 2) #5fb236
* *
- “METHOD” (“TSE-2024-08-0441_Proof_hi”, p. 2) #ffd400
*Before this section, it is necessary to motivate the work. It is necessary to convince the reader that the problem is worth exploring by giving evidence of the difficulties that currently developers encounter when developing and configuring JWA. What's the state of practice? What are existing technologies that are employed to support related challenges? *
- “Extraction Module,” (“TSE-2024-08-0441_Proof_hi”, p. 2) #2ea8e5
* *
- “Recognition Module” (“TSE-2024-08-0441_Proof_hi”, p. 2) #2ea8e5
* *
- “Generation Module” (“TSE-2024-08-0441_Proof_hi”, p. 2) #2ea8e5
* *
- “front-end framework” (“TSE-2024-08-0441_Proof_hi”, p. 2) #2ea8e5
* *
- “dependency for the front-end” (“TSE-2024-08-0441_Proof_hi”, p. 2) #2ea8e5
* *
- “As shown in Table I.” (“TSE-2024-08-0441_Proof_hi”, p. 2) #ff6666
*should it be connected with the subsequent paragraph? *
- “we need to extract dependency file to determine the dependencies the project uses.” (“TSE-2024-08-0441_Proof_hi”, p. 2) #5fb236
* *
- “TABLE I” (“TSE-2024-08-0441_Proof_hi”, p. 3) #ffd400
*The paper is missing the presentation of technical details on the execution environment of the proposed process. How users are supposed to interact with it? IS there any support for the specification of prompts that authors have devised? Is it only based on the diligence of the user the right usage of the defined prompt patterns? *
- “we need to analyze the POM.XML according to the predefined rules.” (“TSE-2024-08-0441_Proof_hi”, p. 3) #5fb236
* *
- “Front-end prompt contains 5 representative examples and Back-end prompt contains 4 examples. Each example includes code snippets, Role with Task, Chain-of-Thought (CoT), and example task output.” (“TSE-2024-08-0441_Proof_hi”, p. 3) #5fb236
* *
- “as JSONformat data.” (“TSE-2024-08-0441_Proof_hi”, p. 3) #5fb236
* *
- “to determine which framework is used, we need to obtain characteristic files from the project.” (“TSE-2024-08-0441_Proof_hi”, p. 3) #ffd400
*Do you have a kind of catalogue of framework that you already support? *
- “It is challenging to confirm the front-end framework by the characteristic files through rule enumeration for two main reasons.” (“TSE-2024-08-0441_Proof_hi”, p. 4) #ffd400
*Right, this can represent a bias, right? *
- “utilize” (“TSE-2024-08-0441_Proof_hi”, p. 4) #ff6666
* *
- “or dependencies, since the location of these dependencies is not fixed, we need to match them throughout the folder. Firstly, we collected 4,387 common front-end dependencies from open source front-end dependency search sites, including BootCDN and Google Hosted Libraries, to match the dependencies used in JWA projects. Most of the dependencies in JWA projects come from this collection, which meets the matching requirements of existing JWA projects. Then, we extract files from the analyzed project that match those in the set of common front-end dependencies.” (“TSE-2024-08-0441_Proof_hi”, p. 4) #ffd400
*???? *
- “2) Extraction of back-end-Related code snippets” (“TSE-2024-08-0441_Proof_hi”, p. 4) #2ea8e5
* *
- “back-end configuration information.” (“TSE-2024-08-0441_Proof_hi”, p. 4) #5fb236
* *
- “application.yaml. In this file, developers use a series of key-value pairs for configuration. While the key names sometimes vary for each project, their format is similar.” (“TSE-2024-08-0441_Proof_hi”, p. 4) #5fb236
* *
- “[’Vue.js’]” (“TSE-2024-08-0441_Proof_hi”, p. 5) #ffd400
*This example is not clear. In particular, it is not clear the usage that one should do with the string ['Vue.js'] *
- “We design one prompt for the frontend and two prompts for the back-end.” (“TSE-2024-08-0441_Proof_hi”, p. 5) #5fb236
* *
- “chain-of-thought reasoning as inference” (“TSE-2024-08-0441_Proof_hi”, p. 5) #5fb236
* *
- “the example configuration information as output” (“TSE-2024-08-0441_Proof_hi”, p. 5) #5fb236
* *
- “In-context Learning” (“TSE-2024-08-0441_Proof_hi”, p. 5) #2ea8e5
* *
- “three” (“TSE-2024-08-0441_Proof_hi”, p. 5) #a28ae5
* *
- “65” (“TSE-2024-08-0441_Proof_hi”, p. 5) #ffd400
*Check if it has been said how these projects have been selected. *
- “9 examples they collectively deemed representative, some of the examples is shown in Table II and III.” (“TSE-2024-08-0441_Proof_hi”, p. 5) #ffd400
*9 examples to cover all the possible cases that the LLM should manage/able to recognize? *
- “Chain-of-thought reasoning” (“TSE-2024-08-0441_Proof_hi”, p. 5) #2ea8e5
* *
- “10 hours of experimentation” (“TSE-2024-08-0441_Proof_hi”, p. 5) #ffd400
*Which LLM has been considered? Have you experimented with more than one to investigate the generalizability of the proposed prompt? *
- “1) Incontext-learning” (“TSE-2024-08-0441_Proof_hi”, p. 6) #2ea8e5
* *
- “The selection of examples also involved a meeting with the same three professional developers who examined our dataset of open-source projects as before” (“TSE-2024-08-0441_Proof_hi”, p. 6) #5fb236
* *
- “JWA setup steps?” (“TSE-2024-08-0441_Proof_hi”, p. 7) #ffd400
*Is it really difficult? See my comments on the early sections of the paper. *
- “Data Preparation” (“TSE-2024-08-0441_Proof_hi”, p. 7) #ffd400
*Can you give more details on the dataset creation by referring to the typical train,test,validation phases? Currently, it is not clear if there existis some overlap between the selected JWA projects and those used as discussed in the previous section. *
- “,” (“TSE-2024-08-0441_Proof_hi”, p. 7) #ff6666
* *
- “i.e.,C” (“TSE-2024-08-0441_Proof_hi”, p. 7) #ff6666
*many typos and grammatical mistakes *
- “Front-end and back-end separation in an application indicates that the front-end and back-end components are independently developed [3].” (“TSE-2024-08-0441_Proof_hi”, p. 7) #5fb236
* *
- “KeyWord Search (KWS).” (“TSE-2024-08-0441_Proof_hi”, p. 7) #ffd400
*This is not clear. The ground true construction is critical and there are no discussions on the possible alternatives to construct it. Since the goal is generating the configuration specifications for the used framework, the authors could have compared what they are able to generate with the real files belonging to the considered projects. Again, the process for the ground true data needs to be clarified. *
- “matche” (“TSE-2024-08-0441_Proof_hi”, p. 8) #ff6666
* *
- “From the Table V, it can be observed that PAL4JavaWeb achieves a success rate of 91.43%, generating complete setup steps of 32 JWAs.” (“TSE-2024-08-0441_Proof_hi”, p. 8) #ffd400
*In the previous section, it seems that the authors considered 35 projects and not 32. Again, it is important to improve the presentation of the dataset creation. *
- “58 projects with issues requesting for the setup step” (“TSE-2024-08-0441_Proof_hi”, p. 10) #ffd400
*Again, it is necessary to motivate the selection of the considered projects. The authors have not properly and clearly motivated the size of the considered data and the characteristics of the projects they used to perform the experiments. *
- “use PAL4JavaWeb to generate setup steps, and then submit pull requests (PRs) for them, while also addressing issues raised by other developers, and then wait for feedback from the developers.” (“TSE-2024-08-0441_Proof_hi”, p. 10) #ffd400
*This is obscured. What does it mean that authors submitted pull requests to randomly selected projects? PAL4JavaWeb is supposed to generate configuration specifications and I don't understand how these can be the subject of PRs. *