67 lines
9.9 KiB
Markdown
67 lines
9.9 KiB
Markdown
type:: [[REVIEWS]]
|
||
tags::
|
||
year:: 2024
|
||
venue:: [[ESEM]]
|
||
full-title:: AutomaticLibraryMigrationUsingLargeLanguageModels: FirstResults
|
||
date-start:: [[15-07-2024]] - 09:23
|
||
date-submitted::
|
||
external-links::
|
||
status:: [[DONE]]
|
||
deadline-submission:: [[16-07-2024]]
|
||
file:: [[@ESEM24_paper_205.pdf]]
|
||
parent::
|
||
todoist:: https://app.todoist.com/app/task/esem-24-submission-assignment-6VVGWMVqvVwcp598
|
||
|
||
- ### [[Highlights]]
|
||
- (15/7/2024, 11:13:13)
|
||
- - “we report the preliminary results of a study in which we are exploring the use of ChatGPT to support API migration tasks” ([“ESEM24_paper_205.pdf”, p. 1](zotero://select/library/items/HISLHCJC)) ([pdf](zotero://open-pdf/library/items/3DRA2NCJ?page=1&annotation=ID5E2XV2)) #5fb236
|
||
- - “the use of ChatGPT to migrate a client application to use a newer version of SQLAlchemy” ([“ESEM24_paper_205.pdf”, p. 1](zotero://select/library/items/HISLHCJC)) ([pdf](zotero://open-pdf/library/items/3DRA2NCJ?page=1&annotation=QUTW9SEZ)) #5fb236
|
||
- - “evolve rapidly” ([“ESEM24_paper_205.pdf”, p. 1](zotero://select/library/items/HISLHCJC)) ([pdf](zotero://open-pdf/library/items/3DRA2NCJ?page=1&annotation=DGP3YSPW)) #ffd400
|
||
- *How can the rapid evolution managed by LLMs without any dedicated training?*
|
||
- - “this task is manual since we lack established and consolidated tools to support it, at least in production software” ([“ESEM24_paper_205.pdf”, p. 1](zotero://select/library/items/HISLHCJC)) ([pdf](zotero://open-pdf/library/items/3DRA2NCJ?page=1&annotation=SBEBFFHV)) #5fb236
|
||
- - “we use this version as a ground-truth, particularly for analysing the quality scores generated by the mentioned static analysis tools.” ([“ESEM24_paper_205.pdf”, p. 1](zotero://select/library/items/HISLHCJC)) ([pdf](zotero://open-pdf/library/items/3DRA2NCJ?page=1&annotation=MGF3V8GK)) #5fb236
|
||
- - “we also evaluated and presented the results of the migration of the application’s tests” ([“ESEM24_paper_205.pdf”, p. 1](zotero://select/library/items/HISLHCJC)) ([pdf](zotero://open-pdf/library/items/3DRA2NCJ?page=1&annotation=VYP9Z5HA)) #ffd400
|
||
- *so, also the tests of the original application have been migrated.*
|
||
- - “set of prompts” ([“ESEM24_paper_205.pdf”, p. 1](zotero://select/library/items/HISLHCJC)) ([pdf](zotero://open-pdf/library/items/3DRA2NCJ?page=1&annotation=4EPRUQ6Z)) #ffd400
|
||
- *the issue here is that we don't know how generalizable prompts are and to what extent they are robust with respect to the evolution of the underlying LLMs.*
|
||
- - “analysis of the different prompting methods” ([“ESEM24_paper_205.pdf”, p. 1](zotero://select/library/items/HISLHCJC)) ([pdf](zotero://open-pdf/library/items/3DRA2NCJ?page=1&annotation=GK96X3JF)) #5fb236
|
||
- - “migrating from version 1 to version 2 of the SQLAlchemy” ([“ESEM24_paper_205.pdf”, p. 2](zotero://select/library/items/HISLHCJC)) ([pdf](zotero://open-pdf/library/items/3DRA2NCJ?page=2&annotation=98KNAGEC)) #5fb236
|
||
- - “Python’s asyncio was introduced in version 1.4, version 2.0 includes performance improvements and optimizations for asynchronous operations.” ([“ESEM24_paper_205.pdf”, p. 2](zotero://select/library/items/HISLHCJC)) ([pdf](zotero://open-pdf/library/items/3DRA2NCJ?page=2&annotation=7WUEXE4F)) #5fb236
|
||
- - “after migration” ([“ESEM24_paper_205.pdf”, p. 2](zotero://select/library/items/HISLHCJC)) ([pdf](zotero://open-pdf/library/items/3DRA2NCJ?page=2&annotation=AKE528S2)) #ffd400
|
||
- *Is it fully automated? I'm not sure, I think developers have to take care of integrating the output produced by the used LLM with the original source code that needs to be updated.*
|
||
- - “The first step consists on migrating only the application code with the assistance of GPT” ([“ESEM24_paper_205.pdf”, p. 2](zotero://select/library/items/HISLHCJC)) ([pdf](zotero://open-pdf/library/items/3DRA2NCJ?page=2&annotation=8IFEBVC9)) #5fb236
|
||
- - “assess the migrated tests by running them on the manually migrated application” ([“ESEM24_paper_205.pdf”, p. 2](zotero://select/library/items/HISLHCJC)) ([pdf](zotero://open-pdf/library/items/3DRA2NCJ?page=2&annotation=EQGLBXB9)) #5fb236
|
||
- - “yielded slightly different results each time, even though the temperature setting was set to 0.” ([“ESEM24_paper_205.pdf”, p. 2](zotero://select/library/items/HISLHCJC)) ([pdf](zotero://open-pdf/library/items/3DRA2NCJ?page=2&annotation=386D6DT4)) #ffd400
|
||
- *This is related to one of my previous comments about the robustness of the LLM-based solution.*
|
||
- - “9. Make sure the code works.” ([“ESEM24_paper_205.pdf”, p. 3](zotero://select/library/items/HISLHCJC)) ([pdf](zotero://open-pdf/library/items/3DRA2NCJ?page=3&annotation=RKY6JBR3)) #ffd400
|
||
- *What's the role of this? Have you checked the quality of the results produced without such a line?*
|
||
- - “Application Migration Results” ([“ESEM24_paper_205.pdf”, p. 4](zotero://select/library/items/HISLHCJC)) ([pdf](zotero://open-pdf/library/items/3DRA2NCJ?page=4&annotation=MMD8NX4A)) #ffd400
|
||
- *Some details about the migration and execution environment is needed. Moreover, it is necessary to present the context that is given together with the source code to be migrated. How is it passed? To make the approach applicable in practice, it is necessary to show such details.*
|
||
- - “Runs Successfully” ([“ESEM24_paper_205.pdf”, p. 4](zotero://select/library/items/HISLHCJC)) ([pdf](zotero://open-pdf/library/items/3DRA2NCJ?page=4&annotation=X47R7HUQ)) #ffd400
|
||
- *How many executions have you done? By considering the nature of the considered tools, it is necessary to perform several executions for each query and then calculate the average of the considered metrics.*
|
||
- - “a fixture is a function that runs before each test)” ([“ESEM24_paper_205.pdf”, p. 5](zotero://select/library/items/HISLHCJC)) ([pdf](zotero://open-pdf/library/items/3DRA2NCJ?page=5&annotation=VDHX3H6C)) #5fb236
|
||
- - “It is interesting to point out that all three approaches managed to migrate the tests in the same way, including making the same mistakes while migrating the mentioned fixture.” ([“ESEM24_paper_205.pdf”, p. 5](zotero://select/library/items/HISLHCJC)) ([pdf](zotero://open-pdf/library/items/3DRA2NCJ?page=5&annotation=IL87L46B)) #5fb236
|
||
- - “The first threat that we address in this study is related to the selection of only one repository as target client for migration.” ([“ESEM24_paper_205.pdf”, p. 6](zotero://select/library/items/HISLHCJC)) ([pdf](zotero://open-pdf/library/items/3DRA2NCJ?page=6&annotation=TIFLTEFQ)) #5fb236
|
||
- - “To mitigate this threat, we have set to 0 the temperature option in its settings, which should make it less "nondeterministic"” ([“ESEM24_paper_205.pdf”, p. 6](zotero://select/library/items/HISLHCJC)) ([pdf](zotero://open-pdf/library/items/3DRA2NCJ?page=6&annotation=FG5Q588X)) #5fb236
|
||
- - “with a Few-Shot approach positively influences the results.” ([“ESEM24_paper_205.pdf”, p. 6](zotero://select/library/items/HISLHCJC)) ([pdf](zotero://open-pdf/library/items/3DRA2NCJ?page=6&annotation=6DH53PNX)) #5fb236
|
||
- - “CoCoNut and Codex” ([“ESEM24_paper_205.pdf”, p. 6](zotero://select/library/items/HISLHCJC)) ([pdf](zotero://open-pdf/library/items/3DRA2NCJ?page=6&annotation=BZBBCP55)) #ffd400
|
||
- *What about OpenRewrite?*
|
||
- - “We concluded that LLMs are able to correctly migrate the project when provided with at least one example of an already migrated application, making only minor mistakes such as leaving unused imports or incorrectly typed variables, which are errors than can be later fixed by a developer.” ([“ESEM24_paper_205.pdf”, p. 6](zotero://select/library/items/HISLHCJC)) ([pdf](zotero://open-pdf/library/items/3DRA2NCJ?page=6&annotation=9DM8WNIB)) #5fb236
|
||
- ### [[Comments]]
|
||
- SUMMARY: This vision paper presents preliminary results on using LLMs to support library migration. Specifically, it explores the use of ChatGPT for API migration tasks, focusing on migrating client applications to newer versions of SQLAlchemy. The evaluation also includes the migration of application tests. Zero-shot, one-shot, and chain-of-thought approaches have been employed, and the results have been compared using various metrics.
|
||
- COMMENTS: The paper is about an interesting topic. Even though this is a vision paper, several aspects need to be at least mentioned to make the work stronger as suggested below:
|
||
- The results yielded slightly different outcomes each time, even with the temperature setting at 0. How does this affect the reliability of the LLM-based solution?
|
||
- The generalizability of the prompts and their robustness with respect to the evolution of underlying LLMs are crucial. How can the prompts be tested for these factors?
|
||
- More robust empirical evidence is needed to support the claims. Specifically, how can LLMs manage rapid evolution without dedicated training (this is a kind of cold-start problem)?
|
||
- The comparison with state-of-the-art approaches needs to be added. Consider including tools like OpenRewrite (even though it is for Java-based systems) in the comparison.
|
||
- Details about the migration and execution environment are needed. How is the context passed to the LLM? How many executions were performed? Consider performing multiple executions for each query and calculating the average of the considered metrics to provide more reliable results.
|
||
- Is the migration fully automated, or do developers need to integrate the LLM-generated output with the original source code? Clarify the extent of manual intervention required. Can the migration be fully automated, or do developers need to integrate the LLM-generated output with the original source code?
|
||
- ### [[REVIEWS/Notes]]
|
||
- ### YELLOW CONCERNS
|
||
background-color:: yellow
|
||
- {{query (and [[ffd400]] [[ESEM24_paper_205]] )}}
|
||
collapsed:: true
|
||
- ### ❓️Questions
|
||
- {{query (and [[question]] [[ESEM24_paper_205]] )[[question]]}}
|
||
query-table:: true
|
||
query-properties:: [:block] |