94 lines
12 KiB
Markdown
94 lines
12 KiB
Markdown
- full-title:: [Breaking the Silence: the Threats of Using LLMs in Software Engineering](https://omnivore.app/me/file-18c732585b2)
|
||
site:: [d1ysz50cxb9zwl.cloudfront.net](https://d1ysz50cxb9zwl.cloudfront.net/pOrqYDyTV9CKBD0eeANYu2WZHCNdDQB7wDT_y58lU4lasW4JqbXmMuD6RpY9fY_0/by/4646762/as/file.pdf?Expires=1702738936&Key-Pair-Id=APKAJAERRT46LD6FN4NA&Signature=daWQlTU0pvSQ2v1qvc%7Etw7E%7E-YHmoS0DO-9oApc-%7EvNdYYJJ7noVKPh47u8AcrCK7C%7ESsdvDy5srp0X3NgRO5sxex5baLc0JaXyEcmN1zZGfiiowxrWGrm64ANfBf1sRLBF6jzPxCPesgeqjRsAGp3NAZcVSLO21idC9Hnc5BBh%7ERMWJ29kZ7CmXvOLMJn0STgFIPhFKKqUz1FNequeaA3R0-DL7A7bqtpfMCzv-JtyhFWoabWul10mLO9I%7E3-3DMXzPT%7EOyGLGIv4STefVk-R0MWAo-NcXssAdPsV0epbks3GNnwhveFR9Za7j5h4HSM6sxl7ymxQvIwSChBLrIwA__)
|
||
labels:: [[PROJECTS/MOSAICO]] [[ResearchPaper]] [[LLMs]] [[P1]]
|
||
date-saved:: [[16-12-2023]]
|
||
date-archived:: [[10-01-2024]]
|
||
is-archived:: 10
|
||
source:: [[Omnivore]]
|
||
state:: [[archived]]
|
||
- ### Highlights
|
||
- id:: ca01de43-6653-404a-a516-5f2761c38163
|
||
omnivore-note:: This is important to mention as a real example of when it is essential to evaluate different LLMs (AI agents) that are supposed to do the same tasks but with different accuracy [[PROJECTS/MOSAICO]] #omnivore-note-color
|
||
> Schäfer et al. [34] investigated the performances of three LLMs (Codex, ChatGPT3.5 [28], and CodeGen [27]) in generating unit tests for Java programs. They reported remarkable performance discrepan- cies between the HumanEval [12] (>69% branch coverage) and the SF110 [21] (2% branch coverage) datasets. We remark that the for- mer is available on GitHub [12] while the latter is is not (available on SourceForge instead) [⤴️](https://omnivore.app/me/file-18c732585b2#ca01de43-6653-404a-a516-5f2761c38163)
|
||
- id:: 233aa621-eaea-458e-90cf-742ebfd300a1
|
||
> researchers can use well-established code clone detection techniques [4] to check if the generated code (e.g., test cases) is similar to code seen in online repositories [⤴️](https://omnivore.app/me/file-18c732585b2#233aa621-eaea-458e-90cf-742ebfd300a1)
|
||
- id:: da62209c-c2df-4a2a-a2e5-560b5cf34225
|
||
> although deterministic answer is not fully guaranteed due to different back-end settings [⤴️](https://omnivore.app/me/file-18c732585b2#da62209c-c2df-4a2a-a2e5-560b5cf34225)
|
||
- id:: 73f136ed-385f-4a9f-8ecd-f2beb89aa737
|
||
omnivore-note:: #BRAINSTORMING #IDEAS #omnivore-note-color
|
||
> Threats of Using LLMs in Software Engineering [⤴️](https://omnivore.app/me/file-18c732585b2#73f136ed-385f-4a9f-8ecd-f2beb89aa737)
|
||
- id:: 10f4cae7-6f8f-472c-9fc3-6995e08acfbe
|
||
> impacting vari- ous SE tasks from code completion to test generation, from program repair to code summarization. [⤴️](https://omnivore.app/me/file-18c732585b2#10f4cae7-6f8f-472c-9fc3-6995e08acfbe)
|
||
- id:: 3418fce5-019e-4df5-8123-341b008a5203
|
||
> the use of such platforms is not yet a regular and consistent practice [⤴️](https://omnivore.app/me/file-18c732585b2#3418fce5-019e-4df5-8123-341b008a5203)
|
||
- id:: 451741b2-ab24-4b96-95ba-15780ec568ce
|
||
> run experiments using both open- source and closed-source LLMs. [⤴️](https://omnivore.app/me/file-18c732585b2#451741b2-ab24-4b96-95ba-15780ec568ce)
|
||
- id:: a5758c9c-f4c3-47b8-8cd2-f22a2df824f3
|
||
> intricate factors can influence the outcomes of experiments involving LLMs [⤴️](https://omnivore.app/me/file-18c732585b2#a5758c9c-f4c3-47b8-8cd2-f22a2df824f3)
|
||
- id:: 86bd781b-428f-4434-a546-a5d16eccaf9c
|
||
omnivore-note:: #BRAINSTORMING #omnivore-note-color
|
||
> potential threats to the validity of LLM-based research including issues such as closed-source models, possible data leakage between LLM training data and research evaluation, and the reproducibility of LLM-based findings [⤴️](https://omnivore.app/me/file-18c732585b2#86bd781b-428f-4434-a546-a5d16eccaf9c)
|
||
- id:: 7b2371f5-5680-4726-a86e-3df00ff159e9
|
||
> llama2 models, in particular, offer the advantage of running on consumer-grade devices [⤴️](https://omnivore.app/me/file-18c732585b2#7b2371f5-5680-4726-a86e-3df00ff159e9)
|
||
- id:: 3c948615-ce6d-4d7f-a20d-dd3251336347
|
||
> guidelines for SE researchers [⤴️](https://omnivore.app/me/file-18c732585b2#3c948615-ce6d-4d7f-a20d-dd3251336347)
|
||
- id:: d1a945e4-6178-4105-9bed-d0af57e54778
|
||
> deploying an open-source LLM model comparable to ChatGPT, like the Falcon 180B model, would incur a monthly cost of $29,495 on AWS [⤴️](https://omnivore.app/me/file-18c732585b2#d1a945e4-6178-4105-9bed-d0af57e54778)
|
||
- id:: c107fb67-4ea3-4e91-bf0c-d14d32a8cb7b
|
||
omnivore-note:: #challenges #llms #omnivore-note-color
|
||
> Consequently, a scenario of data leakage can arise if the LLM is trained on “project A” that employs a specific API, and the resulting model is subsequently used to fix the usage of the same API in another project within the test set [⤴️](https://omnivore.app/me/file-18c732585b2#c107fb67-4ea3-4e91-bf0c-d14d32a8cb7b)
|
||
- id:: d7b35ee6-fa3f-4dd9-a71c-8497cc92023f
|
||
omnivore-note:: This seems to be very relevant for [[PROJECTS/MOSAICO]] #omnivore-note-color
|
||
> ONNX simplifies the transition between various models, enhancing the efficiency and consistency of experimentation [⤴️](https://omnivore.app/me/file-18c732585b2#d7b35ee6-fa3f-4dd9-a71c-8497cc92023f)
|
||
- id:: c8788939-497a-4634-9a9b-0ae110c16cb4
|
||
> we advise researchers to complement the analysis of the LLMs performance with new data samples generated with metamorphic testing [⤴️](https://omnivore.app/me/file-18c732585b2#c8788939-497a-4634-9a9b-0ae110c16cb4)
|
||
- id:: 484fa9bb-5853-4046-bac8-641a2145e821
|
||
> need to thoroughly examine the validity of research findings when LLMs are involved [⤴️](https://omnivore.app/me/file-18c732585b2#484fa9bb-5853-4046-bac8-641a2145e821)
|
||
- id:: 0c1796c1-8811-4328-8353-6e21e3650fdc
|
||
> Assess output variability [⤴️](https://omnivore.app/me/file-18c732585b2#0c1796c1-8811-4328-8353-6e21e3650fdc)
|
||
- id:: 1d76385b-6728-4490-aa7d-7a116f505de3
|
||
> The ability to obtain identical results following the same procedure by external parties is proven to be challenging. [⤴️](https://omnivore.app/me/file-18c732585b2#1d76385b-6728-4490-aa7d-7a116f505de3)
|
||
- id:: 60d18545-d114-4117-9032-59df991b9b61
|
||
omnivore-note:: How many? :-) #question #omnivore-note-color
|
||
> multiple replication runs [⤴️](https://omnivore.app/me/file-18c732585b2#60d18545-d114-4117-9032-59df991b9b61)
|
||
- id:: 747dad1b-4c99-41cf-bf51-e85a5c4005ed
|
||
omnivore-note:: #challenges #llms #omnivore-note-color
|
||
> reliance on closed-source LLMs, potential data leakages, and concerns about reproducibility. [⤴️](https://omnivore.app/me/file-18c732585b2#747dad1b-4c99-41cf-bf51-e85a5c4005ed)
|
||
- id:: 70eed40f-c4c6-4098-a43a-bde9e83e54e4
|
||
> Such changes can occur during or after the research approach has been presented, potentially making the presented results obsolete. [⤴️](https://omnivore.app/me/file-18c732585b2#70eed40f-c4c6-4098-a43a-bde9e83e54e4)
|
||
- id:: 5770f4f3-5921-47e2-af6a-3095c88742d4
|
||
> LLMs exhibit variability in their outputs, even when using identical input. [⤴️](https://omnivore.app/me/file-18c732585b2#5770f4f3-5921-47e2-af6a-3095c88742d4)
|
||
- id:: ba21f06f-2514-4af7-b666-ca8e4b528a90
|
||
omnivore-note:: #challenges #llms #omnivore-note-color
|
||
> Running the same prompt several times may not result in identical output, rendering the usage of LLMs non-deterministic. [⤴️](https://omnivore.app/me/file-18c732585b2#ba21f06f-2514-4af7-b666-ca8e4b528a90)
|
||
- id:: 6cde10cd-4ef0-4f92-8746-03fb01b5ff2b
|
||
omnivore-note:: #challenges #llms #omnivore-note-color
|
||
> distinguish- ing whether the improvements claimed in the new contribution are the result of changes to the LLMs’ models or due to the novelty of the contribution becomes a complex task. [⤴️](https://omnivore.app/me/file-18c732585b2#6cde10cd-4ef0-4f92-8746-03fb01b5ff2b)
|
||
- id:: 91ddb7a6-0602-46d6-b8b9-0d36f6b9be36
|
||
> API service could be established, enabling users to verify if a partic- ular data source was included in the model’s training or validation datasets. [⤴️](https://omnivore.app/me/file-18c732585b2#91ddb7a6-0602-46d6-b8b9-0d36f6b9be36)
|
||
- id:: 5a92d5e4-fa78-4cc0-9aea-cd4be0ca7571
|
||
> This raises severe concerns about the threats to both construct (training and evaluating on the same dataset) and external (do the results hold for unknown projects/code?) validity. [⤴️](https://omnivore.app/me/file-18c732585b2#5a92d5e4-fa78-4cc0-9aea-cd4be0ca7571)
|
||
- id:: e346583b-fd7c-49ca-8cfc-5ee1e3da3de5
|
||
> model version, and they should adopt a versioning nomenclature that distinguishes major revisions from minor updates [⤴️](https://omnivore.app/me/file-18c732585b2#e346583b-fd7c-49ca-8cfc-5ee1e3da3de5)
|
||
- id:: bdab387c-7683-459b-9a3e-777f8bf9ac8e
|
||
> From code generation to bug detection and natural language interactions with codebases, LLMs have played a pivotal role in recent SE advancements [⤴️](https://omnivore.app/me/file-18c732585b2#bdab387c-7683-459b-9a3e-777f8bf9ac8e)
|
||
- id:: 07075ff4-ce4e-4f71-9c71-1b0683f69132
|
||
> closed-source LLMs and their implications w.r.t. data privacy and the models’ evolution unpredictability [⤴️](https://omnivore.app/me/file-18c732585b2#07075ff4-ce4e-4f71-9c71-1b0683f69132)
|
||
- id:: c7a3aeeb-886c-4703-8d79-97f54937b4bb
|
||
> blurry separation between training, validation, and test sets and the corresponding potential explicit/implicit data leakage [⤴️](https://omnivore.app/me/file-18c732585b2#c7a3aeeb-886c-4703-8d79-97f54937b4bb)
|
||
- id:: af58f31b-9d26-429a-84bf-0157c5469db1
|
||
omnivore-note:: This seems to be related to the model transparency point above. #omnivore-note-color
|
||
> A positive example is CodeBERT, whose provides do not disclose pre-training code but enable verification of included projects for pre-training through the train split data [⤴️](https://omnivore.app/me/file-18c732585b2#af58f31b-9d26-429a-84bf-0157c5469db1)
|
||
- id:: f7cb923b-27a2-4a3a-a890-fcc1b45381b3
|
||
> Reproducibility of the published research outcomes over time, due to the non-stochastic nature of LLMs answers, the non-transparent releases of new model versions, and the lack of complete traceability. [⤴️](https://omnivore.app/me/file-18c732585b2#f7cb923b-27a2-4a3a-a890-fcc1b45381b3)
|
||
- id:: e673ca80-3af5-43d8-8fa1-c106c2c21b76
|
||
> These models contain millions (e.g., BERT) to billions (e.g., ChatGPT- 4 and LLaMA) of parameters, which undergo a series of iterative optimizations during pre-training that minimize the loss function. [⤴️](https://omnivore.app/me/file-18c732585b2#e673ca80-3af5-43d8-8fa1-c106c2c21b76)
|
||
- id:: f18d63e9-8a44-433d-80b0-73db9ea6d1ae
|
||
> there is no assurance that the results will remain consistent over time. [⤴️](https://omnivore.app/me/file-18c732585b2#f18d63e9-8a44-433d-80b0-73db9ea6d1ae)
|
||
- id:: 7b8fcfda-06e9-4d60-b243-1236909e2c42
|
||
omnivore-note:: Is this really possible despite the fixed random seed? #omnivore-note-color
|
||
> LLM providers should ensure the inclusion of a settable random seed during the inference of LLMs, render the inference deterministic for each specific case. [⤴️](https://omnivore.app/me/file-18c732585b2#7b8fcfda-06e9-4d60-b243-1236909e2c42)
|
||
- id:: 39460a9a-fdee-474a-a26d-5d7eb0ccb9df
|
||
> Use different sources. As shown by Schäfer et al. [34], LLMs achieve much worse results on projects from SourceForge com- pared to GitHub. Hence, we recommend researchers gather soft- ware projects and data from multiple sources [⤴️](https://omnivore.app/me/file-18c732585b2#39460a9a-fdee-474a-a26d-5d7eb0ccb9df)
|
||
- id:: b1b715f5-5e01-4061-a03c-3d9d5c280bc7
|
||
> initial set of guidelines aimed at mitigating these threats, specifically targeting SE researchers and Language Model (LM) providers [⤴️](https://omnivore.app/me/file-18c732585b2#b1b715f5-5e01-4061-a03c-3d9d5c280bc7) |