- full-title:: [Breaking the Silence: the Threats of Using LLMs in Software Engineering](https://omnivore.app/me/file-18c732585b2) site:: [d1ysz50cxb9zwl.cloudfront.net](https://d1ysz50cxb9zwl.cloudfront.net/pOrqYDyTV9CKBD0eeANYu2WZHCNdDQB7wDT_y58lU4lasW4JqbXmMuD6RpY9fY_0/by/4646762/as/file.pdf?Expires=1702738936&Key-Pair-Id=APKAJAERRT46LD6FN4NA&Signature=daWQlTU0pvSQ2v1qvc%7Etw7E%7E-YHmoS0DO-9oApc-%7EvNdYYJJ7noVKPh47u8AcrCK7C%7ESsdvDy5srp0X3NgRO5sxex5baLc0JaXyEcmN1zZGfiiowxrWGrm64ANfBf1sRLBF6jzPxCPesgeqjRsAGp3NAZcVSLO21idC9Hnc5BBh%7ERMWJ29kZ7CmXvOLMJn0STgFIPhFKKqUz1FNequeaA3R0-DL7A7bqtpfMCzv-JtyhFWoabWul10mLO9I%7E3-3DMXzPT%7EOyGLGIv4STefVk-R0MWAo-NcXssAdPsV0epbks3GNnwhveFR9Za7j5h4HSM6sxl7ymxQvIwSChBLrIwA__) labels:: [[PROJECTS/MOSAICO]] [[ResearchPaper]] [[LLMs]] [[P1]] date-saved:: [[16-12-2023]] date-archived:: [[10-01-2024]] is-archived:: 10 source:: [[Omnivore]] state:: [[archived]] - ### Highlights - id:: ca01de43-6653-404a-a516-5f2761c38163 omnivore-note:: This is important to mention as a real example of when it is essential to evaluate different LLMs (AI agents) that are supposed to do the same tasks but with different accuracy [[PROJECTS/MOSAICO]] #omnivore-note-color > Schäfer et al. [34] investigated the performances of three LLMs (Codex, ChatGPT3.5 [28], and CodeGen [27]) in generating unit tests for Java programs. They reported remarkable performance discrepan- cies between the HumanEval [12] (>69% branch coverage) and the SF110 [21] (2% branch coverage) datasets. We remark that the for- mer is available on GitHub [12] while the latter is is not (available on SourceForge instead) [⤴️](https://omnivore.app/me/file-18c732585b2#ca01de43-6653-404a-a516-5f2761c38163) - id:: 233aa621-eaea-458e-90cf-742ebfd300a1 > researchers can use well-established code clone detection techniques [4] to check if the generated code (e.g., test cases) is similar to code seen in online repositories [⤴️](https://omnivore.app/me/file-18c732585b2#233aa621-eaea-458e-90cf-742ebfd300a1) - id:: da62209c-c2df-4a2a-a2e5-560b5cf34225 > although deterministic answer is not fully guaranteed due to different back-end settings [⤴️](https://omnivore.app/me/file-18c732585b2#da62209c-c2df-4a2a-a2e5-560b5cf34225) - id:: 73f136ed-385f-4a9f-8ecd-f2beb89aa737 omnivore-note:: #BRAINSTORMING #IDEAS #omnivore-note-color > Threats of Using LLMs in Software Engineering [⤴️](https://omnivore.app/me/file-18c732585b2#73f136ed-385f-4a9f-8ecd-f2beb89aa737) - id:: 10f4cae7-6f8f-472c-9fc3-6995e08acfbe > impacting vari- ous SE tasks from code completion to test generation, from program repair to code summarization. [⤴️](https://omnivore.app/me/file-18c732585b2#10f4cae7-6f8f-472c-9fc3-6995e08acfbe) - id:: 3418fce5-019e-4df5-8123-341b008a5203 > the use of such platforms is not yet a regular and consistent practice [⤴️](https://omnivore.app/me/file-18c732585b2#3418fce5-019e-4df5-8123-341b008a5203) - id:: 451741b2-ab24-4b96-95ba-15780ec568ce > run experiments using both open- source and closed-source LLMs. [⤴️](https://omnivore.app/me/file-18c732585b2#451741b2-ab24-4b96-95ba-15780ec568ce) - id:: a5758c9c-f4c3-47b8-8cd2-f22a2df824f3 > intricate factors can influence the outcomes of experiments involving LLMs [⤴️](https://omnivore.app/me/file-18c732585b2#a5758c9c-f4c3-47b8-8cd2-f22a2df824f3) - id:: 86bd781b-428f-4434-a546-a5d16eccaf9c omnivore-note:: #BRAINSTORMING #omnivore-note-color > potential threats to the validity of LLM-based research including issues such as closed-source models, possible data leakage between LLM training data and research evaluation, and the reproducibility of LLM-based findings [⤴️](https://omnivore.app/me/file-18c732585b2#86bd781b-428f-4434-a546-a5d16eccaf9c) - id:: 7b2371f5-5680-4726-a86e-3df00ff159e9 > llama2 models, in particular, offer the advantage of running on consumer-grade devices [⤴️](https://omnivore.app/me/file-18c732585b2#7b2371f5-5680-4726-a86e-3df00ff159e9) - id:: 3c948615-ce6d-4d7f-a20d-dd3251336347 > guidelines for SE researchers [⤴️](https://omnivore.app/me/file-18c732585b2#3c948615-ce6d-4d7f-a20d-dd3251336347) - id:: d1a945e4-6178-4105-9bed-d0af57e54778 > deploying an open-source LLM model comparable to ChatGPT, like the Falcon 180B model, would incur a monthly cost of $29,495 on AWS [⤴️](https://omnivore.app/me/file-18c732585b2#d1a945e4-6178-4105-9bed-d0af57e54778) - id:: c107fb67-4ea3-4e91-bf0c-d14d32a8cb7b omnivore-note:: #challenges #llms #omnivore-note-color > Consequently, a scenario of data leakage can arise if the LLM is trained on “project A” that employs a specific API, and the resulting model is subsequently used to fix the usage of the same API in another project within the test set [⤴️](https://omnivore.app/me/file-18c732585b2#c107fb67-4ea3-4e91-bf0c-d14d32a8cb7b) - id:: d7b35ee6-fa3f-4dd9-a71c-8497cc92023f omnivore-note:: This seems to be very relevant for [[PROJECTS/MOSAICO]] #omnivore-note-color > ONNX simplifies the transition between various models, enhancing the efficiency and consistency of experimentation [⤴️](https://omnivore.app/me/file-18c732585b2#d7b35ee6-fa3f-4dd9-a71c-8497cc92023f) - id:: c8788939-497a-4634-9a9b-0ae110c16cb4 > we advise researchers to complement the analysis of the LLMs performance with new data samples generated with metamorphic testing [⤴️](https://omnivore.app/me/file-18c732585b2#c8788939-497a-4634-9a9b-0ae110c16cb4) - id:: 484fa9bb-5853-4046-bac8-641a2145e821 > need to thoroughly examine the validity of research findings when LLMs are involved [⤴️](https://omnivore.app/me/file-18c732585b2#484fa9bb-5853-4046-bac8-641a2145e821) - id:: 0c1796c1-8811-4328-8353-6e21e3650fdc > Assess output variability [⤴️](https://omnivore.app/me/file-18c732585b2#0c1796c1-8811-4328-8353-6e21e3650fdc) - id:: 1d76385b-6728-4490-aa7d-7a116f505de3 > The ability to obtain identical results following the same procedure by external parties is proven to be challenging. [⤴️](https://omnivore.app/me/file-18c732585b2#1d76385b-6728-4490-aa7d-7a116f505de3) - id:: 60d18545-d114-4117-9032-59df991b9b61 omnivore-note:: How many? :-) #question #omnivore-note-color > multiple replication runs [⤴️](https://omnivore.app/me/file-18c732585b2#60d18545-d114-4117-9032-59df991b9b61) - id:: 747dad1b-4c99-41cf-bf51-e85a5c4005ed omnivore-note:: #challenges #llms #omnivore-note-color > reliance on closed-source LLMs, potential data leakages, and concerns about reproducibility. [⤴️](https://omnivore.app/me/file-18c732585b2#747dad1b-4c99-41cf-bf51-e85a5c4005ed) - id:: 70eed40f-c4c6-4098-a43a-bde9e83e54e4 > Such changes can occur during or after the research approach has been presented, potentially making the presented results obsolete. [⤴️](https://omnivore.app/me/file-18c732585b2#70eed40f-c4c6-4098-a43a-bde9e83e54e4) - id:: 5770f4f3-5921-47e2-af6a-3095c88742d4 > LLMs exhibit variability in their outputs, even when using identical input. [⤴️](https://omnivore.app/me/file-18c732585b2#5770f4f3-5921-47e2-af6a-3095c88742d4) - id:: ba21f06f-2514-4af7-b666-ca8e4b528a90 omnivore-note:: #challenges #llms #omnivore-note-color > Running the same prompt several times may not result in identical output, rendering the usage of LLMs non-deterministic. [⤴️](https://omnivore.app/me/file-18c732585b2#ba21f06f-2514-4af7-b666-ca8e4b528a90) - id:: 6cde10cd-4ef0-4f92-8746-03fb01b5ff2b omnivore-note:: #challenges #llms #omnivore-note-color > distinguish- ing whether the improvements claimed in the new contribution are the result of changes to the LLMs’ models or due to the novelty of the contribution becomes a complex task. [⤴️](https://omnivore.app/me/file-18c732585b2#6cde10cd-4ef0-4f92-8746-03fb01b5ff2b) - id:: 91ddb7a6-0602-46d6-b8b9-0d36f6b9be36 > API service could be established, enabling users to verify if a partic- ular data source was included in the model’s training or validation datasets. [⤴️](https://omnivore.app/me/file-18c732585b2#91ddb7a6-0602-46d6-b8b9-0d36f6b9be36) - id:: 5a92d5e4-fa78-4cc0-9aea-cd4be0ca7571 > This raises severe concerns about the threats to both construct (training and evaluating on the same dataset) and external (do the results hold for unknown projects/code?) validity. [⤴️](https://omnivore.app/me/file-18c732585b2#5a92d5e4-fa78-4cc0-9aea-cd4be0ca7571) - id:: e346583b-fd7c-49ca-8cfc-5ee1e3da3de5 > model version, and they should adopt a versioning nomenclature that distinguishes major revisions from minor updates [⤴️](https://omnivore.app/me/file-18c732585b2#e346583b-fd7c-49ca-8cfc-5ee1e3da3de5) - id:: bdab387c-7683-459b-9a3e-777f8bf9ac8e > From code generation to bug detection and natural language interactions with codebases, LLMs have played a pivotal role in recent SE advancements [⤴️](https://omnivore.app/me/file-18c732585b2#bdab387c-7683-459b-9a3e-777f8bf9ac8e) - id:: 07075ff4-ce4e-4f71-9c71-1b0683f69132 > closed-source LLMs and their implications w.r.t. data privacy and the models’ evolution unpredictability [⤴️](https://omnivore.app/me/file-18c732585b2#07075ff4-ce4e-4f71-9c71-1b0683f69132) - id:: c7a3aeeb-886c-4703-8d79-97f54937b4bb > blurry separation between training, validation, and test sets and the corresponding potential explicit/implicit data leakage [⤴️](https://omnivore.app/me/file-18c732585b2#c7a3aeeb-886c-4703-8d79-97f54937b4bb) - id:: af58f31b-9d26-429a-84bf-0157c5469db1 omnivore-note:: This seems to be related to the model transparency point above. #omnivore-note-color > A positive example is CodeBERT, whose provides do not disclose pre-training code but enable verification of included projects for pre-training through the train split data [⤴️](https://omnivore.app/me/file-18c732585b2#af58f31b-9d26-429a-84bf-0157c5469db1) - id:: f7cb923b-27a2-4a3a-a890-fcc1b45381b3 > Reproducibility of the published research outcomes over time, due to the non-stochastic nature of LLMs answers, the non-transparent releases of new model versions, and the lack of complete traceability. [⤴️](https://omnivore.app/me/file-18c732585b2#f7cb923b-27a2-4a3a-a890-fcc1b45381b3) - id:: e673ca80-3af5-43d8-8fa1-c106c2c21b76 > These models contain millions (e.g., BERT) to billions (e.g., ChatGPT- 4 and LLaMA) of parameters, which undergo a series of iterative optimizations during pre-training that minimize the loss function. [⤴️](https://omnivore.app/me/file-18c732585b2#e673ca80-3af5-43d8-8fa1-c106c2c21b76) - id:: f18d63e9-8a44-433d-80b0-73db9ea6d1ae > there is no assurance that the results will remain consistent over time. [⤴️](https://omnivore.app/me/file-18c732585b2#f18d63e9-8a44-433d-80b0-73db9ea6d1ae) - id:: 7b8fcfda-06e9-4d60-b243-1236909e2c42 omnivore-note:: Is this really possible despite the fixed random seed? #omnivore-note-color > LLM providers should ensure the inclusion of a settable random seed during the inference of LLMs, render the inference deterministic for each specific case. [⤴️](https://omnivore.app/me/file-18c732585b2#7b8fcfda-06e9-4d60-b243-1236909e2c42) - id:: 39460a9a-fdee-474a-a26d-5d7eb0ccb9df > Use different sources. As shown by Schäfer et al. [34], LLMs achieve much worse results on projects from SourceForge com- pared to GitHub. Hence, we recommend researchers gather soft- ware projects and data from multiple sources [⤴️](https://omnivore.app/me/file-18c732585b2#39460a9a-fdee-474a-a26d-5d7eb0ccb9df) - id:: b1b715f5-5e01-4061-a03c-3d9d5c280bc7 > initial set of guidelines aimed at mitigating these threats, specifically targeting SE researchers and Language Model (LM) providers [⤴️](https://omnivore.app/me/file-18c732585b2#b1b715f5-5e01-4061-a03c-3d9d5c280bc7)