logseq/pages/Omnivore___16-12-2023___Breaking the Silence- the Threats of Using LLMs in Software Engineering.md

• full-title:: Breaking the Silence: the Threats of Using LLMs in Software Engineering site:: d1ysz50cxb9zwl.cloudfront.net labels:: PROJECTS/MOSAICO ResearchPaper LLMs P1 date-saved:: 16-12-2023 date-archived:: 10-01-2024 is-archived:: 10 source:: Omnivore state:: archived

  • Highlights

    • id:: ca01de43-6653-404a-a516-5f2761c38163 omnivore-note:: This is important to mention as a real example of when it is essential to evaluate different LLMs (AI agents) that are supposed to do the same tasks but with different accuracy PROJECTS/MOSAICO #omnivore-note-color

      Schäfer et al. [34] investigated the performances of three LLMs (Codex, ChatGPT3.5 [28], and CodeGen [27]) in generating unit tests for Java programs. They reported remarkable performance discrepan- cies between the HumanEval [12] (>69% branch coverage) and the SF110 [21] (2% branch coverage) datasets. We remark that the for- mer is available on GitHub [12] while the latter is is not (available on SourceForge instead) ⤴️

    • id:: 233aa621-eaea-458e-90cf-742ebfd300a1

      researchers can use well-established code clone detection techniques [4] to check if the generated code (e.g., test cases) is similar to code seen in online repositories ⤴️

    • id:: da62209c-c2df-4a2a-a2e5-560b5cf34225

      although deterministic answer is not fully guaranteed due to different back-end settings ⤴️

    • id:: 73f136ed-385f-4a9f-8ecd-f2beb89aa737 omnivore-note:: #BRAINSTORMING #IDEAS #omnivore-note-color

      Threats of Using LLMs in Software Engineering ⤴️

    • id:: 10f4cae7-6f8f-472c-9fc3-6995e08acfbe

      impacting vari- ous SE tasks from code completion to test generation, from program repair to code summarization. ⤴️

    • id:: 3418fce5-019e-4df5-8123-341b008a5203

      the use of such platforms is not yet a regular and consistent practice ⤴️

    • id:: 451741b2-ab24-4b96-95ba-15780ec568ce

      run experiments using both open- source and closed-source LLMs. ⤴️

    • id:: a5758c9c-f4c3-47b8-8cd2-f22a2df824f3

      intricate factors can influence the outcomes of experiments involving LLMs ⤴️

    • id:: 86bd781b-428f-4434-a546-a5d16eccaf9c omnivore-note:: #BRAINSTORMING #omnivore-note-color

      potential threats to the validity of LLM-based research including issues such as closed-source models, possible data leakage between LLM training data and research evaluation, and the reproducibility of LLM-based findings ⤴️

    • id:: 7b2371f5-5680-4726-a86e-3df00ff159e9

      llama2 models, in particular, offer the advantage of running on consumer-grade devices ⤴️

    • id:: 3c948615-ce6d-4d7f-a20d-dd3251336347

      guidelines for SE researchers ⤴️

    • id:: d1a945e4-6178-4105-9bed-d0af57e54778

      deploying an open-source LLM model comparable to ChatGPT, like the Falcon 180B model, would incur a monthly cost of $29,495 on AWS ⤴️

    • id:: c107fb67-4ea3-4e91-bf0c-d14d32a8cb7b omnivore-note:: #challenges #llms #omnivore-note-color

      Consequently, a scenario of data leakage can arise if the LLM is trained on “project A” that employs a specific API, and the resulting model is subsequently used to fix the usage of the same API in another project within the test set ⤴️

    • id:: d7b35ee6-fa3f-4dd9-a71c-8497cc92023f omnivore-note:: This seems to be very relevant for PROJECTS/MOSAICO #omnivore-note-color

      ONNX simplifies the transition between various models, enhancing the efficiency and consistency of experimentation ⤴️

    • id:: c8788939-497a-4634-9a9b-0ae110c16cb4

      we advise researchers to complement the analysis of the LLMs performance with new data samples generated with metamorphic testing ⤴️

    • id:: 484fa9bb-5853-4046-bac8-641a2145e821

      need to thoroughly examine the validity of research findings when LLMs are involved ⤴️

    • id:: 0c1796c1-8811-4328-8353-6e21e3650fdc

      Assess output variability ⤴️

    • id:: 1d76385b-6728-4490-aa7d-7a116f505de3

      The ability to obtain identical results following the same procedure by external parties is proven to be challenging. ⤴️

    • id:: 60d18545-d114-4117-9032-59df991b9b61 omnivore-note:: How many? :-) #question #omnivore-note-color

      multiple replication runs ⤴️

    • id:: 747dad1b-4c99-41cf-bf51-e85a5c4005ed omnivore-note:: #challenges #llms #omnivore-note-color

      reliance on closed-source LLMs, potential data leakages, and concerns about reproducibility. ⤴️

    • id:: 70eed40f-c4c6-4098-a43a-bde9e83e54e4

      Such changes can occur during or after the research approach has been presented, potentially making the presented results obsolete. ⤴️

    • id:: 5770f4f3-5921-47e2-af6a-3095c88742d4

      LLMs exhibit variability in their outputs, even when using identical input. ⤴️

    • id:: ba21f06f-2514-4af7-b666-ca8e4b528a90 omnivore-note:: #challenges #llms #omnivore-note-color

      Running the same prompt several times may not result in identical output, rendering the usage of LLMs non-deterministic. ⤴️

    • id:: 6cde10cd-4ef0-4f92-8746-03fb01b5ff2b omnivore-note:: #challenges #llms #omnivore-note-color

      distinguish- ing whether the improvements claimed in the new contribution are the result of changes to the LLMs’ models or due to the novelty of the contribution becomes a complex task. ⤴️

    • id:: 91ddb7a6-0602-46d6-b8b9-0d36f6b9be36

      API service could be established, enabling users to verify if a partic- ular data source was included in the model’s training or validation datasets. ⤴️

    • id:: 5a92d5e4-fa78-4cc0-9aea-cd4be0ca7571

      This raises severe concerns about the threats to both construct (training and evaluating on the same dataset) and external (do the results hold for unknown projects/code?) validity. ⤴️

    • id:: e346583b-fd7c-49ca-8cfc-5ee1e3da3de5

      model version, and they should adopt a versioning nomenclature that distinguishes major revisions from minor updates ⤴️

    • id:: bdab387c-7683-459b-9a3e-777f8bf9ac8e

      From code generation to bug detection and natural language interactions with codebases, LLMs have played a pivotal role in recent SE advancements ⤴️

    • id:: 07075ff4-ce4e-4f71-9c71-1b0683f69132

      closed-source LLMs and their implications w.r.t. data privacy and the models’ evolution unpredictability ⤴️

    • id:: c7a3aeeb-886c-4703-8d79-97f54937b4bb

      blurry separation between training, validation, and test sets and the corresponding potential explicit/implicit data leakage ⤴️

    • id:: af58f31b-9d26-429a-84bf-0157c5469db1 omnivore-note:: This seems to be related to the model transparency point above. #omnivore-note-color

      A positive example is CodeBERT, whose provides do not disclose pre-training code but enable verification of included projects for pre-training through the train split data ⤴️

    • id:: f7cb923b-27a2-4a3a-a890-fcc1b45381b3

      Reproducibility of the published research outcomes over time, due to the non-stochastic nature of LLMs answers, the non-transparent releases of new model versions, and the lack of complete traceability. ⤴️

    • id:: e673ca80-3af5-43d8-8fa1-c106c2c21b76

      These models contain millions (e.g., BERT) to billions (e.g., ChatGPT- 4 and LLaMA) of parameters, which undergo a series of iterative optimizations during pre-training that minimize the loss function. ⤴️

    • id:: f18d63e9-8a44-433d-80b0-73db9ea6d1ae

      there is no assurance that the results will remain consistent over time. ⤴️

    • id:: 7b8fcfda-06e9-4d60-b243-1236909e2c42 omnivore-note:: Is this really possible despite the fixed random seed? #omnivore-note-color

      LLM providers should ensure the inclusion of a settable random seed during the inference of LLMs, render the inference deterministic for each specific case. ⤴️

    • id:: 39460a9a-fdee-474a-a26d-5d7eb0ccb9df

      Use different sources. As shown by Schäfer et al. [34], LLMs achieve much worse results on projects from SourceForge com- pared to GitHub. Hence, we recommend researchers gather soft- ware projects and data from multiple sources ⤴️

    • id:: b1b715f5-5e01-4061-a03c-3d9d5c280bc7

      initial set of guidelines aimed at mitigating these threats, specifically targeting SE researchers and Language Model (LM) providers ⤴️