8.1 KiB
type:: meeting
external-links::
tags:: PROJECTS/PRIN-2022-TREXSE
people:: people/juri people/PhuongNguyen people/dipenta people/fedelucio
isconsortium:: true
date:: 06-12-2023 - 22:23
duration::
- ## Agenda
- Mattino: Sala meeting RCOST - Via Traiano 9
- 11:00-11.30 Introduzione (Max)
- 11:30-12:00 Presentazione gruppo UnivAQ (Davide)
- [UDA.pptx](https://univaq-my.sharepoint.com/:p:/g/personal/davide_diruscio_univaq_it/Ed-9fwv3WAVCqbgpUoO1ChwBAHk35zECCTQVplskYa-9GA?e=UNxJL1)
- 12:00-12:30 Presentazione gruppo PoliBa (Lucio)
- 12:30-13:00 Presentazione gruppo Unisannio (Max)
- 13:00-14:30 Pausa Pranzo
- 14:30-17:00 Brainstorming + Presentazioni individuali
- **Max presentation**
- We focus on tools that support software developers.
- Idntification of vulnerabilities and bias in RSSE recommendations
- Design and Evaluation of Trustworthy RSSEs
- Huma-in-the-loop RSSEs
- Outcomes
- Create the training set (UniSannio)
- Dealing with prompting
- COnceptualization of trustworthiness (UniBari)
- Feedback management (Univaq)
-
-
Ideas
- Revisit everything with the presence of LLMs, RAG Architecture, ChatGPT
- Replicate some of the works done in this context
- There are gray ares we should pay attention, like security and bias
- {{query (and Ideas 07-12-2023)}} query-table:: true
-
Tasks
- DONE Prossimo meeting in L'Aquila (Estate 2024) @goal id:: 65c8d43d-09c9-416b-a127-a865e0fd3b08 DEADLINE: <2024-02-26 Mon> :LOGBOOK: CLOCK: [2025-05-26 Mon 22:20:40]--[2025-05-26 Mon 22:20:41] => 00:00:01 :END:
- DONE Create a Webiste on Github
- Create a X account
-
Readings
- Learn AI-Assisted Python Programming (manning.com)
- Demistifying and Misuse of LLMs...
- No more fine-tuning? an experimental evaluation of prompt tuning in code intelligence | Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (oclc.org) id:: 6571d35e-27b9-4de3-b228-c989aca40852
- ((6571dab5-f6fe-42e0-b35d-ed1f52625070))
-
Ideas
- Retrieval Augmented Generation (#RAG) (generated by LAMA)
-
is a recently proposed approach to language generation that combines the strengths of two existing techniques: retrieval-based language models and generative language models.
Traditionally, language generation has been approached through either retrieval-based models or generative models. Retrieval-based models, such as memory-augmented neural networks (MANN) and memory-based language models (MBLM), rely on retrieving relevant information from a large corpus of text to generate new language. On the other hand, generative models, such as variational autoencoders (VAEs) and generative adversarial networks (GANs), use probabilistic generative techniques to produce new language from scratch.
RAG combines these two approaches by first using a retrieval model to identify relevant information in the corpus, and then using a generative model to augment this information and generate new language. The key idea is to leverage the strengths of both approaches to generate more coherent and contextually relevant text.
RAG has been shown to achieve state-of-the-art results in various natural language generation tasks, including text summarization, machine translation, and dialogue generation. By combining retrieval and generative capabilities, RAG can generate text that is not only grammatically correct but also semantically meaningful and contextually appropriate.
RAG has a wide range of applications, including but not limited to:
- Chatbots: RAG can be used to generate responses in chatbots that are more coherent and relevant to the user's input.
- Language translation: RAG can be used to improve machine translation by generating more contextually appropriate translations.
- Text summarization: RAG can be used to generate summaries that are more accurate and concise.
- Creative writing: RAG can be used to generate creative writing, such as poetry or short stories, that are more coherent and contextually relevant.
- Data-to-text generation: RAG can be used to generate text from data, such as generating a weather report from atmospheric conditions.
In summary, Retrieval Augmented Generation (RAG) is a powerful approach to language generation that combines the strengths of retrieval-based and generative models to generate more coherent and contextually relevant text. Its applications range from chatbots and machine translation to creative writing and data-to-text generation.
-
- Hard vs Soft prompting
- Conceive a set of prompt templates to specify recommender systems or to improve their accuracy
- see ((6571d35e-27b9-4de3-b228-c989aca40852))
- Retrieval Augmented Generation (RAG)
{:height 546, :width 672}- [DSC Europe 23] Spela Poklukar & Tea Brasanac - Retrieval Augmented Generation | PPT (slideshare.net)
{:height 388, :width 673}
collapsed:: true
- You don't need to fine tune the model, you use the model as it is.
- WHat happens if you add mutations in the knowledge base?
- RAG vs conventioanl way
collapsed:: true
- What is more resilient with respect to adversarial attack?
- What kinds of attacks do we want to support in the project?
- The model is trained with buggy code
- Bing seems to be less protective than ChatGPT
- Ask the model to generate malicious code
- Is it possible to understand if malicious code is part of the training dataset?
- Retrieval Augmented Generation (#RAG) (generated by LAMA)
-
Objectives
-
Characterizing vulnerabilities of RAG vs vulnerabilities of models
- We should focus on bugs, vulnerabilities, and maybe not on bias
- Assess the quality of generated code
- Take the MSR paper (IEEE Xplore Full-Text PDF: (oclc.org)) and do an extension for a journal with LLMs ideas
- So the thing to be done here is to understand how to create the query, e.g., "I have a project that includes these libraries, what are the additional ones that I should include"
- How can we ask the language model to check if the libraries that I'm including is really needed? Ideas
- To answer this question we should include some context about the project ()for instance a description about what it does)
-
Trustworthiness
- Proxies: some features (e.g. neighbor where a person resides) that are proxy of some bias-introducing features (e.g. societal status)
- Transparency and Explainability are very relevant
- For recommendations the explanation is able to highlight the features that the recommended has and that the user is interested in. So recommendations are explained based on the user inputs.
- LIME explanation marcotcr/lime: Lime: Explaining the predictions of any machine learning classifier (github.com)
id:: 6571dab5-f6fe-42e0-b35d-ed1f52625070
- LIME seems to be the most used one and it's very generic
- Example: A recommender has been trained so that it can recommend A, B and the malicious D. I could use LIME to identify that A and B are the sources for recomming D and then I can refine the data
-
Human-in-the-loop RSSEs
- How to create IDEs context, as smarter as possible, to interact with ChatGPT
- How to create prompts or taxonomy of prompt templates for improving the accuracy of RSSEs
- The context can be not only the files, but also the sequence of files that are opened by developers
- Avoid that these tools help attackers to write attacks (like DDOS attacks)
- We can replicate what we have done with the feedback paper with LLMs (we need to understand how to encode, positive, negative, and additive feedback) Ideas
-
-
