logseq/pages/@Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks.md

links:: Local library, Web library authors:: Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, Douwe Kiela tags:: Computer Science - Computation and Language, Computer Science - Machine Learning, #zotero date:: 12-04-2021 item-type:: preprint title:: @Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks

• Abstract
  • Large pre-trained language models have been shown to store factual knowledge in their parameters, and achieve state-of-the-art results when fine-tuned on downstream NLP tasks. However, their ability to access and precisely manipulate knowledge is still limited, and hence on knowledge-intensive tasks, their performance lags behind task-specific architectures. Additionally, providing provenance for their decisions and updating their world knowledge remain open research problems. Pretrained models with a differentiable access mechanism to explicit non-parametric memory have so far been only investigated for extractive downstream tasks. We explore a general-purpose fine-tuning recipe for retrieval-augmented generation (RAG) — models which combine pre-trained parametric and non-parametric memory for language generation. We introduce RAG models where the parametric memory is a pre-trained seq2seq model and the non-parametric memory is a dense vector index of Wikipedia, accessed with a pre-trained neural retriever. We compare two RAG formulations, one which conditions on the same retrieved passages across the whole generated sequence, and another which can use different passages per token. We fine-tune and evaluate our models on a wide range of knowledgeintensive NLP tasks and set the state of the art on three open domain QA tasks, outperforming parametric seq2seq models and task-specific retrieve-and-extract architectures. For language generation tasks, we find that RAG models generate more specific, diverse and factual language than a state-of-the-art parametric-only seq2seq baseline.

• Attachments

  • Lewis et al. - 2021 - Retrieval-Augmented Generation for Knowledge-Inten.pdf {{zotero-imported-file 67EMVX36, "Lewis et al. - 2021 - Retrieval-Augmented Generation for Knowledge-Inten.pdf"}}

• Notes

  • Comment: Accepted at NeurIPS 2020