Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks

Knowledge stored only in parameters is hard to expand or revise, gives no insight into predictions (no provenance), and can produce hallucinations; on knowledge-intensive tasks such models lagged task-specific architectures.

A general fine-tuning recipe combining parametric memory (a pretrained seq2seq model, BART) with non-parametric memory (a dense vector index of Wikipedia accessed by a pretrained neural retriever, DPR), trained end to end with retrieved documents as latent variables.

Those combined masked language models with a differentiable retriever but only for extractive open-domain QA; RAG brings retrieval to seq2seq generation, so one architecture serves QA, generation and classification. Both memories are pretrained, so access to knowledge works without extra training.

Tasks that humans couldn’t reasonably be expected to perform without access to an external knowledge source, such as open-domain question answering or fact verification.

The same retrieved document generates the whole output: \(p(y \mid x) \approx \sum_{z \in \text{top-}k} p_\eta(z \mid x) \prod_{i=1}^{N} p_\theta(y_i \mid x, z, y_{1:i-1})\), marginalizing over the top-\(k\) documents once per sequence.

Each output token can come from a different document: \(p(y \mid x) \approx \prod_{i=1}^{N} \sum_{z \in \text{top-}k} p_\eta(z \mid x)\, p_\theta(y_i \mid x, z, y_{1:i-1})\). It lets the generator combine content from several documents in one answer; for classification (one-token targets) the two variants coincide.

Dense Passage Retrieval, a bi-encoder: \(p_\eta(z \mid x) \propto \exp(d(z)^\top q(x))\), with \(d(z) = \text{BERT}_d(z)\) and \(q(x) = \text{BERT}_q(x)\), both BERT-base. Finding the top-\(k\) documents is maximum inner product search (MIPS), solved approximately in sublinear time.

BART-large (400M parameters), a denoising-pretrained seq2seq Transformer; the input \(x\) and retrieved passage \(z\) are simply concatenated as its input.

By minimizing the negative marginal log-likelihood \(-\sum_j \log p(y_j \mid x_j)\) with Adam, with no supervision on which document to retrieve. Only the query encoder and BART are fine-tuned; the document encoder and index stay fixed, avoiding costly re-indexing (unlike REALM), without hurting performance.

RAG-Token is an ordinary autoregressive model with per-step marginalized probabilities, so standard beam search works. RAG-Sequence’s likelihood doesn’t factor per token: run beam search per document, then score each candidate across documents (“thorough decoding”, extra forward passes) or approximate missing scores as zero (“fast decoding”).

A December 2018 Wikipedia dump split into disjoint 100-word chunks, 21M passages, embedded by the DPR document encoder and indexed with FAISS (HNSW approximation); 5 or 10 documents were retrieved per query in training.

New state of the art on Natural Questions (44.5 EM), WebQuestions (45.5), CuratedTrec (52.2) and the T5-comparable TriviaQA split (68.0), beating closed-book T5-11B+SSM and extractive DPR, without salient-span-masking pre-training or a re-ranker.

Documents that contain clues but not the verbatim answer can still contribute, and RAG can produce correct answers absent from all retrieved documents: 11.8% accuracy in such cases on NQ, where an extractive model scores 0.

RAG-Sequence beat BART by 2.6 BLEU and 2.6 ROUGE-L and approached systems that use the gold passages, hallucinating less and producing factually correct text more often.

Against BART, RAG-Token was judged more factual in 42.7% of pairs vs 7.1% for BART (both factual in 17%), and much more specific. RAG-Token beat RAG-Sequence on Q-BLEU-1, perhaps because Jeopardy clues combine facts from several documents.

Generating a Hemingway clue, the document posterior peaks on the passage about “The Sun Also Rises” when the first title token is produced, then flattens: retrieved text cues the right knowledge, and BART’s parametric memory completes the title.

Within 4.3% of state-of-the-art pipelines on 3-way classification and within 2.7% of a RoBERTa classifier given gold evidence on 2-way, without any supervision on retrieved evidence; its top document came from a gold article 71% of the time.

Learning the retriever improved all tasks over freezing it. Replacing it with BM25 hurt everywhere except FEVER, whose entity-centric claims suit word-overlap retrieval; differentiable retrieval mattered most for open-domain QA.

By hot-swapping the index: with a 2016 Wikipedia index RAG answered 70% of “Who is {position}?” questions correctly for 2016 world leaders, with the 2018 index 68% for 2018 leaders, while mismatched index and year gave 12% and 4%.

It is human-readable, giving a form of interpretability and provenance, and human-writable: the model’s knowledge can be updated by editing the document index.