Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks

1 Introduction

1. What weaknesses of purely parametric language models does RAG address?
   Knowledge stored only in parameters is hard to expand or revise, gives no insight into predictions (no provenance), and can produce hallucinations; on knowledge-intensive tasks such models lagged task-specific architectures.
   • Lewis et al., 2020, Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (Advances in Neural Information Processing Systems 33 (NeurIPS 2020)), §1
2. What is retrieval-augmented generation (RAG)?
   A general fine-tuning recipe combining parametric memory (a pretrained seq2seq model, BART) with non-parametric memory (a dense vector index of Wikipedia accessed by a pretrained neural retriever, DPR), trained end to end with retrieved documents as latent variables.

   • Lewis et al., 2020, Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (Advances in Neural Information Processing Systems 33 (NeurIPS 2020)), §1
3. How does RAG differ from REALM and ORQA?
   Those combined masked language models with a differentiable retriever but only for extractive open-domain QA; RAG brings retrieval to seq2seq generation, so one architecture serves QA, generation and classification. Both memories are pretrained, so access to knowledge works without extra training.
   • Lewis et al., 2020, Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (Advances in Neural Information Processing Systems 33 (NeurIPS 2020)), §1
4. What are knowledge-intensive tasks?
   Tasks that humans couldn’t reasonably be expected to perform without access to an external knowledge source, such as open-domain question answering or fact verification.
   • Lewis et al., 2020, Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (Advances in Neural Information Processing Systems 33 (NeurIPS 2020)), §1