Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks

4.3 Jeopardy Question Generation

1. What did human evaluation of Jeopardy question generation show?
   Against BART, RAG-Token was judged more factual in 42.7% of pairs vs 7.1% for BART (both factual in 17%), and much more specific. RAG-Token beat RAG-Sequence on Q-BLEU-1, perhaps because Jeopardy clues combine facts from several documents.
   • Lewis et al., 2020, Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (Advances in Neural Information Processing Systems 33 (NeurIPS 2020)), §4.3
2. How do parametric and non-parametric memory interact in RAG-Token?
   Generating a Hemingway clue, the document posterior peaks on the passage about “The Sun Also Rises” when the first title token is produced, then flattens: retrieved text cues the right knowledge, and BART’s parametric memory completes the title.
   • Lewis et al., 2020, Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (Advances in Neural Information Processing Systems 33 (NeurIPS 2020)), §4.3