Information Retrieval and Retrieval-Augmented Generation

11.3 Dense retrieval

1. What is the vocabulary mismatch problem?
   Sparse methods like tf-idf and BM25 only work when query and document share exact words, so the user must guess the words the answer’s writer used. Dense embeddings handle synonymy and solve this.
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 11, §11.3, p. 264
2. How does a cross-encoder (full-interaction) model score a query–document pair?
   Feed both to one encoder, “query [SEP] document”, so self-attention sees all tokens of both, then a linear layer on the [CLS] output predicts a relevance score. Documents are usually split into passages (e.g. ~100-token chunks) to fit the window.
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 11, §11.3, p. 264
3. Why can’t the cross-encoder be used to search a whole collection?
   Every new query would require running every document in the collection through the encoder jointly with it, which is far too expensive; it is used only for reranking.
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 11, §11.3, p. 265
4. What is a bi-encoder?
   Separate encoders for queries and documents: each document is encoded once in advance (e.g. its [CLS] vector) and stored; a new query is encoded and scored against all documents by dot product, \(\text{score} = z_q \cdot z_d\).

   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 11, §11.3, p. 265
5. Why is a bi-encoder less accurate than a cross-encoder?
   Its relevance decision can’t exploit interactions between query tokens and document tokens; each side is compressed into a vector independently.
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 11, §11.3, p. 265
6. What is a common way to combine cheap and expensive retrieval methods?
   Rank with a cheap first pass (e.g. BM25), then rerank only the top N documents with an expensive model like a full cross-encoder.
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 11, §11.3, p. 265
7. How does ColBERT score a query against a document?
   Encode query and document separately into one contextual vector per token (document vectors can be precomputed); for each query token find the most similar document token (MaxSim) and sum: \(\text{score}(q, d) = \sum_{i=1}^{N}\max_{j} E_{q_i}\cdot E_{d_j}\).
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 11, §11.3, p. 266
8. How are neural retrievers like ColBERT trained, and where do training examples come from?
   End-to-end with a cross-entropy loss on triples (query, positive document, negative document). Positives from datasets with gold relevant passages (e.g. MS MARCO), negatives sampled from an existing system’s top results; without labels, relevance-guided supervision iteratively harvests passages that do or don’t contain short answer strings.
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 11, §11.3, pp. 266–267
9. How is efficient search done with dense vectors?
   Finding the document vectors with the highest dot product with the query is a nearest neighbor search problem, solved with approximate nearest neighbor libraries like Faiss. (Sparse retrieval uses the inverted index.)
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 11, §11.3, p. 267