Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection

3.2.2 Training the Generator Model

1. How is the generator taught to produce reflection tokens itself?
   The critic and a retriever annotate training outputs offline: retrieval tokens, retrieved passages and critique tokens are inserted where they apply. The generator is trained with ordinary next-token prediction on this augmented corpus (retrieved text masked from the loss, vocabulary extended with reflection tokens), so no critic is needed at inference.
   • Asai et al., 2024, Self-RAG (The Twelfth International Conference on Learning Representations (ICLR 2024)), §3.2.2
2. Why is Self-RAG’s training cheaper than RLHF?
   Critique is computed once, offline, and inserted into the training text; the generator then learns with a standard language-modelling objective instead of PPO, which needs separate reward models during training.
   • Asai et al., 2024, Self-RAG (The Twelfth International Conference on Learning Representations (ICLR 2024)), §3.2.2