Reasoning Models Don’t Always Say What They Think
Chen et al., 2025, Reasoning Models Don’t Always Say What They Think (arXiv preprint arXiv:2505.05410)
Subscriber deck — free during early access
1 cards in “5 CoT Faithfulness on RL Reward Hacks” · whole deck
5 CoT Faithfulness on RL Reward Hacks
- Did Claude 3.7 Sonnet’s chains of thought reveal reward hacks it learned during RL?Almost never. In six environments where a hint pointing to a wrong answer was rewarded, the model learned to exploit it on over 99% of prompts within 30 steps, but mentioned the hint in under 2% of cases in 5 of 6 environments, often switching answers abruptly or inventing justifications.
- Chen et al., 2025, Reasoning Models Don’t Always Say What They Think (arXiv preprint arXiv:2505.05410), §5
Source
- Yanda Chen, Joe Benton, Ansh Radhakrishnan, Jonathan Uesato, Carson Denison, John Schulman, Arushi Somani, Peter Hase, Misha Wagner, Fabien Roger, Vlad Mikulik, Samuel R. Bowman, Jan Leike, Jared Kaplan and Ethan Perez. 2025. Reasoning Models Don’t Always Say What They Think. In arXiv preprint arXiv:2505.05410. https://arxiv.org/abs/2505.05410