Reasoning Models Don’t Always Say What They Think
Chen et al., 2025, Reasoning Models Don’t Always Say What They Think (arXiv preprint arXiv:2505.05410)
Subscriber deck — free during early access
2 cards in “2.1 Metric” · whole deck
2.1 Metric
- How do Chen et al. measure chain-of-thought faithfulness?With prompt pairs: a multiple-choice question with and without a hint pointing to an answer. Keeping cases where the model changes from a non-hint answer to the hint answer once the hint is added, the score is the fraction whose chain of thought verbalizes the hint, i.e. mentions it and relies on it, normalized for random answer changes.
Faithfulness = share of hint-following answers whose reasoning admits the hint - Chen et al., 2025, Reasoning Models Don’t Always Say What They Think (arXiv preprint arXiv:2505.05410), §2.1
- Why is Chen et al.’s faithfulness score likely an upper bound?A chain of thought that uses the hint without mentioning it is clearly unfaithful, but one that mentions the hint may still hide other important reasons, so counting all verbalizations as faithful overestimates faithfulness.
- Chen et al., 2025, Reasoning Models Don’t Always Say What They Think (arXiv preprint arXiv:2505.05410), §2.1
Source
- Yanda Chen, Joe Benton, Ansh Radhakrishnan, Jonathan Uesato, Carson Denison, John Schulman, Arushi Somani, Peter Hase, Misha Wagner, Fabien Roger, Vlad Mikulik, Samuel R. Bowman, Jan Leike, Jared Kaplan and Ethan Perez. 2025. Reasoning Models Don’t Always Say What They Think. In arXiv preprint arXiv:2505.05410. https://arxiv.org/abs/2505.05410