Self-Consistency Improves Chain of Thought Reasoning in Language Models

3.5 Additional Studies

1. Why can self-consistency serve as an uncertainty estimate?
   The share of sampled paths agreeing with the final answer is highly correlated with accuracy, so low consistency signals low confidence: the model gains some ability to “know when it doesn’t know”.
   • Wang et al., 2023, Self-Consistency Improves Chain of Thought Reasoning in Language Models (The Eleventh International Conference on Learning Representations (ICLR 2023)), §3.5
2. How robust is self-consistency to prompts and reasoning formats?
   It helps with imperfect prompts (exemplars with wrong intermediate numbers), with equation-only reasoning (less gain, since short equations leave less room for diversity), and with zero-shot chain-of-thought (+26.2 points on GSM8K with PaLM-540B), and across sampling settings and model sizes.
   • Wang et al., 2023, Self-Consistency Improves Chain of Thought Reasoning in Language Models (The Eleventh International Conference on Learning Representations (ICLR 2023)), §3.5