Self-Consistency Improves Chain of Thought Reasoning in Language Models

3.2 Main Results

1. What gains did self-consistency give over chain-of-thought prompting with greedy decoding?
   Large gains on all four models, e.g. GSM8K +17.9, SVAMP +11.0, AQuA +12.2, StrategyQA +6.4 and ARC-challenge +3.9 points (absolute), reaching new state of the art on most tasks without task-specific training.
   • Wang et al., 2023, Self-Consistency Improves Chain of Thought Reasoning in Language Models (The Eleventh International Conference on Learning Representations (ICLR 2023)), §3.2
2. How did self-consistency’s gains depend on model scale?
   They grew with scale: about +3–6 points for UL2-20B but +9–23 for LaMDA-137B and GPT-3; for small models gains are lower because abilities such as arithmetic only emerge at sufficient scale.
   • Wang et al., 2023, Self-Consistency Improves Chain of Thought Reasoning in Language Models (The Eleventh International Conference on Learning Representations (ICLR 2023)), §3.2
3. How does accuracy change with the number of sampled paths?
   It rises consistently from 1 to 40 paths, but most of the gain comes early: in practice 5 or 10 paths capture most of the benefit at modest cost.
   • Wang et al., 2023, Self-Consistency Improves Chain of Thought Reasoning in Language Models (The Eleventh International Conference on Learning Representations (ICLR 2023)), §3.2