Chain-of-Thought Prompting Elicits Reasoning in Large Language Models

3.2 Results

1. In what sense is chain-of-thought reasoning an emergent ability of scale?
   It doesn’t help small models, and only gives gains at around 100B parameters; smaller models produced fluent but illogical chains of thought and did worse than with standard prompting.
   • Wei et al., 2022, Chain-of-Thought Prompting Elicits Reasoning in Large Language Models (Advances in Neural Information Processing Systems 35 (NeurIPS 2022)), §3.2
2. For which problems does chain of thought help most?
   More complicated ones: on GSM8K, with the lowest baselines, performance more than doubled for the largest GPT and PaLM models; on MAWPS SingleOp, solvable in one step, gains were negative or very small.
   • Wei et al., 2022, Chain-of-Thought Prompting Elicits Reasoning in Large Language Models (Advances in Neural Information Processing Systems 35 (NeurIPS 2022)), §3.2
3. What did manual analysis of LaMDA 137B’s chains of thought on GSM8K show?
   Of 50 correct answers, all but two had logically and mathematically correct chains (two were right by coincidence). Of 50 wrong answers, 46% had nearly correct chains with minor errors (calculator error, symbol mapping, one missing step) and 54% had major semantic or coherence errors; scaling PaLM from 62B to 540B fixed many one-step-missing and semantic errors.
   • Wei et al., 2022, Chain-of-Thought Prompting Elicits Reasoning in Large Language Models (Advances in Neural Information Processing Systems 35 (NeurIPS 2022)), §3.2