Chain-of-Thought Prompting Elicits Reasoning in Large Language Models

6 Discussion

1. What does chain-of-thought prompting imply about standard prompting as a measure of model ability?
   Standard prompting gives only a lower bound: on many reasoning tasks it has flat scaling curves, while chain-of-thought prompting turns them into steeply rising ones, expanding the set of tasks large models can do.
   • Wei et al., 2022, Chain-of-Thought Prompting Elicits Reasoning in Large Language Models (Advances in Neural Information Processing Systems 35 (NeurIPS 2022)), §6
2. What limitations of chain-of-thought prompting did Wei et al. acknowledge?
   It doesn’t answer whether the network is actually “reasoning”; annotation could be costly for fine-tuning; reasoning paths aren’t guaranteed correct, leading to both right and wrong answers; and it emerges only at large scale, which is costly to serve.
   • Wei et al., 2022, Chain-of-Thought Prompting Elicits Reasoning in Large Language Models (Advances in Neural Information Processing Systems 35 (NeurIPS 2022)), §6