Toolformer: Language Models Can Teach Themselves to Use Tools

4.2 Downstream Tasks

1. How did Toolformer do on factual completion (LAMA) and math word problems?
   On LAMA it called the QA tool in 98% of cases and beat same-size baselines by 5–19 points, outperforming OPT-66B and GPT-3. On math benchmarks it called the calculator for 98% of examples, more than doubling GPT-J’s scores and beating OPT and GPT-3.
   • Schick et al., 2023, Toolformer (Advances in Neural Information Processing Systems 36 (NeurIPS 2023)), §4.2
2. Where did Toolformer still fall short of GPT-3, and why?
   On open question answering (with the QA tool disabled) it relied on Wikipedia search and beat same-size models but trailed GPT-3: the BM25 search often returned poor matches, and Toolformer couldn’t reformulate queries or browse several results.
   • Schick et al., 2023, Toolformer (Advances in Neural Information Processing Systems 36 (NeurIPS 2023)), §4.2
3. What did the temporal benchmarks reveal about Toolformer’s tool choices?
   On DateSet (questions like “what day was it 30 days ago?”) its gains came from the calendar, used for 55% of examples. On TempLAMA it improved but barely used the calendar (0.2%); the ideal chain (get the date, then query the QA tool) was impossible with one call per input.
   • Schick et al., 2023, Toolformer (Advances in Neural Information Processing Systems 36 (NeurIPS 2023)), §4.2