learn.ifs.name

Toolformer: Language Models Can Teach Themselves to Use Tools

Schick et al., 2023, Toolformer (Advances in Neural Information Processing Systems 36 (NeurIPS 2023))

Subscriber deck — free during early access

16 cards

1 Introduction

  1. What paradox motivates Toolformer?
    Large language models solve new tasks from a few examples, yet struggle with basic things that small specialized tools do well: precise arithmetic, factual lookup, up-to-date information, low-resource languages and awareness of the current date.
    • Schick et al., 2023, Toolformer (Advances in Neural Information Processing Systems 36 (NeurIPS 2023)), §1
  2. What two requirements did Schick et al. set for learning tool use?
    It should be self-supervised, without large amounts of human annotation (which is costly, and what humans find useful may differ from what helps the model); and the model should keep its generality and decide for itself when and how to use which tool, not only in task-specific settings.
    • Schick et al., 2023, Toolformer (Advances in Neural Information Processing Systems 36 (NeurIPS 2023)), §1

2 Approach

  1. How are API calls represented in Toolformer’s text?
    As linearized sequences \(e(c, r) = \) <API> \(a_c(i_c) \to r\) </API>, with API name \(a_c\), input \(i_c\) and result \(r\), inserted inline in the text; in practice the special tokens are written as “[”, “]” and “->”, so the vocabulary needn’t change.
    • Schick et al., 2023, Toolformer (Advances in Neural Information Processing Systems 36 (NeurIPS 2023)), §2
  2. Outline the steps of Toolformer’s data generation.
    (1) With a few-shot prompt per tool, let the model itself propose API calls at positions where it assigns high probability to starting one; (2) execute the calls; (3) keep only calls whose results make the following tokens easier to predict; (4) merge the kept calls into the original texts and fine-tune the model on this augmented dataset.
    plain text from the corpus1 model samples API calls2 execute the calls3 keep calls that cut the loss4 fine-tune on text + callsdropsame texts, now with useful tool calls
    Self-supervised tool learning: the model labels its own training data
    • Schick et al., 2023, Toolformer (Advances in Neural Information Processing Systems 36 (NeurIPS 2023)), §2
  3. What is Toolformer’s filtering criterion for API calls?
    Compare the weighted loss on the following tokens with the call and its result as prefix, \(L_i^+\), against the smaller of the losses with no call and with the call but no result, \(L_i^-\). Keep the call only if \(L_i^- - L_i^+ \ge \tau_f\): input and output together must reduce the loss by at least a threshold.
    • Schick et al., 2023, Toolformer (Advances in Neural Information Processing Systems 36 (NeurIPS 2023)), §2
  4. Why does Toolformer compare against “call without result” as well as “no call”?
    To make sure it is the tool’s output that helps: a call whose input alone (e.g. a question restating useful words) lowers the loss would otherwise be kept even though the tool contributes nothing.
    • Schick et al., 2023, Toolformer (Advances in Neural Information Processing Systems 36 (NeurIPS 2023)), §2
  5. Why does fine-tuning on Toolformer’s augmented data preserve the model’s general abilities?
    The augmented dataset contains exactly the same texts as the original (here a subset of CCNet, like the pretraining data), with API calls added only where they help, so the model sees the same content while learning when and how to call tools.
    • Schick et al., 2023, Toolformer (Advances in Neural Information Processing Systems 36 (NeurIPS 2023)), §2
  6. How does Toolformer call a tool during generation?
    It decodes normally until it produces the “→” token, then decoding pauses, the API is called, and its response and the closing </API> token are inserted before decoding continues.
    • Schick et al., 2023, Toolformer (Advances in Neural Information Processing Systems 36 (NeurIPS 2023)), §2

3 Tools

  1. Which tools did Toolformer learn to use?
    A question answering system (Atlas), a calculator for the four basic operations, a BM25 Wikipedia search engine returning snippets, a machine translation system into English (NLLB 600M) and a calendar returning the current date. The only requirements: text input and output, and a few demonstrations.
    • Schick et al., 2023, Toolformer (Advances in Neural Information Processing Systems 36 (NeurIPS 2023)), §3

4.1 Experimental Setup

  1. What models did Schick et al. compare to isolate the effect of tools?
    GPT-J (6.7B); GPT-J fine-tuned on the same CCNet subset without API calls; Toolformer; Toolformer with API calls disabled at decoding; and much larger OPT-66B and GPT-3 175B, all prompted zero-shot without task-specific examples of tool use.
    • Schick et al., 2023, Toolformer (Advances in Neural Information Processing Systems 36 (NeurIPS 2023)), §4.1

4.2 Downstream Tasks

  1. How did Toolformer do on factual completion (LAMA) and math word problems?
    On LAMA it called the QA tool in 98% of cases and beat same-size baselines by 5–19 points, outperforming OPT-66B and GPT-3. On math benchmarks it called the calculator for 98% of examples, more than doubling GPT-J’s scores and beating OPT and GPT-3.
    • Schick et al., 2023, Toolformer (Advances in Neural Information Processing Systems 36 (NeurIPS 2023)), §4.2
  2. Where did Toolformer still fall short of GPT-3, and why?
    On open question answering (with the QA tool disabled) it relied on Wikipedia search and beat same-size models but trailed GPT-3: the BM25 search often returned poor matches, and Toolformer couldn’t reformulate queries or browse several results.
    • Schick et al., 2023, Toolformer (Advances in Neural Information Processing Systems 36 (NeurIPS 2023)), §4.2
  3. What did the temporal benchmarks reveal about Toolformer’s tool choices?
    On DateSet (questions like “what day was it 30 days ago?”) its gains came from the calendar, used for 55% of examples. On TempLAMA it improved but barely used the calendar (0.2%); the ideal chain (get the date, then query the QA tool) was impossible with one call per input.
    • Schick et al., 2023, Toolformer (Advances in Neural Information Processing Systems 36 (NeurIPS 2023)), §4.2

4.3 Language Modeling

  1. Did learning tool use hurt Toolformer’s language modeling?
    No: with API calls disabled, its perplexity on WikiText and held-out CCNet was about the same as GPT-J fine-tuned on the same data without calls.
    • Schick et al., 2023, Toolformer (Advances in Neural Information Processing Systems 36 (NeurIPS 2023)), §4.3

4.4 Scaling Laws

  1. How did the ability to use tools depend on model size?
    Applied to GPT-2 models from 124M to 1.6B parameters, useful tool use emerged only around 775M; smaller models did about the same with and without tools. Larger models got better without tools, but also better at using them, so the gap persisted.
    • Schick et al., 2023, Toolformer (Advances in Neural Information Processing Systems 36 (NeurIPS 2023)), §4.4

6 Limitations

  1. What are Toolformer’s main limitations?
    It can’t chain tools (calls were sampled independently, one per example), can’t use tools interactively (e.g. refine a search), is sensitive to the input wording when deciding to call, is sample-inefficient for some tools (a million documents yield a few thousand useful calculator calls), and ignores the cost of calls.
    • Schick et al., 2023, Toolformer (Advances in Neural Information Processing Systems 36 (NeurIPS 2023)), §6

Source