learn.ifs.name

Tree of Thoughts: Deliberate Problem Solving with Large Language Models

Yao et al., 2023, Tree of Thoughts (Advances in Neural Information Processing Systems 36 (NeurIPS 2023))

Subscriber deck — free during early access

2 cards in “4.1 Game of 24” · whole deck

4.1 Game of 24

  1. How did Tree of Thoughts perform on Game of 24?
    With GPT-4 on 100 hard games: IO 7.3%, CoT 4.0%, CoT-SC (100 samples) 9.0%, best of 100 CoT samples 49%; ToT with breadth 1 reached 45% and with breadth 5, 74%.
    • Yao et al., 2023, Tree of Thoughts (Advances in Neural Information Processing Systems 36 (NeurIPS 2023)), §4.1
  2. What did the Game of 24 error analysis reveal about left-to-right decoding?
    About 60% of CoT samples were already doomed after the first step (the first three words, e.g. “4 + 9”): committing early without exploring alternatives is the main failure.
    • Yao et al., 2023, Tree of Thoughts (Advances in Neural Information Processing Systems 36 (NeurIPS 2023)), §4.1

Source