learn.ifs.name

Tree of Thoughts: Deliberate Problem Solving with Large Language Models

Yao et al., 2023, Tree of Thoughts (Advances in Neural Information Processing Systems 36 (NeurIPS 2023))

Subscriber deck — free during early access

2 cards in “4.3 Mini crosswords” · whole deck

4.3 Mini crosswords

  1. How did Tree of Thoughts solve 5×5 mini crosswords?
    Depth-first search filling one clue at a time, ordering candidate words by the model’s confidence and pruning a state if any remaining clue seemed impossible given its letters. Word-level success rose from under 16% (IO, CoT) to 60%, solving 4 of 20 games.
    • Yao et al., 2023, Tree of Thoughts (Advances in Neural Information Processing Systems 36 (NeurIPS 2023)), §4.3
  2. What did the crossword ablations show?
    Backtracking matters: a greedy version without it got only 20% of words right. Pruning matters too but is imperfect: GPT-4 sometimes pruned correct states because of rare or obsolete words it didn’t recognize; outputting the best explored state instead of the heuristic choice solved 7 of 20.
    • Yao et al., 2023, Tree of Thoughts (Advances in Neural Information Processing Systems 36 (NeurIPS 2023)), §4.3

Source