LIMA: Less Is More for Alignment
Zhou et al., 2023, LIMA (Advances in Neural Information Processing Systems 36 (NeurIPS 2023))
Subscriber deck — free during early access
1 cards in “3 Training LIMA” · whole deck
3 Training LIMA
- What training details distinguish LIMA’s fine-tuning?A new end-of-turn token separating speakers (to avoid the pretrained meaning of end-of-sequence); 15 epochs; residual dropout rising from 0 at the bottom layer to 0.3 at the top; and checkpoints chosen by hand on a 50-example development set, because perplexity didn’t correlate with generation quality.
- Zhou et al., 2023, LIMA (Advances in Neural Information Processing Systems 36 (NeurIPS 2023)), §3
Source
- Chunting Zhou, Pengfei Liu, Puxin Xu, Srini Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, Susan Zhang, Gargi Ghosh, Mike Lewis and Omer Levy. 2023. LIMA: Less Is More for Alignment. In Advances in Neural Information Processing Systems 36 (NeurIPS 2023). https://papers.nips.cc/paper_files/paper/2023/hash/ac662d74829e4407ce1d126477f4a03a-Abstract-Conference.html