LIMA: Less Is More for Alignment
Zhou et al., 2023, LIMA (Advances in Neural Information Processing Systems 36 (NeurIPS 2023))
Subscriber deck — free during early access
1 cards in “5 Why is Less More? Ablations on Data Diversity, Quality, and Quantity” · whole deck
5 Why is Less More? Ablations on Data Diversity, Quality, and Quantity
- What did the ablations on diversity, quality and quantity show?With LLaMA 7B and 2,000 examples: diverse Stack Exchange prompts beat homogeneous wikiHow “how to” prompts; quality-filtered answers beat unfiltered ones by about 0.5 on a 6-point scale; but increasing the amount of data up to 16× gave no gain.
- Zhou et al., 2023, LIMA (Advances in Neural Information Processing Systems 36 (NeurIPS 2023)), §5
Source
- Chunting Zhou, Pengfei Liu, Puxin Xu, Srini Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, Susan Zhang, Gargi Ghosh, Mike Lewis and Omer Levy. 2023. LIMA: Less Is More for Alignment. In Advances in Neural Information Processing Systems 36 (NeurIPS 2023). https://papers.nips.cc/paper_files/paper/2023/hash/ac662d74829e4407ce1d126477f4a03a-Abstract-Conference.html