learn.ifs.name

LoRA: Low-Rank Adaptation of Large Language Models

Hu et al., 2022, LoRA (The Tenth International Conference on Learning Representations (ICLR 2022))

Subscriber deck — free during early access

1 cards in “5.5 Scaling up to GPT-3 175B” · whole deck

5.5 Scaling up to GPT-3 175B

  1. How did LoRA compare with full fine-tuning on GPT-3 175B?
    With 4.7M trainable parameters it matched or beat fine-tuning (175B parameters) on all three data sets: WikiSQL 73.4 vs 73.8, MNLI-m 91.7 vs 89.5, SAMSum ROUGE-1 53.8 vs 52.0. Prefix methods dropped sharply with too many special tokens, while LoRA scaled better.
    • Hu et al., 2022, LoRA (The Tenth International Conference on Learning Representations (ICLR 2022)), §5.5

Source

  • Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang and Weizhu Chen. 2022. LoRA: Low-Rank Adaptation of Large Language Models. In The Tenth International Conference on Learning Representations (ICLR 2022). https://arxiv.org/abs/2106.09685