learn.ifs.name

LoRA: Low-Rank Adaptation of Large Language Models

Hu et al., 2022, LoRA (The Tenth International Conference on Learning Representations (ICLR 2022))

Subscriber deck — free during early access

1 cards in “7.1 Which Weight Matrices in Transformer Should We Apply LoRA to?” · whole deck

7.1 Which Weight Matrices in Transformer Should We Apply LoRA to?

  1. Which attention weights should LoRA adapt under a fixed parameter budget?
    With 18M parameters on GPT-3, putting everything in \(W_q\) or \(W_k\) did worst; adapting both \(W_q\) and \(W_v\) (rank 4) did best, and all four with rank 2 equally well: it is better to adapt more weight types at low rank than one type at higher rank.
    • Hu et al., 2022, LoRA (The Tenth International Conference on Learning Representations (ICLR 2022)), §7.1

Source

  • Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang and Weizhu Chen. 2022. LoRA: Low-Rank Adaptation of Large Language Models. In The Tenth International Conference on Learning Representations (ICLR 2022). https://arxiv.org/abs/2106.09685