learn.ifs.name

LoRA: Low-Rank Adaptation of Large Language Models

Hu et al., 2022, LoRA (The Tenth International Conference on Learning Representations (ICLR 2022))

Subscriber deck — free during early access

2 cards in “7.2 What is the Optimal Rank r for LoRA?” · whole deck

7.2 What is the Optimal Rank r for LoRA?

  1. How small can the LoRA rank be?
    Surprisingly small: for \(W_q\) and \(W_v\) on GPT-3, rank 1 already matched higher ranks on WikiSQL and MNLI, while \(W_q\) alone needed larger \(r\). Small ranks aren’t expected to work for every task, e.g. adapting to a language unlike the pre-training data.
    • Hu et al., 2022, LoRA (The Tenth International Conference on Learning Representations (ICLR 2022)), §7.2
  2. What did subspace-similarity analysis reveal about LoRA updates?
    The top singular directions of updates learned with \(r = 8\) and \(r = 64\) (and with different seeds) overlap strongly, while the rest look like noise: the useful update lives in a very low-dimensional subspace, which explains why \(r = 1\) works.
    • Hu et al., 2022, LoRA (The Tenth International Conference on Learning Representations (ICLR 2022)), §7.2

Source

  • Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang and Weizhu Chen. 2022. LoRA: Low-Rank Adaptation of Large Language Models. In The Tenth International Conference on Learning Representations (ICLR 2022). https://arxiv.org/abs/2106.09685