learn.ifs.name

LoRA: Low-Rank Adaptation of Large Language Models

Hu et al., 2022, LoRA (The Tenth International Conference on Learning Representations (ICLR 2022))

Subscriber deck — free during early access

1 cards in “7.3 How Does the Adaptation Matrix ΔW Compare to W?” · whole deck

7.3 How Does the Adaptation Matrix ΔW Compare to W?

  1. How does \(\Delta W\) relate to the pretrained \(W\)?
    It correlates with \(W\) more than a random matrix but doesn’t repeat \(W\)’s top singular directions: it amplifies directions already present but not emphasized in \(W\), by a large factor (about 21× for \(r = 4\)), suggesting adaptation amplifies task-relevant features learned but under-weighted in pre-training.
    • Hu et al., 2022, LoRA (The Tenth International Conference on Learning Representations (ICLR 2022)), §7.3

Source

  • Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang and Weizhu Chen. 2022. LoRA: Low-Rank Adaptation of Large Language Models. In The Tenth International Conference on Learning Representations (ICLR 2022). https://arxiv.org/abs/2106.09685