learn.ifs.name

LoRA: Low-Rank Adaptation of Large Language Models

Hu et al., 2022, LoRA (The Tenth International Conference on Learning Representations (ICLR 2022))

Subscriber deck — free during early access

2 cards in “4.2 Applying LoRA to Transformer” · whole deck

4.2 Applying LoRA to Transformer

  1. What did LoRA save when adapting GPT-3 175B?
    Training VRAM fell from 1.2 TB to 350 GB (no optimizer states for frozen weights); with \(r = 4\) on query and value projections the checkpoint shrank about 10,000× (350 GB to 35 MB), so 100 adapted models need about 354 GB instead of 35 TB; training was about 25% faster.
    • Hu et al., 2022, LoRA (The Tenth International Conference on Learning Representations (ICLR 2022)), §4.2
  2. What is a limitation of merging LoRA weights?
    If \(A\) and \(B\) are merged into \(W\) to avoid latency, a single forward pass can’t easily batch inputs for different tasks with different \(A\), \(B\); without merging, modules can be chosen per example where latency matters less.
    • Hu et al., 2022, LoRA (The Tenth International Conference on Learning Representations (ICLR 2022)), §4.2

Source

  • Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang and Weizhu Chen. 2022. LoRA: Low-Rank Adaptation of Large Language Models. In The Tenth International Conference on Learning Representations (ICLR 2022). https://arxiv.org/abs/2106.09685