LoRA: Low-Rank Adaptation of Large Language Models

4.1 Low-Rank-Parametrized Update Matrices

1. Give LoRA’s reparametrization of a weight matrix.
   For a frozen \(W_0 \in \mathbb{R}^{d \times k}\), the update is \(\Delta W = BA\) with \(B \in \mathbb{R}^{d \times r}\), \(A \in \mathbb{R}^{r \times k}\), \(r \ll \min(d, k)\), so \(h = W_0 x + \Delta W x = W_0 x + BAx\). \(A\) starts random Gaussian and \(B\) at zero, so \(\Delta W = 0\) initially; \(\Delta W x\) is scaled by \(\alpha / r\).

   • Hu et al., 2022, LoRA (The Tenth International Conference on Learning Representations (ICLR 2022)), §4.1
2. How many parameters does a rank-4 LoRA update of a \(12288 \times 12288\) matrix have, compared with the full matrix?
   \(r(d + k) = 4 \times 24{,}576 = 98{,}304\), versus \(12288^2 \approx 151\) million, about 1,500× fewer. Adapting \(W_q\) and \(W_v\) in all 96 GPT-3 layers gives \(2 \times 192 \times 12288 \times 4 \approx 18.9\)M parameters.
   • Hu et al., 2022, LoRA (The Tenth International Conference on Learning Representations (ICLR 2022)), §4.1
3. Why does LoRA scale \(\Delta W x\) by \(\alpha / r\)?
   With Adam, tuning \(\alpha\) is roughly like tuning the learning rate; fixing \(\alpha\) to the first \(r\) tried and not tuning it reduces the need to retune hyperparameters when \(r\) changes.
   • Hu et al., 2022, LoRA (The Tenth International Conference on Learning Representations (ICLR 2022)), §4.1
4. In what sense is LoRA a generalization of full fine-tuning?
   Applying LoRA to all weight matrices (and training biases) with rank equal to the weights’ rank roughly recovers full fine-tuning; as trainable parameters grow, LoRA converges to training the original model, while adapters converge to an MLP and prefix methods to a model that can’t take long inputs.
   • Hu et al., 2022, LoRA (The Tenth International Conference on Learning Representations (ICLR 2022)), §4.1
5. Why does LoRA add no inference latency, and how are tasks switched?
   For deployment one computes \(W = W_0 + BA\) and runs inference as usual. To switch tasks, subtract \(BA\) and add another task’s \(B'A'\), a quick operation with little memory overhead.
   • Hu et al., 2022, LoRA (The Tenth International Conference on Learning Representations (ICLR 2022)), §4.1