LoRA: Low-Rank Adaptation of Large Language Models

1 Introduction

1. What problem does LoRA solve?
   Full fine-tuning updates all parameters, so each task needs its own copy of the model, prohibitively expensive at GPT-3’s 175B parameters. LoRA freezes the pretrained weights and trains small low-rank matrices instead, cutting trainable parameters up to 10,000× and GPU memory 3× while matching fine-tuning quality.
   • Hu et al., 2022, LoRA (The Tenth International Conference on Learning Representations (ICLR 2022)), §1
2. What hypothesis motivates LoRA?
   Over-parametrized models have a low intrinsic dimension (Aghajanyan et al.), so the change in weights during adaptation, \(\Delta W\), should also have a low “intrinsic rank”; then \(\Delta W\) can be learned as a product of two small matrices.
   • Hu et al., 2022, LoRA (The Tenth International Conference on Learning Representations (ICLR 2022)), §1
3. What practical advantages does LoRA have?
   One frozen model can serve many tasks by swapping small \(A\), \(B\) modules; training needs no gradients or optimizer states for most parameters (up to 3× lower hardware barrier); the update can be merged into the weights, so no extra inference latency; and it combines with other methods like prefix tuning.
   • Hu et al., 2022, LoRA (The Tenth International Conference on Learning Representations (ICLR 2022)), §1