LoRA: Low-Rank Adaptation of Large Language Models

Full fine-tuning updates all parameters, so each task needs its own copy of the model, prohibitively expensive at GPT-3’s 175B parameters. LoRA freezes the pretrained weights and trains small low-rank matrices instead, cutting trainable parameters up to 10,000× and GPU memory 3× while matching fine-tuning quality.

Over-parametrized models have a low intrinsic dimension (Aghajanyan et al.), so the change in weights during adaptation, \(\Delta W\), should also have a low “intrinsic rank”; then \(\Delta W\) can be learned as a product of two small matrices.

One frozen model can serve many tasks by swapping small \(A\), \(B\) modules; training needs no gradients or optimizer states for most parameters (up to 3× lower hardware barrier); the update can be merged into the weights, so no extra inference latency; and it combines with other methods like prefix tuning.

Full fine-tuning maximizes \(\sum_{(x,y)} \sum_t \log P_\Phi(y_t \mid x, y_{

• Hu et al., 2022, LoRA (The Tenth International Conference on Learning Representations (ICLR 2022)), §2

They add depth that must be processed sequentially; even with few parameters and FLOPs, large models rely on parallelism, so with small online batches the extra layers noticeably slow inference (e.g. up to 20–30% for GPT-2 medium at batch size 1), and model sharding makes it worse.

It is hard to optimize, its performance changes non-monotonically with the number of trainable parameters, and reserving sequence positions for the prefix reduces the length available for the task input.

For a frozen \(W_0 \in \mathbb{R}^{d \times k}\), the update is \(\Delta W = BA\) with \(B \in \mathbb{R}^{d \times r}\), \(A \in \mathbb{R}^{r \times k}\), \(r \ll \min(d, k)\), so \(h = W_0 x + \Delta W x = W_0 x + BAx\). \(A\) starts random Gaussian and \(B\) at zero, so \(\Delta W = 0\) initially; \(\Delta W x\) is scaled by \(\alpha / r\).

\(r(d + k) = 4 \times 24{,}576 = 98{,}304\), versus \(12288^2 \approx 151\) million, about 1,500× fewer. Adapting \(W_q\) and \(W_v\) in all 96 GPT-3 layers gives \(2 \times 192 \times 12288 \times 4 \approx 18.9\)M parameters.

With Adam, tuning \(\alpha\) is roughly like tuning the learning rate; fixing \(\alpha\) to the first \(r\) tried and not tuning it reduces the need to retune hyperparameters when \(r\) changes.

Applying LoRA to all weight matrices (and training biases) with rank equal to the weights’ rank roughly recovers full fine-tuning; as trainable parameters grow, LoRA converges to training the original model, while adapters converge to an MLP and prefix methods to a model that can’t take long inputs.

For deployment one computes \(W = W_0 + BA\) and runs inference as usual. To switch tasks, subtract \(BA\) and add another task’s \(B'A'\), a quick operation with little memory overhead.

Training VRAM fell from 1.2 TB to 350 GB (no optimizer states for frozen weights); with \(r = 4\) on query and value projections the checkpoint shrank about 10,000× (350 GB to 35 MB), so 100 adapted models need about 354 GB instead of 35 TB; training was about 25% faster.

If \(A\) and \(B\) are merged into \(W\) to avoid latency, a single forward pass can’t easily batch inputs for different tasks with different \(A\), \(B\); without merging, modules can be chosen per example where latency matters less.

RoBERTa-base/large and DeBERTa-XXL on GLUE, GPT-2 medium/large on E2E NLG (plus WebNLG and DART), and GPT-3 175B on WikiSQL, MNLI-matched and SAMSum summarization, against full fine-tuning, BitFit (biases only), prefix-embedding and prefix-layer tuning, and several adapter variants.

With 4.7M trainable parameters it matched or beat fine-tuning (175B parameters) on all three data sets: WikiSQL 73.4 vs 73.8, MNLI-m 91.7 vs 89.5, SAMSum ROUGE-1 53.8 vs 52.0. Prefix methods dropped sharply with too many special tokens, while LoRA scaled better.

With 18M parameters on GPT-3, putting everything in \(W_q\) or \(W_k\) did worst; adapting both \(W_q\) and \(W_v\) (rank 4) did best, and all four with rank 2 equally well: it is better to adapt more weight types at low rank than one type at higher rank.

Surprisingly small: for \(W_q\) and \(W_v\) on GPT-3, rank 1 already matched higher ranks on WikiSQL and MNLI, while \(W_q\) alone needed larger \(r\). Small ranks aren’t expected to work for every task, e.g. adapting to a language unlike the pre-training data.

The top singular directions of updates learned with \(r = 8\) and \(r = 64\) (and with different seeds) overlap strongly, while the rest look like noise: the useful update lives in a very low-dimensional subspace, which explains why \(r = 1\) works.

It correlates with \(W\) more than a random matrix but doesn’t repeat \(W\)’s top singular directions: it amplifies directions already present but not emphasized in \(W\), by a large factor (about 21× for \(r = 4\)), suggesting adaptation amplifies task-relevant features learned but under-weighted in pre-training.