QLoRA: Efficient Finetuning of Quantized LLMs

Fine-tuning a 65B-parameter model on a single 48 GB GPU (instead of more than 780 GB for 16-bit fine-tuning) without losing 16-bit fine-tuning performance, by backpropagating through a frozen 4-bit quantized model into LoRA adapters.

Quantization methods had reduced memory only for inference; they broke down during training. QLoRA was the first to fine-tune a 4-bit model without performance degradation.

4-bit NormalFloat (NF4), a data type suited to normally distributed weights; Double Quantization, which quantizes the quantization constants; and Paged Optimizers, which move optimizer states between GPU and CPU memory to survive memory spikes.

Scaling a tensor by its absolute maximum to fit the low-bit range wastes bins when there are outliers. Splitting the tensor into small blocks, each quantized with its own constant, confines an outlier’s effect to its block.

Weights are stored in 4-bit NF4; whenever a weight is used it is dequantized to 16-bit BrainFloat for the matrix multiplication. Gradients flow through the dequantized weights, but only the LoRA adapter parameters (in BF16) receive weight updates.

Quantile quantization: give each quantization bin an equal share of values. Because pretrained weights are roughly zero-centred normal, the bins can be computed once from \(\mathcal{N}(0,1)\) quantiles and normalized to \([-1, 1]\); weight blocks are rescaled to that range, avoiding expensive per-tensor quantile estimation.

A symmetric quantile scheme has no exact zero. NF4 is built asymmetrically from separate quantiles for the negative and positive halves, sharing one zero, so padding and other zero values are represented exactly while all 16 codes are used.

With blocks of 64 weights and 32-bit constants, constants cost 32/64 = 0.5 bits per parameter. Quantizing them to 8-bit floats in blocks of 256 cuts this to \(8/64 + 32/(64 \cdot 256) \approx 0.127\) bits, saving about 0.37 bits per parameter (about 3 GB for a 65B model).

Gradient checkpointing on mini-batches with long sequences causes memory spikes that can crash training on one GPU. Using NVIDIA unified memory, optimizer states are paged out to CPU RAM when the GPU runs out and paged back when needed for the update.

Yes: with NF4 and double quantization it matched 16-bit full fine-tuning (RoBERTa on GLUE, T5 on Super-NaturalInstructions) and 16-bit LoRA on LLaMA 7B–65B (5-shot MMLU), while QLoRA with plain 4-bit floats lagged by about a point. LoRA adapters on every layer were key.

For a fixed fine-tuning and inference memory budget, it is better to use a larger base model at lower precision than a smaller one at higher precision.

LLaMA fine-tuned with QLoRA on OASST1, a small (about 9K examples) high-quality crowd-sourced dialogue dataset. Guanaco 65B reached 99.3% of ChatGPT’s score on the Vicuna benchmark after 24 hours on one GPU; Guanaco 7B needs only 5 GB of memory.

Data quality and suitability matter more than size: 9K OASST1 examples beat 450K FLAN v2 examples on chatbot benchmarks, while FLAN v2 was best on MMLU. Strong MMLU performance does not imply strong chatbot performance, and vice versa.

With tournaments of pairwise matches judged by GPT-4 or humans, aggregated into Elo ratings, rather than absolute scores with wide confidence intervals. GPT-4 and human rankings agreed moderately at the system level (Kendall τ 0.43) but weakly per example (Fleiss κ 0.25).

They couldn’t afford to show that QLoRA matches full 16-bit fine-tuning at 33B and 65B; they didn’t evaluate other benchmarks (BigBench, HELM), other bit widths or other adapter methods; and benchmark results seem to depend on how similar fine-tuning data is to the benchmark.