learn.ifs.name

QLoRA: Efficient Finetuning of Quantized LLMs

Dettmers et al., 2023, QLoRA (Advances in Neural Information Processing Systems 36 (NeurIPS 2023))

Subscriber deck — free during early access

5 cards in “3 QLoRA Finetuning” · whole deck

3 QLoRA Finetuning

  1. What are QLoRA’s storage and computation data types?
    Weights are stored in 4-bit NF4; whenever a weight is used it is dequantized to 16-bit BrainFloat for the matrix multiplication. Gradients flow through the dequantized weights, but only the LoRA adapter parameters (in BF16) receive weight updates.
    • Dettmers et al., 2023, QLoRA (Advances in Neural Information Processing Systems 36 (NeurIPS 2023)), §3
  2. What idea underlies the 4-bit NormalFloat data type?
    Quantile quantization: give each quantization bin an equal share of values. Because pretrained weights are roughly zero-centred normal, the bins can be computed once from \(\mathcal{N}(0,1)\) quantiles and normalized to \([-1, 1]\); weight blocks are rescaled to that range, avoiding expensive per-tensor quantile estimation.
    0−+weight value (normalized)
    NF4 idea: bin boundaries at normal quantiles, so each of the 16 bins holds an equal share of the weights
    • Dettmers et al., 2023, QLoRA (Advances in Neural Information Processing Systems 36 (NeurIPS 2023)), §3
  3. How does NF4 get an exact zero, and why does that matter?
    A symmetric quantile scheme has no exact zero. NF4 is built asymmetrically from separate quantiles for the negative and positive halves, sharing one zero, so padding and other zero values are represented exactly while all 16 codes are used.
    • Dettmers et al., 2023, QLoRA (Advances in Neural Information Processing Systems 36 (NeurIPS 2023)), §3
  4. How much memory does Double Quantization save?
    With blocks of 64 weights and 32-bit constants, constants cost 32/64 = 0.5 bits per parameter. Quantizing them to 8-bit floats in blocks of 256 cuts this to \(8/64 + 32/(64 \cdot 256) \approx 0.127\) bits, saving about 0.37 bits per parameter (about 3 GB for a 65B model).
    • Dettmers et al., 2023, QLoRA (Advances in Neural Information Processing Systems 36 (NeurIPS 2023)), §3
  5. What problem do Paged Optimizers solve?
    Gradient checkpointing on mini-batches with long sequences causes memory spikes that can crash training on one GPU. Using NVIDIA unified memory, optimizer states are paged out to CPU RAM when the GPU runs out and paged back when needed for the update.
    • Dettmers et al., 2023, QLoRA (Advances in Neural Information Processing Systems 36 (NeurIPS 2023)), §3

Source