QLoRA: Efficient Finetuning of Quantized LLMs

1 Introduction

1. What does QLoRA make possible?
   Fine-tuning a 65B-parameter model on a single 48 GB GPU (instead of more than 780 GB for 16-bit fine-tuning) without losing 16-bit fine-tuning performance, by backpropagating through a frozen 4-bit quantized model into LoRA adapters.
   • Dettmers et al., 2023, QLoRA (Advances in Neural Information Processing Systems 36 (NeurIPS 2023)), §1
2. Why was fine-tuning quantized models an open problem before QLoRA?
   Quantization methods had reduced memory only for inference; they broke down during training. QLoRA was the first to fine-tune a 4-bit model without performance degradation.
   • Dettmers et al., 2023, QLoRA (Advances in Neural Information Processing Systems 36 (NeurIPS 2023)), §1
3. What three innovations does QLoRA introduce to save memory?
   4-bit NormalFloat (NF4), a data type suited to normally distributed weights; Double Quantization, which quantizes the quantization constants; and Paged Optimizers, which move optimizer states between GPU and CPU memory to survive memory spikes.
   • Dettmers et al., 2023, QLoRA (Advances in Neural Information Processing Systems 36 (NeurIPS 2023)), §1