learn.ifs.name

QLoRA: Efficient Finetuning of Quantized LLMs

Dettmers et al., 2023, QLoRA (Advances in Neural Information Processing Systems 36 (NeurIPS 2023))

Subscriber deck — free during early access

2 cards in “4 QLoRA vs. Standard Finetuning” · whole deck

4 QLoRA vs. Standard Finetuning

  1. Did 4-bit QLoRA match 16-bit fine-tuning?
    Yes: with NF4 and double quantization it matched 16-bit full fine-tuning (RoBERTa on GLUE, T5 on Super-NaturalInstructions) and 16-bit LoRA on LLaMA 7B–65B (5-shot MMLU), while QLoRA with plain 4-bit floats lagged by about a point. LoRA adapters on every layer were key.
    • Dettmers et al., 2023, QLoRA (Advances in Neural Information Processing Systems 36 (NeurIPS 2023)), §4
  2. What trade-off between model size and precision does QLoRA suggest?
    For a fixed fine-tuning and inference memory budget, it is better to use a larger base model at lower precision than a smaller one at higher precision.
    • Dettmers et al., 2023, QLoRA (Advances in Neural Information Processing Systems 36 (NeurIPS 2023)), §4

Source