QLoRA: Efficient Finetuning of Quantized LLMs
Dettmers et al., 2023, QLoRA (Advances in Neural Information Processing Systems 36 (NeurIPS 2023))
Subscriber deck — free during early access
1 cards in “2 Background” · whole deck
2 Background
- What is block-wise quantization, and why is it used?Scaling a tensor by its absolute maximum to fit the low-bit range wastes bins when there are outliers. Splitting the tensor into small blocks, each quantized with its own constant, confines an outlier’s effect to its block.
- Dettmers et al., 2023, QLoRA (Advances in Neural Information Processing Systems 36 (NeurIPS 2023)), §2
Source
- Tim Dettmers, Artidoro Pagnoni, Ari Holtzman and Luke Zettlemoyer. 2023. QLoRA: Efficient Finetuning of Quantized LLMs. In Advances in Neural Information Processing Systems 36 (NeurIPS 2023). https://papers.nips.cc/paper_files/paper/2023/hash/1feb87871436031bdc0f2beaa62a049b-Abstract-Conference.html