QLoRA: Efficient Finetuning of Quantized LLMs
Dettmers et al., 2023, QLoRA (Advances in Neural Information Processing Systems 36 (NeurIPS 2023))
Subscriber deck — free during early access
1 cards in “7 Limitations and Discussion” · whole deck
7 Limitations and Discussion
- What limitations of the QLoRA study do Dettmers et al. acknowledge?They couldn’t afford to show that QLoRA matches full 16-bit fine-tuning at 33B and 65B; they didn’t evaluate other benchmarks (BigBench, HELM), other bit widths or other adapter methods; and benchmark results seem to depend on how similar fine-tuning data is to the benchmark.
- Dettmers et al., 2023, QLoRA (Advances in Neural Information Processing Systems 36 (NeurIPS 2023)), §7
Source
- Tim Dettmers, Artidoro Pagnoni, Ari Holtzman and Luke Zettlemoyer. 2023. QLoRA: Efficient Finetuning of Quantized LLMs. In Advances in Neural Information Processing Systems 36 (NeurIPS 2023). https://papers.nips.cc/paper_files/paper/2023/hash/1feb87871436031bdc0f2beaa62a049b-Abstract-Conference.html