learn.ifs.name

QLoRA: Efficient Finetuning of Quantized LLMs

Dettmers et al., 2023, QLoRA (Advances in Neural Information Processing Systems 36 (NeurIPS 2023))

Subscriber deck — free during early access

3 cards in “5 Pushing the Chatbot State-of-the-art with QLoRA” · whole deck

5 Pushing the Chatbot State-of-the-art with QLoRA

  1. What is Guanaco, and how good was it?
    LLaMA fine-tuned with QLoRA on OASST1, a small (about 9K examples) high-quality crowd-sourced dialogue dataset. Guanaco 65B reached 99.3% of ChatGPT’s score on the Vicuna benchmark after 24 hours on one GPU; Guanaco 7B needs only 5 GB of memory.
    • Dettmers et al., 2023, QLoRA (Advances in Neural Information Processing Systems 36 (NeurIPS 2023)), §5
  2. What did the QLoRA study reveal about instruction-tuning data?
    Data quality and suitability matter more than size: 9K OASST1 examples beat 450K FLAN v2 examples on chatbot benchmarks, while FLAN v2 was best on MMLU. Strong MMLU performance does not imply strong chatbot performance, and vice versa.
    • Dettmers et al., 2023, QLoRA (Advances in Neural Information Processing Systems 36 (NeurIPS 2023)), §5
  3. How did Dettmers et al. evaluate chatbots, and how reliable was GPT-4 as a judge?
    With tournaments of pairwise matches judged by GPT-4 or humans, aggregated into Elo ratings, rather than absolute scores with wide confidence intervals. GPT-4 and human rankings agreed moderately at the system level (Kendall τ 0.43) but weakly per example (Fleiss κ 0.25).
    • Dettmers et al., 2023, QLoRA (Advances in Neural Information Processing Systems 36 (NeurIPS 2023)), §5

Source