learn.ifs.name

LoRA: Low-Rank Adaptation of Large Language Models

Hu et al., 2022, LoRA (The Tenth International Conference on Learning Representations (ICLR 2022))

Subscriber deck — free during early access

1 cards in “5 Empirical Experiments” · whole deck

5 Empirical Experiments

  1. On which models and tasks was LoRA evaluated?
    RoBERTa-base/large and DeBERTa-XXL on GLUE, GPT-2 medium/large on E2E NLG (plus WebNLG and DART), and GPT-3 175B on WikiSQL, MNLI-matched and SAMSum summarization, against full fine-tuning, BitFit (biases only), prefix-embedding and prefix-layer tuning, and several adapter variants.
    • Hu et al., 2022, LoRA (The Tenth International Conference on Learning Representations (ICLR 2022)), §5

Source

  • Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang and Weizhu Chen. 2022. LoRA: Low-Rank Adaptation of Large Language Models. In The Tenth International Conference on Learning Representations (ICLR 2022). https://arxiv.org/abs/2106.09685