learn.ifs.name

LoRA: Low-Rank Adaptation of Large Language Models

Hu et al., 2022, LoRA (The Tenth International Conference on Learning Representations (ICLR 2022))

Subscriber deck — free during early access

2 cards in “3 Aren’t Existing Solutions Good Enough?” · whole deck

3 Aren’t Existing Solutions Good Enough?

  1. Why do adapter layers add inference latency, according to Hu et al.?
    They add depth that must be processed sequentially; even with few parameters and FLOPs, large models rely on parallelism, so with small online batches the extra layers noticeably slow inference (e.g. up to 20–30% for GPT-2 medium at batch size 1), and model sharding makes it worse.
    • Hu et al., 2022, LoRA (The Tenth International Conference on Learning Representations (ICLR 2022)), §3
  2. What is wrong with prefix tuning, according to Hu et al.?
    It is hard to optimize, its performance changes non-monotonically with the number of trainable parameters, and reserving sequence positions for the prefix reduces the length available for the task input.
    • Hu et al., 2022, LoRA (The Tenth International Conference on Learning Representations (ICLR 2022)), §3

Source

  • Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang and Weizhu Chen. 2022. LoRA: Low-Rank Adaptation of Large Language Models. In The Tenth International Conference on Learning Representations (ICLR 2022). https://arxiv.org/abs/2106.09685