learn.ifs.name

LoRA: Low-Rank Adaptation of Large Language Models

Hu et al., 2022, LoRA (The Tenth International Conference on Learning Representations (ICLR 2022))

Subscriber deck — free during early access

1 cards in “2 Problem Statement” · whole deck

2 Problem Statement

  1. How does LoRA reframe adaptation as an optimization problem?
    Full fine-tuning maximizes \(\sum_{(x,y)} \sum_t \log P_\Phi(y_t \mid x, y_{
    • Hu et al., 2022, LoRA (The Tenth International Conference on Learning Representations (ICLR 2022)), §2

Source

  • Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang and Weizhu Chen. 2022. LoRA: Low-Rank Adaptation of Large Language Models. In The Tenth International Conference on Learning Representations (ICLR 2022). https://arxiv.org/abs/2106.09685