Byte Latent Transformer: Patches Scale Better Than Tokens
Pagnoni et al., 2025, Byte Latent Transformer (Proceedings of the 63rd Annual Meeting of ACL (Volume 1: Long Papers))
Subscriber deck — free during early access
1 cards in “Limitations” · whole deck
Limitations
- What limitations of BLT do its authors list?The scaling recipes used were derived for BPE Transformers and may be suboptimal for BLT; current libraries are tuned for token-based models, so wall-clock speed may not yet match the FLOP savings; and the patcher is trained separately rather than end to end.
- Pagnoni et al., 2025, Byte Latent Transformer (Proceedings of the 63rd Annual Meeting of ACL (Volume 1: Long Papers)), Limitations, p. 9246
Source
- Artidoro Pagnoni, Ramakanth Pasunuru, Pedro Rodriguez, John Nguyen, Benjamin Muller, Margaret Li, Chunting Zhou, Lili Yu, Jason E. Weston, Luke Zettlemoyer, Gargi Ghosh, Mike Lewis, Ari Holtzman and Srini Iyer. 2025. Byte Latent Transformer: Patches Scale Better Than Tokens. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 9238–9258. License: CC BY 4.0. Flashcards adapted from this work. doi:10.18653/v1/2025.acl-long.453 https://aclanthology.org/2025.acl-long.453/