Byte Latent Transformer: Patches Scale Better Than Tokens
Pagnoni et al., 2025, Byte Latent Transformer (Proceedings of the 63rd Annual Meeting of ACL (Volume 1: Long Papers))
Subscriber deck — free during early access
1 cards in “5.1 Compute Optimal Scaling Trends” · whole deck
5.1 Compute Optimal Scaling Trends
- How did BLT compare with BPE models in compute-optimal scaling up to 8B?It matched or beat them in bits per byte: the first byte-level architecture to do so. With an average patch size of 8 bytes (versus 3.7 and 4.4 for Llama 2 and 3 tokens), it scaled between Llama 2 and 3 while needing nearly 50% fewer inference FLOPs.
- Pagnoni et al., 2025, Byte Latent Transformer (Proceedings of the 63rd Annual Meeting of ACL (Volume 1: Long Papers)), §5.1, p. 9242
Source
- Artidoro Pagnoni, Ramakanth Pasunuru, Pedro Rodriguez, John Nguyen, Benjamin Muller, Margaret Li, Chunting Zhou, Lili Yu, Jason E. Weston, Luke Zettlemoyer, Gargi Ghosh, Mike Lewis, Ari Holtzman and Srini Iyer. 2025. Byte Latent Transformer: Patches Scale Better Than Tokens. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 9238–9258. License: CC BY 4.0. Flashcards adapted from this work. doi:10.18653/v1/2025.acl-long.453 https://aclanthology.org/2025.acl-long.453/