Byte Latent Transformer: Patches Scale Better Than Tokens
Pagnoni et al., 2025, Byte Latent Transformer (Proceedings of the 63rd Annual Meeting of ACL (Volume 1: Long Papers))
Subscriber deck — free during early access
1 cards in “2.3 Entropy Patching” · whole deck
2.3 Entropy Patching
- How does BLT’s entropy patching decide where patches start?A small byte-level LM (100M parameters) computes the entropy of each next byte, \(H(x_i) = -\sum_v p_e(x_i{=}v \mid x_{<i}) \log p_e(x_i{=}v \mid x_{<i})\). A new patch starts where it exceeds a global threshold (or rises sharply over the previous byte’s), so uncertain stretches get more patches, and more compute.
A new patch starts at each byte whose next-byte entropy exceeds the threshold - Pagnoni et al., 2025, Byte Latent Transformer (Proceedings of the 63rd Annual Meeting of ACL (Volume 1: Long Papers)), §2.3, p. 9240
Source
- Artidoro Pagnoni, Ramakanth Pasunuru, Pedro Rodriguez, John Nguyen, Benjamin Muller, Margaret Li, Chunting Zhou, Lili Yu, Jason E. Weston, Luke Zettlemoyer, Gargi Ghosh, Mike Lewis, Ari Holtzman and Srini Iyer. 2025. Byte Latent Transformer: Patches Scale Better Than Tokens. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 9238–9258. License: CC BY 4.0. Flashcards adapted from this work. doi:10.18653/v1/2025.acl-long.453 https://aclanthology.org/2025.acl-long.453/