learn.ifs.name

Byte Latent Transformer: Patches Scale Better Than Tokens

Pagnoni et al., 2025, Byte Latent Transformer (Proceedings of the 63rd Annual Meeting of ACL (Volume 1: Long Papers))

Subscriber deck — free during early access

1 cards in “2.2 Strided and Space Patching” · whole deck

2.2 Strided and Space Patching

  1. What are strided and space patching, and what is wrong with each?
    Strided patching (MegaByte) cuts fixed groups of \(k\) bytes, ignoring how hard the text is and splitting the same word differently in different places. Space patching starts a patch after each space-like byte, which is consistent and puts compute after spaces where predictions are hard, but fails for languages and domains without spaces.
    • Pagnoni et al., 2025, Byte Latent Transformer (Proceedings of the 63rd Annual Meeting of ACL (Volume 1: Long Papers)), §2.2, p. 9239

Source

  • Artidoro Pagnoni, Ramakanth Pasunuru, Pedro Rodriguez, John Nguyen, Benjamin Muller, Margaret Li, Chunting Zhou, Lili Yu, Jason E. Weston, Luke Zettlemoyer, Gargi Ghosh, Mike Lewis, Ari Holtzman and Srini Iyer. 2025. Byte Latent Transformer: Patches Scale Better Than Tokens. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 9238–9258. License: CC BY 4.0. Flashcards adapted from this work. doi:10.18653/v1/2025.acl-long.453 https://aclanthology.org/2025.acl-long.453/