learn.ifs.name

Byte Latent Transformer: Patches Scale Better Than Tokens

Pagnoni et al., 2025, Byte Latent Transformer (Proceedings of the 63rd Annual Meeting of ACL (Volume 1: Long Papers))

Subscriber deck — free during early access

1 cards in “6 Byte Modeling Improves Robustness” · whole deck

6 Byte Modeling Improves Robustness

  1. Where did byte modeling give BLT a clear advantage over a BPE Llama 3 model trained on the same 4T bytes?
    On noised inputs (e.g. corrupted HellaSwag), character-level tasks (CUTE, with 99.9% on spelling) and grapheme-to-phoneme conversion, and in low-resource FLORES-101 translation. Llama 3.1, trained on far more data, often still trailed BLT, so this awareness does not just come from more data.
    • Pagnoni et al., 2025, Byte Latent Transformer (Proceedings of the 63rd Annual Meeting of ACL (Volume 1: Long Papers)), §6, p. 9244

Source

  • Artidoro Pagnoni, Ramakanth Pasunuru, Pedro Rodriguez, John Nguyen, Benjamin Muller, Margaret Li, Chunting Zhou, Lili Yu, Jason E. Weston, Luke Zettlemoyer, Gargi Ghosh, Mike Lewis, Ari Holtzman and Srini Iyer. 2025. Byte Latent Transformer: Patches Scale Better Than Tokens. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 9238–9258. License: CC BY 4.0. Flashcards adapted from this work. doi:10.18653/v1/2025.acl-long.453 https://aclanthology.org/2025.acl-long.453/