learn.ifs.name

Byte Latent Transformer: Patches Scale Better Than Tokens

Pagnoni et al., 2025, Byte Latent Transformer (Proceedings of the 63rd Annual Meeting of ACL (Volume 1: Long Papers))

Subscriber deck — free during early access

1 cards in “4 Experimental Setup” · whole deck

4 Experimental Setup

  1. How did BLT’s comparison with tokenizer models stay fair when patch sizes change context length?
    All comparisons are FLOP-matched, and the number of bytes per batch was held constant on average: the latent Transformer’s sequence length was adjusted so every model sees the same 8k or 16k bytes of context, giving larger patches no advantage from seeing more text.
    • Pagnoni et al., 2025, Byte Latent Transformer (Proceedings of the 63rd Annual Meeting of ACL (Volume 1: Long Papers)), §4, p. 9242

Source

  • Artidoro Pagnoni, Ramakanth Pasunuru, Pedro Rodriguez, John Nguyen, Benjamin Muller, Margaret Li, Chunting Zhou, Lili Yu, Jason E. Weston, Luke Zettlemoyer, Gargi Ghosh, Mike Lewis, Ari Holtzman and Srini Iyer. 2025. Byte Latent Transformer: Patches Scale Better Than Tokens. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 9238–9258. License: CC BY 4.0. Flashcards adapted from this work. doi:10.18653/v1/2025.acl-long.453 https://aclanthology.org/2025.acl-long.453/