learn.ifs.name

Byte Latent Transformer: Patches Scale Better Than Tokens

Pagnoni et al., 2025, Byte Latent Transformer (Proceedings of the 63rd Annual Meeting of ACL (Volume 1: Long Papers))

Subscriber deck — free during early access

1 cards in “5.3 Patches Scale Better Than Tokens” · whole deck

5.3 Patches Scale Better Than Tokens

  1. What new scaling axis does BLT open?
    Patch size: larger patches mean fewer latent-Transformer steps, so model size can grow with no change in training or inference FLOPs. At a fixed inference budget, BLT models with patch size 6 or 8 fell behind BPE at small training budgets but overtook it soon after the compute-optimal point.
    • Pagnoni et al., 2025, Byte Latent Transformer (Proceedings of the 63rd Annual Meeting of ACL (Volume 1: Long Papers)), §5.3, p. 9244

Source

  • Artidoro Pagnoni, Ramakanth Pasunuru, Pedro Rodriguez, John Nguyen, Benjamin Muller, Margaret Li, Chunting Zhou, Lili Yu, Jason E. Weston, Luke Zettlemoyer, Gargi Ghosh, Mike Lewis, Ari Holtzman and Srini Iyer. 2025. Byte Latent Transformer: Patches Scale Better Than Tokens. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 9238–9258. License: CC BY 4.0. Flashcards adapted from this work. doi:10.18653/v1/2025.acl-long.453 https://aclanthology.org/2025.acl-long.453/