Byte Latent Transformer: Patches Scale Better Than Tokens
Pagnoni et al., 2025, Byte Latent Transformer (Proceedings of the 63rd Annual Meeting of ACL (Volume 1: Long Papers))
Subscriber deck — free during early access
1 cards in “3.2.1 Encoder Hash n-gram Embeddings” · whole deck
3.2.1 Encoder Hash n-gram Embeddings
- What are hash n-gram embeddings in BLT, and why use them?Each byte’s embedding is enriched with embeddings of the byte n-grams ending at it (\(n = 3\) to \(8\)), each n-gram hashed into a fixed-size table per \(n\). This gives the small encoder information about preceding bytes cheaply; the ablations found them vital for matching tokenizer-based models.
- Pagnoni et al., 2025, Byte Latent Transformer (Proceedings of the 63rd Annual Meeting of ACL (Volume 1: Long Papers)), §3.2.1, p. 9240
Source
- Artidoro Pagnoni, Ramakanth Pasunuru, Pedro Rodriguez, John Nguyen, Benjamin Muller, Margaret Li, Chunting Zhou, Lili Yu, Jason E. Weston, Luke Zettlemoyer, Gargi Ghosh, Mike Lewis, Ari Holtzman and Srini Iyer. 2025. Byte Latent Transformer: Patches Scale Better Than Tokens. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 9238–9258. License: CC BY 4.0. Flashcards adapted from this work. doi:10.18653/v1/2025.acl-long.453 https://aclanthology.org/2025.acl-long.453/