learn.ifs.name

Byte Latent Transformer: Patches Scale Better Than Tokens

Pagnoni et al., 2025, Byte Latent Transformer (Proceedings of the 63rd Annual Meeting of ACL (Volume 1: Long Papers))

Subscriber deck — free during early access

13 cards

1 Introduction

  1. What problems with tokenization motivate the Byte Latent Transformer (BLT)?
    Tokenization is the one step of an LLM that is not learned end to end: a fixed heuristic vocabulary that makes models sensitive to domain and to input noise, hides spelling from them, and treats languages unequally. Training on raw bytes avoids this but was too costly because byte sequences are long.
    • Pagnoni et al., 2025, Byte Latent Transformer (Proceedings of the 63rd Annual Meeting of ACL (Volume 1: Long Papers)), §1, p. 9238
  2. In a large byte-level Transformer, where does the compute go, and how does BLT reduce it?
    Mostly into the large feed-forward layers that run once per position, not into attention. BLT groups bytes into patches and runs its big Transformer once per patch, so the average patch size, not the number of bytes, sets the cost.
    • Pagnoni et al., 2025, Byte Latent Transformer (Proceedings of the 63rd Annual Meeting of ACL (Volume 1: Long Papers)), §1, p. 9238

2.1 Tokenization vs. Patching

  1. How do “patches” in BLT differ from “tokens”?
    Tokens come from a finite vocabulary fixed before training; patches are byte groups formed dynamically with no vocabulary, so any byte sequence can be a patch and the model still sees the bytes inside it, e.g. spelling.
    • Pagnoni et al., 2025, Byte Latent Transformer (Proceedings of the 63rd Annual Meeting of ACL (Volume 1: Long Papers)), §2.1, p. 9239

2.2 Strided and Space Patching

  1. What are strided and space patching, and what is wrong with each?
    Strided patching (MegaByte) cuts fixed groups of \(k\) bytes, ignoring how hard the text is and splitting the same word differently in different places. Space patching starts a patch after each space-like byte, which is consistent and puts compute after spaces where predictions are hard, but fails for languages and domains without spaces.
    • Pagnoni et al., 2025, Byte Latent Transformer (Proceedings of the 63rd Annual Meeting of ACL (Volume 1: Long Papers)), §2.2, p. 9239

2.3 Entropy Patching

  1. How does BLT’s entropy patching decide where patches start?
    A small byte-level LM (100M parameters) computes the entropy of each next byte, \(H(x_i) = -\sum_v p_e(x_i{=}v \mid x_{<i}) \log p_e(x_i{=}v \mid x_{<i})\). A new patch starts where it exceeds a global threshold (or rises sharply over the previous byte’s), so uncertain stretches get more patches, and more compute.
    the␣cat␣sat␣on␣a␣matθHpatch5 patches for 20 bytes (invented entropies)
    A new patch starts at each byte whose next-byte entropy exceeds the threshold
    • Pagnoni et al., 2025, Byte Latent Transformer (Proceedings of the 63rd Annual Meeting of ACL (Volume 1: Long Papers)), §2.3, p. 9240

3 BLT Architecture

  1. What are the three modules of BLT?
    A lightweight local encoder that turns bytes into patch representations; a large latent global Transformer over patches (block-causal, most of the FLOPs); and a lightweight local decoder that turns each output patch representation back into the next patch’s bytes.
    • Pagnoni et al., 2025, Byte Latent Transformer (Proceedings of the 63rd Annual Meeting of ACL (Volume 1: Long Papers)), §3, p. 9240

3.2.1 Encoder Hash n-gram Embeddings

  1. What are hash n-gram embeddings in BLT, and why use them?
    Each byte’s embedding is enriched with embeddings of the byte n-grams ending at it (\(n = 3\) to \(8\)), each n-gram hashed into a fixed-size table per \(n\). This gives the small encoder information about preceding bytes cheaply; the ablations found them vital for matching tokenizer-based models.
    • Pagnoni et al., 2025, Byte Latent Transformer (Proceedings of the 63rd Annual Meeting of ACL (Volume 1: Long Papers)), §3.2.1, p. 9240

3.2.2 Encoder Multi-Headed Cross-Attention

  1. How do BLT’s cross-attention layers connect bytes and patches?
    In the encoder, each patch is a query (initialized by pooling its bytes) that attends only to the bytes in that patch, like Perceiver with a variable number of latents. In the decoder the roles swap: bytes are the queries and patch representations the keys and values.
    • Pagnoni et al., 2025, Byte Latent Transformer (Proceedings of the 63rd Annual Meeting of ACL (Volume 1: Long Papers)), §3.2.2, p. 9241

4 Experimental Setup

  1. How did BLT’s comparison with tokenizer models stay fair when patch sizes change context length?
    All comparisons are FLOP-matched, and the number of bytes per batch was held constant on average: the latent Transformer’s sequence length was adjusted so every model sees the same 8k or 16k bytes of context, giving larger patches no advantage from seeing more text.
    • Pagnoni et al., 2025, Byte Latent Transformer (Proceedings of the 63rd Annual Meeting of ACL (Volume 1: Long Papers)), §4, p. 9242

5.1 Compute Optimal Scaling Trends

  1. How did BLT compare with BPE models in compute-optimal scaling up to 8B?
    It matched or beat them in bits per byte: the first byte-level architecture to do so. With an average patch size of 8 bytes (versus 3.7 and 4.4 for Llama 2 and 3 tokens), it scaled between Llama 2 and 3 while needing nearly 50% fewer inference FLOPs.
    • Pagnoni et al., 2025, Byte Latent Transformer (Proceedings of the 63rd Annual Meeting of ACL (Volume 1: Long Papers)), §5.1, p. 9242

5.3 Patches Scale Better Than Tokens

  1. What new scaling axis does BLT open?
    Patch size: larger patches mean fewer latent-Transformer steps, so model size can grow with no change in training or inference FLOPs. At a fixed inference budget, BLT models with patch size 6 or 8 fell behind BPE at small training budgets but overtook it soon after the compute-optimal point.
    • Pagnoni et al., 2025, Byte Latent Transformer (Proceedings of the 63rd Annual Meeting of ACL (Volume 1: Long Papers)), §5.3, p. 9244

6 Byte Modeling Improves Robustness

  1. Where did byte modeling give BLT a clear advantage over a BPE Llama 3 model trained on the same 4T bytes?
    On noised inputs (e.g. corrupted HellaSwag), character-level tasks (CUTE, with 99.9% on spelling) and grapheme-to-phoneme conversion, and in low-resource FLORES-101 translation. Llama 3.1, trained on far more data, often still trailed BLT, so this awareness does not just come from more data.
    • Pagnoni et al., 2025, Byte Latent Transformer (Proceedings of the 63rd Annual Meeting of ACL (Volume 1: Long Papers)), §6, p. 9244

Limitations

  1. What limitations of BLT do its authors list?
    The scaling recipes used were derived for BPE Transformers and may be suboptimal for BLT; current libraries are tuned for token-based models, so wall-clock speed may not yet match the FLOP savings; and the patcher is trained separately rather than end to end.
    • Pagnoni et al., 2025, Byte Latent Transformer (Proceedings of the 63rd Annual Meeting of ACL (Volume 1: Long Papers)), Limitations, p. 9246

Source

  • Artidoro Pagnoni, Ramakanth Pasunuru, Pedro Rodriguez, John Nguyen, Benjamin Muller, Margaret Li, Chunting Zhou, Lili Yu, Jason E. Weston, Luke Zettlemoyer, Gargi Ghosh, Mike Lewis, Ari Holtzman and Srini Iyer. 2025. Byte Latent Transformer: Patches Scale Better Than Tokens. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 9238–9258. License: CC BY 4.0. Flashcards adapted from this work. doi:10.18653/v1/2025.acl-long.453 https://aclanthology.org/2025.acl-long.453/