Byte Latent Transformer: Patches Scale Better Than Tokens

1 Introduction

1. What problems with tokenization motivate the Byte Latent Transformer (BLT)?
   Tokenization is the one step of an LLM that is not learned end to end: a fixed heuristic vocabulary that makes models sensitive to domain and to input noise, hides spelling from them, and treats languages unequally. Training on raw bytes avoids this but was too costly because byte sequences are long.
   • Pagnoni et al., 2025, Byte Latent Transformer (Proceedings of the 63rd Annual Meeting of ACL (Volume 1: Long Papers)), §1, p. 9238
2. In a large byte-level Transformer, where does the compute go, and how does BLT reduce it?
   Mostly into the large feed-forward layers that run once per position, not into attention. BLT groups bytes into patches and runs its big Transformer once per patch, so the average patch size, not the number of bytes, sets the cost.
   • Pagnoni et al., 2025, Byte Latent Transformer (Proceedings of the 63rd Annual Meeting of ACL (Volume 1: Long Papers)), §1, p. 9238