learn.ifs.name

Byte Latent Transformer: Patches Scale Better Than Tokens

Pagnoni et al., 2025, Byte Latent Transformer (Proceedings of the 63rd Annual Meeting of ACL (Volume 1: Long Papers))

Subscriber deck — free during early access

1 cards in “3.2.2 Encoder Multi-Headed Cross-Attention” · whole deck

3.2.2 Encoder Multi-Headed Cross-Attention

  1. How do BLT’s cross-attention layers connect bytes and patches?
    In the encoder, each patch is a query (initialized by pooling its bytes) that attends only to the bytes in that patch, like Perceiver with a variable number of latents. In the decoder the roles swap: bytes are the queries and patch representations the keys and values.
    • Pagnoni et al., 2025, Byte Latent Transformer (Proceedings of the 63rd Annual Meeting of ACL (Volume 1: Long Papers)), §3.2.2, p. 9241

Source

  • Artidoro Pagnoni, Ramakanth Pasunuru, Pedro Rodriguez, John Nguyen, Benjamin Muller, Margaret Li, Chunting Zhou, Lili Yu, Jason E. Weston, Luke Zettlemoyer, Gargi Ghosh, Mike Lewis, Ari Holtzman and Srini Iyer. 2025. Byte Latent Transformer: Patches Scale Better Than Tokens. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 9238–9258. License: CC BY 4.0. Flashcards adapted from this work. doi:10.18653/v1/2025.acl-long.453 https://aclanthology.org/2025.acl-long.453/