BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

3 BERT

1. What is BERT’s architecture, and what are the two standard sizes?
   A multi-layer bidirectional Transformer encoder. BERT-base: \(L = 12\) layers, hidden size \(H = 768\), \(A = 12\) heads, 110M parameters (same size as GPT). BERT-large: \(L = 24\), \(H = 1024\), \(A = 16\), 340M parameters. The feed-forward size is \(4H\).
   • Devlin et al., 2019, BERT (Proceedings of the 2019 Conference of the North American Chapter of ACL: Human Language Technologies (NAACL-HLT 2019)), §3, pp. 4173–4174
2. How does BERT represent its input?
   WordPiece tokens (30,000 vocabulary), always starting with [CLS], whose final hidden state \(C\) serves as the sequence representation for classification. Sentence pairs are packed into one sequence, separated by [SEP], and each token’s input is the sum of its token, segment (A or B) and position embeddings.

   input	token	segment	position
   [CLS]	E[CLS]	EA	E0
   the	Ethe	EA	E1
   cat	Ecat	EA	E2
   sat	Esat	EA	E3
   [SEP]	E[SEP]	EA	E4
   it	Eit	EB	E5
   purr	Epurr	EB	E6
   ##ed	E##ed	EB	E7
   [SEP]	E[SEP]	EB	E8

   Each row’s three embeddings are summed; purred is split into the WordPieces purr ##ed.
   • Devlin et al., 2019, BERT (Proceedings of the 2019 Conference of the North American Chapter of ACL: Human Language Technologies (NAACL-HLT 2019)), §3, pp. 4173–4174