BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

4.2 SQuAD v1.1

1. How does BERT predict an answer span for SQuAD?
   It learns a start vector \(S\) and an end vector \(E\): \(P_i^{\text{start}} = \frac{e^{S \cdot T_i}}{\sum_j e^{S \cdot T_j}}\), likewise for the end; a span \((i, j)\) with \(j \ge i\) scores \(S \cdot T_i + E \cdot T_j\), and the best is predicted. Training maximizes the log-likelihood of the correct start and end. Test F1 reached 93.2 (ensemble with TriviaQA).
   • Devlin et al., 2019, BERT (Proceedings of the 2019 Conference of the North American Chapter of ACL: Human Language Technologies (NAACL-HLT 2019)), §4.2, pp. 4176–4177