Language Models are Unsupervised Multitask Learners

3.8. Question Answering

1. How well did GPT-2 answer Natural Questions without retrieval, and how well calibrated was it?
   It answered 4.1% exactly (5.3 times the smallest model, which didn’t beat a 1% most-common-answer baseline), suggesting capacity matters for stored knowledge. Its probabilities were well calibrated: 63.1% accuracy on the 1% of questions it was most confident about. Retrieval-based systems reached 30–50%.
   • Radford et al., 2019, Language Models are Unsupervised Multitask Learners (OpenAI technical report), 3.8. Question Answering
2. How did the GPT-2 paper turn a language model into a task solver for each task?
   By writing the task into the context and letting the model continue:

   task	how the task is invoked
   summarization	article … TL;DR:
   translation	example pairs english = french, then english =
   reading comprehension	document, dialogue history, A:
   question answering	example question–answer pairs, then the question

   • Radford et al., 2019, Language Models are Unsupervised Multitask Learners (OpenAI technical report), 3.8. Question Answering