Language Models are Unsupervised Multitask Learners

2. Approach

1. How can a single language model perform many tasks, according to GPT-2?
   A general system should model \(p(\text{output} \mid \text{input}, \text{task})\), and language can specify the task, input and output all as a sequence of symbols, e.g. (translate to french, english text, french text). A language model may learn to infer and perform tasks demonstrated in natural text in order to predict it better.
   • Radford et al., 2019, Language Models are Unsupervised Multitask Learners (OpenAI technical report), 2. Approach
2. Why does the GPT-2 paper argue the unsupervised LM objective contains supervised objectives?
   A supervised objective is the same as the language-modelling objective evaluated only on a subset of the sequence (the output), so the global minimum of the unsupervised objective is also a minimum of the supervised one; the question is whether it can be optimized well enough in practice.
   • Radford et al., 2019, Language Models are Unsupervised Multitask Learners (OpenAI technical report), 2. Approach