Introduction

1.1 What is a large language model?

1. What is the specific technical meaning of “language model”?
   A computational system that predicts the next word from previous words: given a context (prefix), it assigns a probability distribution over possible next words.
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 1, §1.1, p. 5
2. Given the context “so long and thanks for”, which continuations should a language model rate as likely and which as unlikely?
   Likely: all, everything, the. Unlikely: of, wasn’t. The model expresses this as a probability distribution \(p(w \mid \text{context})\) over the vocabulary.
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 1, §1.1, p. 5
3. How does a model that only predicts next-word probabilities become a model that generates text?
   By sampling: pick the next word according to its probability, append it to the context, compute a new distribution, sample again, and so on. The model conditions both on the prompt and on its own previously generated words.

   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 1, §1.1, p. 6
4. What are causal (autoregressive) language models?
   Models that iteratively predict and generate words left-to-right, each from the earlier words. They are the most common LMs in use today; their left context can be tens or hundreds of thousands of tokens long.
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 1, §1.1, p. 6
5. What kind of language model predicts words using information from both the left and the right context?
   A masked language model such as BERT (a non-autoregressive model, covered in Chapter 9).
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 1, §1.1, p. 6
6. What is the difference between a monolingual and a multilingual language model?
   A monolingual model is trained on a single language (most commonly English); a multilingual model is trained on, and interacts with users in, many languages.
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 1, §1.1, p. 6
7. What is a token, and why do LLMs use tokens rather than words?
   A token is a word or a word-part. The first step of language modeling is converting a word sequence into a token sequence; tokens let a fixed vocabulary cover any text, including rare words and other languages.
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 1, §1.1, p. 6
8. Why are tokens typically closer to whole words in English than in a language like German?
   LLM tokenizers are trained on data dominated by English, so frequent English words get their own tokens. In other languages words are more often split into several pieces (e.g. German Trägheit or Energieinhalt become 3 tokens each).
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 1, §1.1, p. 7
9. What is an n-gram language model, and why does the book introduce it first?
   A very simple LM that works by counting how often words appear in particular orders. It is a convenient setting for introducing core concepts such as sampling and perplexity, where the arithmetic can be done by hand.
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 1, §1.1, p. 7
10. In what two senses is a large language model “large”?
    It is a neural network with many layers and many parameters, and it is trained on very large amounts of text.
    • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 1, §1.1, p. 7
11. What is a parameter of a neural network, and what two kinds are there?
    A single trained real-valued number used in the network’s computation. Parameters are weights and biases. Model size is usually measured in number of parameters (e.g. the 405B in “Llama 3.1 405B”).
    • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 1, §1.1, p. 7
12. What three factors mainly determine the performance of an LLM?
    Model size (number of parameters), training data size (in tokens), and the amount of compute used for training.
    • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 1, §1.1, p. 7
13. What are scaling laws?
    Relationships between model size, data size, training compute and performance. They predict how performance improves as parameters, data or training iterations are increased.
    • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 1, §1.1, p. 7
14. According to Hoffmann et al. (2022), how should parameters \(N\) and training tokens \(D\) grow with a fixed compute budget \(C\)?
    Roughly equally, each as a power law of compute: \(N_{opt} \propto C^{1/2}\) and \(D_{opt} \propto C^{1/2}\).
    • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 1, §1.1, p. 7
15. Why do scaling laws also push developers to train smaller models on more data?
    Very big models are expensive at inference, every time they are prompted. A smaller model trained longer on more data can reach good performance while being cheaper to run.
    • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 1, §1.1, p. 7