Introduction

1.3 Neural networks and embeddings

1. What two big ideas underlie modern language models?
   Neural networks (trainable machine-learning systems that do the computation) and embeddings (vectors representing the meanings of tokens and concepts inside the network).
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 1, §1.3, p. 11
2. Why are neural networks called “neural”?
   Only for historical reasons: they originate in the McCulloch–Pitts neuron (1943), a simplified logical model of a human neuron. Modern networks no longer draw on these biological inspirations.
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 1, §1.3, p. 11
3. What is a modern neural network, in computational terms?
   A network of small computing units, each of which takes a vector of inputs, multiplies them by learned weights, and outputs a single value. Its strength is an efficient learning algorithm based on gradient descent.
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 1, §1.3, p. 11
4. What is the difference between open-weight and proprietary (closed-weight) models?
   Open-weight models’ creators release all the weights, so anyone can run them locally. Proprietary models’ weights are not released; people can only use them via a web app or an API.
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 1, §1.3, p. 11
5. What is an embedding?
   A vector (list of real numbers) used inside the network to represent the meaning of a token, phrase or sentence in context: a point in a high-dimensional space.
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 1, §1.3, p. 11
6. What does the geometric view of embeddings make possible?
   Representing symbols as points in a continuous space, so that similarity of meaning can be modeled as distance between points (e.g. sweet lies near honey and candy, far from soup).

   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 1, §1.3, p. 11
7. Who first proposed modeling word meaning as a point in space, with similarity as distance?
   The psychologist Charles Osgood (Osgood et al., 1957).
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 1, §1.3, p. 11
8. At a high level, how does an LLM turn an input string into a prediction?
   It converts the tokens into a sequence of embeddings, passes them through a structured neural network (e.g. a transformer), and outputs a probability distribution over the next token.
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 1, §1.3, p. 12