learn.ifs.name

Machine Translation

Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 13

12 cards in “13.2 MT using encoder-decoder” · whole deck

13.2 MT using encoder-decoder

  1. What is the basic MT task and its usual simplification?
    Generate a target-language sentence corresponding to a source sentence, typically translating each sentence independently.
  2. What is MT trained on, and what probability does it maximize?
    A large set of parallel sentences; it is trained to maximize \(P(y_1, \ldots, y_m \mid x_1, \ldots, x_n)\) of the target subword tokens given the source tokens.
  3. How does an encoder-decoder translate?
    The encoder maps the source tokens to a context \(h = \text{encoder}(x)\); the decoder generates the output token by token, \(y_{t+1} = \text{decoder}(h, y_1, \ldots, y_t)\).
  4. Why do MT systems use a shared subword vocabulary for source and target?
    It makes it easy to copy tokens like names from source to target, and handles languages with and without spaces (English vs Chinese, Thai) uniformly. It is learned on combined source and target data.
  5. How does the wordpiece algorithm differ from BPE?
    Instead of merging the most frequent pair, it adds the merge of two existing wordpieces that most increases the language-model probability of the training corpus, until a desired vocabulary size is reached (BPE instead fixes the number of merges).
  6. How does unigram (SentencePiece) tokenization build its vocabulary?
    Top-down: start from a huge vocabulary (all characters plus frequent character sequences), estimate token probabilities (with EM), and iteratively remove tokens that don’t appear in high-probability tokenizations until the desired size is reached.
  7. Why does unigram tokenization often produce better tokens than BPE?
    BPE creates many small meaningless pieces and merges very common tokens like ed onto neighbors (cor rupted), while unigram tends to find meaningful units (corrupt ed).
  8. What is a parallel corpus (bitext)? Give examples.
    Text appearing in two or more languages: Europarl (European Parliament proceedings), the UN Parallel Corpus, OpenSubtitles (movie and TV subtitles), ParaCrawl (web text).
  9. What two components are needed to create sentence alignments from two translated documents?
    A cost function scoring how likely a span of source sentences and a span of target sentences are translations of each other, and an alignment algorithm (usually dynamic programming, like minimum edit distance) that finds a good overall alignment.
  10. How are sentences scored for alignment across languages?
    With a multilingual embedding space where sentences from different languages share one space: cosine similarity, normalized against random sentences and biased toward aligning single sentences (Thompson & Koehn).
  11. What kinds of alignments can sentence alignment produce?
    1–1 mappings, 2–1 (or 1–2) alignments where one sentence is split or merged, and null alignments where a sentence has no counterpart.
  12. How are parallel corpora cleaned?
    Remove noisy pairs with rules (too long, too short, different URLs, suspiciously identical) or by discarding pairs with low multilingual-embedding cosine.

Source