Latent Dirichlet Allocation

7.1 Document modeling

1. How did Blei et al. evaluate LDA as a document model, and what did they find?
   Held-out perplexity \(\exp\big(-\sum_d \log p(\mathbf{w}_d) / \sum_d N_d\big)\) on C. elegans abstracts and TREC AP news, against unigram, mixture-of-unigrams and pLSI models. Latent variable models beat the unigram model, and LDA was consistently best across numbers of topics.
   • Blei et al., 2003, Latent Dirichlet Allocation (Journal of Machine Learning Research), §7.1, pp. 1008–1012
2. Why do the mixture of unigrams and pLSI overfit, while LDA doesn’t?
   Mixture of unigrams: peaked posteriors cluster training documents almost deterministically, so a new document has words with tiny probability in its best cluster (AP perplexity reached \(10^{264}\) at 200 topics). pLSI: a new document must reuse topic proportions seen in training. Folding-in refits \(p(z \mid d_{\text{new}})\), an unfair advantage. LDA infers new proportions for any document.
   • Blei et al., 2003, Latent Dirichlet Allocation (Journal of Machine Learning Research), §7.1, pp. 1008–1012