Latent Dirichlet Allocation

3 Latent Dirichlet allocation

1. Give LDA’s generative process for a document.
   (1) Choose length \(N \sim \text{Poisson}(\xi)\) (not essential). (2) Choose topic proportions \(\theta \sim \text{Dir}(\alpha)\). (3) For each of the \(N\) words: choose a topic \(z_n \sim \text{Multinomial}(\theta)\), then a word \(w_n\) from \(p(w_n \mid z_n, \beta)\), where \(\beta\) is a \(k \times V\) matrix with \(\beta_{ij} = p(w^j = 1 \mid z^i = 1)\).

   • Blei et al., 2003, Latent Dirichlet Allocation (Journal of Machine Learning Research), §3, pp. 996–997
2. What is the Dirichlet distribution, and why is it convenient for LDA?
   A density on the \((k-1)\)-simplex: \(p(\theta \mid \alpha) = \frac{\Gamma(\sum_i \alpha_i)}{\prod_i \Gamma(\alpha_i)} \theta_1^{\alpha_1 - 1} \cdots \theta_k^{\alpha_k - 1}\), with \(\alpha_i > 0\). It is in the exponential family, has finite-dimensional sufficient statistics, and is conjugate to the multinomial.
   • Blei et al., 2003, Latent Dirichlet Allocation (Journal of Machine Learning Research), §3, pp. 996–997
3. Give the marginal probability of a document under LDA.
   \(p(\mathbf{w} \mid \alpha, \beta) = \int p(\theta \mid \alpha) \Big(\prod_{n=1}^{N} \sum_{z_n} p(z_n \mid \theta)\, p(w_n \mid z_n, \beta)\Big) d\theta\); a corpus’s probability is the product over documents.
   • Blei et al., 2003, Latent Dirichlet Allocation (Journal of Machine Learning Research), §3, pp. 996–997
4. What are LDA’s three levels, and how does LDA differ from Dirichlet-multinomial clustering?
   Corpus level: \(\alpha\), \(\beta\), sampled once. Document level: \(\theta_d\), once per document. Word level: \(z_{dn}\), \(w_{dn}\), once per word. A clustering model picks one cluster per document; in LDA the topic is resampled for every word, so a document can exhibit multiple topics.
   • Blei et al., 2003, Latent Dirichlet Allocation (Journal of Machine Learning Research), §3, pp. 996–997