Latent Dirichlet Allocation

5.2 Variational inference

1. What variational distribution approximates the LDA posterior?
   Drop the edges between \(\theta\), \(z\) and \(w\) and the \(w\) nodes: \(q(\theta, \mathbf{z} \mid \gamma, \phi) = q(\theta \mid \gamma) \prod_{n=1}^{N} q(z_n \mid \phi_n)\), with a free Dirichlet parameter \(\gamma\) and free multinomials \(\phi_n\), chosen to minimize \(\text{KL}(q \,\|\, p(\theta, \mathbf{z} \mid \mathbf{w}, \alpha, \beta))\), which tightens a Jensen lower bound on the log likelihood.
   • Blei et al., 2003, Latent Dirichlet Allocation (Journal of Machine Learning Research), §5.2, pp. 1003–1005
2. Give LDA’s variational update equations and their interpretation.
   \(\phi_{ni} \propto \beta_{i w_n} \exp\{\Psi(\gamma_i) - \Psi(\sum_j \gamma_j)\}\) and \(\gamma_i = \alpha_i + \sum_{n=1}^{N} \phi_{ni}\), where \(\Psi\) is the digamma function. The \(\gamma\) update is a posterior Dirichlet given expected topic counts; the \(\phi\) update is Bayes’ rule with \(p(z_n)\) approximated by the exponentiated expected log of \(\theta\). Iterate to convergence.
   • Blei et al., 2003, Latent Dirichlet Allocation (Journal of Machine Learning Research), §5.2, pp. 1003–1005
3. What does a document’s variational parameter \(\gamma^*(\mathbf{w})\) represent, and what does inference cost?
   It is document-specific: a representation of the document in the topic simplex, and \(\gamma_i - \alpha_i\) is the expected number of words generated by topic \(i\). Each iteration costs \(O((N+1)k)\), and about \(N\) iterations are needed, so roughly \(N^2 k\) operations per document.
   • Blei et al., 2003, Latent Dirichlet Allocation (Journal of Machine Learning Research), §5.2, pp. 1003–1005