Latent Dirichlet Allocation

1 Introduction

1. What problem does LDA address?
   Finding short descriptions of the members of a collection (e.g. documents in a corpus) that enable efficient processing while preserving the statistical relationships useful for classification, novelty detection, summarization and similarity judgments.
   • Blei et al., 2003, Latent Dirichlet Allocation (Journal of Machine Learning Research), §1, pp. 993–994
2. What are the shortcomings of tf-idf and LSI as document representations, according to Blei et al.?
   tf-idf reduces documents to fixed-length vectors but compresses little and reveals little inter- or intra-document structure. LSI (an SVD of the term-document matrix) compresses well, but it is not a generative probabilistic model, so it’s unclear why to use it rather than fitting a model directly.
   • Blei et al., 2003, Latent Dirichlet Allocation (Journal of Machine Learning Research), §1, pp. 993–994
3. What are the two main problems with pLSI (the aspect model)?
   It has no generative model at the document level: each training document’s topic mixture is a list of free parameters. So (1) the number of parameters grows linearly with the corpus (\(kV + kM\)), causing overfitting, and (2) there is no natural way to assign probability to a document outside the training set.
   • Blei et al., 2003, Latent Dirichlet Allocation (Journal of Machine Learning Research), §1, pp. 993–994
4. What does exchangeability mean, and how does de Finetti’s theorem motivate LDA?
   Random variables are exchangeable if their joint distribution is invariant to permutation; the bag-of-words assumption is exchangeability of words (and of documents in a corpus). De Finetti: an infinitely exchangeable sequence is distributed as if a random parameter were drawn and the variables were then i.i.d. given it, i.e. a mixture. Exchangeable is “conditionally i.i.d.”, not i.i.d.
   • Blei et al., 2003, Latent Dirichlet Allocation (Journal of Machine Learning Research), §1, pp. 993–994