Thumbs up? Sentiment Classification using Machine Learning Techniques

Classifying whole documents by overall sentiment (positive vs negative) rather than by topic. Topics are often identifiable by keywords alone, but sentiment can be expressed subtly: “How could anyone sit through this movie?” contains no single obviously negative word.

Summarizing reviews (labelling reviews without explicit ratings and normalizing rating schemes), business intelligence, recommender systems, processing free-form survey responses, and message filtering, e.g. recognizing “flames”.

Genre and subjectivity techniques can recognize that a document expresses an opinion (e.g. an editorial), but not what the opinion is: positive or negative.

It was largely knowledge-based: semantic orientation of words from linguistic heuristics or seed words, models from cognitive linguistics, or hand-built discriminant-word lexicons. Pang et al. used prior-knowledge-free supervised machine learning to understand the task’s inherent difficulty.

Unsupervised: the semantic orientation of phrases was estimated from their mutual information with “excellent” vs “poor”, using search-engine hit counts. Movie reviews were the hardest of his domains (65.83% on 120 documents).

Large online collections exist, and reviewers often summarize their sentiment with a machine-extractable rating (stars or a number), so no hand labelling is needed for training or evaluation.

From the IMDb archive of the rec.arts.movies.reviews newsgroup: reviews with star or numeric ratings, automatically mapped to positive, negative or neutral (only positive vs negative used). Fewer than 20 reviews per author per category were kept so prolific reviewers wouldn’t dominate: 752 negative and 1301 positive reviews from 144 reviewers, released publicly.

Two computer science graduate students independently chose positive and negative indicator words; a document was classified by counting proposed positive vs negative words, on balanced data (chance = 50%). The lists reached 58% and 64% accuracy, with high tie rates (75% and 39%).

Seven positive and seven negative items picked by glancing at corpus counts (including test data) reached 69% with a 16% tie rate. It included items people wouldn’t propose by introspection, like “?” (“What was the director thinking?”) and “still” (“Still, though, it was worth seeing”). So corpus-based feature selection beats intuition.

Fix \(m\) features \(f_1, \ldots, f_m\) (e.g. the word “still” or the bigram “really stinks”); with \(n_i(d)\) the count of \(f_i\) in \(d\), the document vector is \(\vec{d} = (n_1(d), \ldots, n_m(d))\).

\(P_{\text{NB}}(c \mid d) = \frac{P(c) \prod_{i=1}^{m} P(f_i \mid c)^{n_i(d)}}{P(d)}\), assuming features are conditionally independent given the class; \(P(c)\) and \(P(f_i \mid c)\) are relative frequencies with add-one smoothing, and \(P(d)\) is ignored when choosing \(\arg\max_c\).

Empirically it does well in text categorization, and Domingos and Pazzani (1997) showed it is optimal for some problem classes with highly dependent features; the classification can be right even when the probabilities are wrong.

\(P_{\text{ME}}(c \mid d) = \frac{1}{Z(d)} \exp\big(\sum_i \lambda_{i,c} F_{i,c}(d, c)\big)\), where \(F_{i,c}(d, c') = 1\) iff \(n_i(d) > 0\) and \(c' = c\). A large \(\lambda_{i,c}\) makes \(f_i\) a strong indicator of class \(c\). Weights were trained with 10 iterations of improved iterative scaling and a Gaussian prior.

It makes no assumptions about relationships between features, so it may do better when conditional independence fails, e.g. with overlapping unigram and bigram features.

Training finds the separating hyperplane \(\vec{w}\) with the largest margin; the solution is \(\vec{w} = \sum_j \alpha_j c_j \vec{d}_j\) with \(c_j \in \{1, -1\}\) and \(\alpha_j \ge 0\) from the dual problem. Documents with \(\alpha_j > 0\) are support vectors; a test document is classified by which side of the hyperplane it falls on.

700 positive and 700 negative reviews, three-fold cross-validation with balanced folds (few folds because MaxEnt training was slow). Rating indicators were removed and punctuation kept as tokens; no stemming or stop lists.

Adapting Das and Chen (2001), they prefixed NOT_ to every word between a negation word (“not”, “isn’t”, “didn’t”…) and the next punctuation mark, so “good” and “not very good” give different features: not very good . → not NOT_very NOT_good .

Unigrams (with negation tags) occurring at least 4 times (16165) and the 16165 most frequent bigrams (each occurring at least 7 times), because MaxEnt training is expensive in the number of features.

All beat the human-word-list baselines (58%, 64%) and, by smaller margins, the 69% baseline. Unigram frequencies gave NB 78.7% and SVM 72.8%; unigram presence gave NB 81.0, MaxEnt 80.4 and SVM 82.9%. Topic classification typically reaches 90%+, even with more classes, suggesting sentiment is harder.

features	NB	ME	SVM
unigrams, frequency	78.7	–	72.8
unigrams, presence	81.0	80.4	82.9
unigrams + bigrams	80.6	80.8	82.7
bigrams only	77.3	77.4	77.1
adjectives only	77.0	77.7	75.1

Binary presence features beat counts, much more so for SVMs (82.9% vs 72.8%). This contradicts McCallum and Nigam’s (1998) finding for Naive Bayes topic classification; perhaps topics are signalled by repeated content words, but sentiment isn’t.

No: adding bigrams to unigram presence didn’t improve accuracy (though it didn’t hurt much, even for NB), and bigrams alone were up to 5.8 points worse than unigrams. So bigrams didn’t capture the context the authors expected to matter.

Appending POS tags (crude word sense disambiguation: love in “I love this movie” vs “a love story”) was a wash. The 2633 adjectives alone did worse than the 2633 most frequent unigrams, which nearly matched all 16165 unigrams, despite earlier work’s focus on adjectives.

Tagging words as in the first quarter, last quarter or middle half (reviews often open and close with the verdict) gave results close to plain unigrams; more refined notions of position might help.

Machine learning beats human-chosen word lists; Naive Bayes tended to be worst and SVMs best, but differences were small. Unigram presence was the most effective feature, and none of the alternatives consistently improved on it. Accuracy stayed well below topic categorization.

Authors set up a contrast to earlier text: many sentences praise (or attack) something before the verdict flips, e.g. “This film should be brilliant… However, it can’t hold up.” A human sees the true sentiment, but a bag of features sees many words of the opposite polarity.

Some discourse analysis, or at least determining each sentence’s focus (whether it is about the film itself, i.e. on-topic, a kind of co-reference problem), e.g. for “a good actor trapped in a bad movie”. As Turney put it, “the whole is not necessarily the sum of the parts”.