Sentiment analysis is the process of identifying people’s attitudes and emotional states from language. In Natural Language Processing, sentiment analysis is an automated task where machine learning is used to rapidly determine the sentiment of large amounts of text or speech. Applications include tasks like determining how excited someone is about an upcoming movie; correlating statements about a political party with people’s likeliness to vote for that party; or converting written restaurant reviews into 5-star scales across categories like ‘quality of food’, ‘a(chǎn)mbience’, and ‘value for money’.

With the amount of information that is shared on social media, forums, blogs, etc, it is easy to see why we need to automate sentiment analysis: there is simply too much information to manually process. The problem is that machine learning approaches are typically not that accurate. For a simple task separating ‘positive’ from ‘negative’ sentiment on social media, many automated solutions only perform with around 80% accuracy. This can still be useful for tracking broad trends over time, but it limits fine-grained analysis.

Some natural language processing tasks are much easier to automate. For example, imagine if you were separating articles about basketball from articles about football. There are many features that an automated approach can use to correctly guess which sport an article is about: the terminology specific to each sport, names of famous players, etc. We expect automated approaches to be almost equal to human performance for this kind of topic identification, even if they look at nothing more than the collection of words being used.

Mixed Sentiment

The subtleties of sentiment

The way that we express sentiment is a complex mix of the linguistic structures of our utterances and the assumed knowledge of the people who we are addressing. Unlike topic identification, pulling out a few key phrases like “good”, “bad” and “I love/hate” will only get you to around 70% accuracy. You need the machine learning algorithms to have a deeper understanding of the context. Compare these examples:

1. It was a great restaurant.
2. It should have been a great restaurant.
3. The restaurant was great in that it will make all future meals seem more delicious.
4. Despite a pleasant experience I can’t support the many reviews that it was a great restaurant.

The first sentence is positive, but the rest are all negative sentiment, despite having the first sentence embedded within them. The second sentence “should have been” indicates the desired outcome, leaving the actual sentiment implied (it’s possible that this sentence could followed by “… and it was!”, but the implication is that it was not). The third is the more complicated, bordering on the kinds of sarcasm that are very hard for machines to identify: even a person might misread this as positive when skimming quickly. The fourth statement is even more complicated because the overall sentiment is negative, but it begins with the (weakly) positive “pleasant experience” and also finishes by reporting that many other people expressed positive sentiment.

Abstracting beyond words and phrases

They key to understanding more complicated expressions is to allow the machine learning algorithms to understand language at a deeper level. For the fourth example above, this is possible by understanding the syntactic and semantic structures of the sentence itself. A recent paper by Richard Socher and colleagues at Stanford University shows a system that does exactly this:

Demonstration of negative sentiment taking scope over positive sentiment

The image is taken with permission from http://nlp./sentiment/index.html, which contains a live demo of the work of Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher Manning, Andrew Ng and Christopher Potts, to be presented in the paper “Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank” at this year’s Conference on Empirical Methods in Natural Language Processing (EMNLP 2013).* The demo is fun to play around with: I recommend that you try it out!

The tree-structure in the image shows the recursive structure of the sentence itself, and how a correct parse will mean that the negation in the main verb of the sentence (“can’t support”) has scope over the positive phrases embedded within. The resulting orange-color of the root of the tree indicates that the sentiment was correctly identified as ‘negative’, meaning that Socher et al’s system gets this one correct. The results of their system improve previous systems by 5%, reaching about 85% accuracy. This might not sound like a lot, but research projects in Natural Language Processing typically only improve accuracy by around 1% a year (at best) so this is a significant leap.

Why so serious?

These examples are typical of the ways in which people express negative sentiment. In some cases, people choose to pad their criticisms with qualifiers to be polite (even on the internet), and more broadly, people just tend to be more creative in how they choose to describe the things that they don’t like. If Tolstoy worked on the problem he would no doubt say:

Happy sentiments are all alike; every unhappy sentiment is uniquely expressed.

This makes negative sentiment particularly tough for machine learning, as machine learning typically works by learning from examples. The greater the variety of the potential examples, the more likely that an ambiguous sentence appears, and the lower the accuracy as a result. Abstractions like the example above get us part of the way, but we are not yet at the point of reliably detecting sarcasm, irony, and other forms of expression where the literal meaning is opposite to the intended.

These socially determined negations get particularly complicated when we look across different languages, as every language does this a little differently. For example, in English we might call something ‘little’ when it is actually large (think “Little John” in Robin Hood), but in Spanish you might use the diminutive suffix ‘-ita’. Even within English, we get cultural differences, especially with the famously understated British English:

With thanks to the (unknown) author of this table, first posted by Duncan Green of Oxfam.

Despite the differences to English in the US and elsewhere, it is relatively easy for machines to learn that a “bit disappointed” means “annoyed” after the algorithms have seen a few examples. The crucial point is to know to apply ‘British English’ criteria. This is the most important next step in sentiment analysis: automatically knowing what kind of analysis to apply, depending on the genre, language or source of the utterance. By knowing something about the social and cultural context of the utterance, we can make smarter assumptions about the assumed the knowledge of speaker and more accurately tailor the sentiment predictions to specific types of communication. As for how we should do this in an automated system? We’ll leave that for a future post.

– Rob Munro

* Disclosure: two of the authors, Christopher Manning and Christopher Potts, are advisors to Idibon

Rob drives the company’s vision and strategic direction. He is a leader in applying big data analytics to human communications, having worked in diverse environments, from Sierra Leone, Haiti and the Amazon to London, Sydney and San Francisco. He completed a PhD in Computational Linguistics as a Graduate Fellow at Stanford University. Outside of work, he learned about the world’s diversity by cycling more than 20,000 kilometers across 20 countries.