Every day, we are swamped with a torrent of text data: tweets, reviews, emails, and more. Within these streams of words lies an untapped goldmine of insights - sentiments. They color our texts with shades of emotions, giving us a comprehensive picture of how people feel about products, services, or just about anything that can be written about. With this, we welcome you to the world of sentiment analysis, a key player in the natural language processing (NLP) realm, where technology learns to identify a text's emotional tone. This article will delve into the nuts and bolts of sentiment analysis, its applications, and how it works.
What is sentiment analysis?
Sentiment analysis is a natural language processing technique that aims to determine a text's overall sentiment. It uses machine learning algorithms to identify whether the emotional tone of a text is positive, negative, or neutral.
Sentiment analysis lies at the intersection of linguistics, computer science, and artificial intelligence. It uses the power of computers to help people process large amounts of textual data, such as reviews, social media posts, or customer feedback. Sentiment analysis can assist businesses and individuals in gaining deeper insights into public opinion, brand perception, and market trends, making more data-driven business decisions, and improving customer experience.
Sentiment analysis use cases
Now that we know what sentiment analysis is, let us look at some of its real-life applications.
Social media monitoring
Today, social media platforms generate petabytes of data per day. The ability to extract structured information from this data can give companies a substantial competitive advantage.
For example, researchers from India studied posts from X, formerly Twitter, related to the elections held in 2019. They performed sentiment analysis on the posts to understand the voters' perception of the candidates. The results of this study were significantly correlated with the outcome; the candidate with more positive posts won the election.
Customer feedback analysis
Companies can use sentiment analysis to analyze reviews and determine the product's strengths and weaknesses. This helps to adjust the sales strategy and improve customer experience.
One of the most well-known cases is Coca-Cola's "Happiness Machine" campaign. Coca-Cola used natural language processing (NLP) to analyze tweets from different regions and identified that a little city in Spain was the least happy. So, they put vending machines around that city. People had to give their Twitter (now X) usernames to the machines, which then decided whether to provide the Coke for free or charge for it.
This decision was made by performing sentiment analysis on tweets. If the user's happiness score was high enough, they got the Coke free of charge; if not, they had to pay for it. This campaign generated a lot of hype around the brand and perfectly aligned with the brand's strategy of customers choosing to be happy by buying Coke.
Brand reputation management
You can use sentiment analysis to understand how customers perceive your product, brand, and company. By analyzing customer feedback, you can get invaluable insights that shape your strategies for brand management, reputation management, and customer experience.
Take Starbucks, for example. Every day, many people tweet their emotions and concerns about Starbucks. This massive textual data is analyzed using sentiment analysis. The goal is to understand the customers’ pain points and address them in order to keep the brand's reputation as well as develop marketing strategies. “Analysis of User’s Sentiment Towards Starbucks on Twitter” paper talks about how natural language processing is applied to classify each tweet as positive or negative as well as extract features that are responsible for customers feeling one way or another.
Market research and competitive analysis
You can use sentiment analysis to conduct market research and perform competitor analysis. Brand managers can gain valuable competitive intelligence by analyzing their competitor's social media posts, forums, news articles, review sites, and more. This analysis can help them identify their competitor's strengths, weaknesses, and customer pain points, giving them opportunities for differentiation and improvement. Brand managers can use this information to adjust strategies, refine offerings, and effectively respond to market dynamics, ultimately securing a stronger position in the industry.
How does sentiment analysis work?
Now that you know what sentiment analysis is and its use cases, let us understand how it works. First, we will go over the different types of sentiment analysis and then learn how real-life solutions are built.
Sentiment analysis types
Sentiment analysis can be categorized in different ways:
1. Type of prediction: Do we only need to predict if the sentiment is positive or negative, or do we also need to predict its intensity?
2. Scope of prediction: Do we need to predict the sentiment of a whole document or for different parts of a document?
The first point distinguishes sentiment classification versus intensity ranking. The second point distinguishes between different levels of granularity in sentiment analysis, ranging from the broader document-level analysis to the more specific aspect-level analysis.
Let us first discuss classification on different levels.
Sentiment classification
Sentiment classification is a simple binary classification task where negative sentiments are assigned a negative class, and positive sentiments are assigned a positive class. That way, we can create simple binary classification algorithms to differentiate documents.
Multiclass sentiment analysis
At the same time, positive and negative sentiments can be more specific. For example, excitement and happiness are two different positive sentiments. In that case, we may need to add more classes to our classifier. The number of classes is only limited by the business's and the researcher's needs. So, you can add as many classes as you want.
Neutral sentiment
But sometimes, we may have documents with no specific sentiment. In that case, we may add a class for neutral sentiments. This class is often hard to distinguish from others, especially when the sentiments are unclear. It might sometimes be helpful to create a two-stage algorithm. In the first stage, we differentiate between neutral and other classes to filter out documents with no particular emotion (for example, they are just factual). In the second stage, we classify the remaining documents.
Aspect-based sentiment analysis
Imagine now that we are performing sentence-level analysis of the following review: "I love this phone's screen, but the sound is awful!". Is the sentiment of this sentence positive, negative, or neutral?
This review contains two sentiments: positive, "I love this phone's screen," and negative, "the sound is awful." So, we cannot assign one sentiment for this sentence; we will need to dissect it and assign a sentiment to each section.
This approach is called aspect-based sentiment analysis (or fine-grained sentiment analysis). With aspect-based sentiment analysis, we divide the text data by aspect and identify the sentiment of each one.
Aspect-based sentiment analysis involves:
- Identifying a text's aspects
- Assign a sentiment to each aspect
Below, you can see aspect-based sentiment analysis using DeBERTa fine-tuned with ABSA datasets, and try it yourself.
Intensity ranking
An interesting thing about the classification between positive, negative, or neutral sentiment is that we can look at it as a regression task, where positive values mean positive sentiment, negative values mean negative sentiment, and values near zero may mean neutral sentiment.
We can generalize this approach to more classes. For each class, we will predict the intensity score, which will show us the degree of a given sentiment in the emotional tone of the text.
Sentiment analysis algorithms
Now that you know the types and applications of sentiment analysis, how can you build your solution? Let us start with an overview of different approaches, from rule-based systems to modern deep-learning techniques.
Rule-based sentiment analysis
Rule-based sentiment analysis is an approach that relies on predefined rules to determine sentiment. Usually, domain experts are hired to write the rules, which can be very diverse: positive and negative words or phrases, sentence structures, and emotions. From these rules, meta-rules may be created.
Since domain experts write the rules, they do not need to use as much data as statistical models require because they have the knowledge and do not need to extract it from texts. However, with their knowledge, they also carry some biases.
Also, since humans write the rules, they are (mostly) inherently interpretable, so the users can easily comprehend the decision-making process. But as time passes, rule sets may become very complex and hard to maintain.
Machine learning algorithms
Machine learning techniques can help us overcome the problems mentioned above. They do not require domain experts to write rules for a system (however, data scientists are needed) because the machine learning algorithm can extract knowledge from data by itself. And here comes the downside: ML sentiment analysis algorithms require training data to learn and understand patterns in text data. And the more complex the algorithm you use, the more data you need. You can read our blog to learn more about how you can get training data for your sentiment analysis datasets. Also, check out our blog post about the best text data annotation tools.
Another good thing about machine learning models is that they are much simpler to maintain than a vast set of rules because you only have to maintain a single model (or a small number of models). This makes your sentiment analysis system's support and maintenance much simpler.
Aside from that, machine learning models can use rules as input features. Having rule-based approaches working together with ML algorithms helps you combine the deep understanding of specialists and the ability of machine learning algorithms to gain deeper insights from massive datasets that humans alone could not analyze.
But what about interpretability? Well, it will not be as simple as with rule-based systems. But popular models, such as random forests or support vector machines, let you inspect feature importances easily, so most of the time, there will not be any issues interpreting simple models for ML sentiment analysis.
Deep learning sentiment analysis models
Deep learning models have gained significant popularity in the field of sentiment analysis. Neural networks are trying to mimic the human brain with billions of neurons and synapses, making their ability to capture complex patterns in large-scale datasets undisputable.
Recent breakthroughs such as transformer models allowed researchers to train large language models (LLM) on terabyte-scale raw text data to extract knowledge about how human language works efficiently. With such knowledge, transformer models achieved state-of-the-art results in every field of natural language processing, including sentiment analysis.
After reading vast amounts of text, LLMs become experts. You will not need to hire field experts like linguists, psychologists, etc., because LLMs would already be fluent in domain-specific knowledge. In general, sentiment analysis based on deep learning performs much better than sentiment analysis that works with the classical ML approach.
But everything comes at a cost. LLMs are compute-intensive, and orders of magnitude are more demanding than in the case of classical algorithms. So, the costs of serving this kind of model are very high. The company must decide: Is this quality improvement worth the additional costs?
One additional issue with deep learning models is their inherent difficulty in interpretation. The human brain cannot track billions of parameters, and we still do not know how to apply mathematics to interpret them reliably. If interpretability is an issue for you, you should stick to the classical sentiment analysis model.
Sentiment analysis challenges
Sentiment analysis tasks seem straightforward now. A data scientist has to get data and train a model with it, right? Well, not exactly. Many challenges make performing sentiment analysis more complex. Some of these challenges are specific to sentiment analysis, and some are common to any natural language processing task.
Let us have a look at some of those challenges.
Irony and sarcasm
The internet is full of irony and sarcasm, and sometimes, it is challenging to understand whether a post is genuine or sarcastic. This poses a significant challenge for sentiment analysis. Irony and sarcasm can skew the otherwise accurate sentiment analysis model and turn sentiment analysis results upside down.
It may be helpful to use a sarcasm detection tool and then conduct sentiment analysis.
Fake reviews
Another major challenge in sentiment analysis is the prevalence of fake online reviews, which can significantly hinder the accuracy and reliability of the task. Fake reviews intentionally deceive readers and manipulate the overall sentiment associated with a product, service, or brand. These reviews may be generated by individuals with ulterior motives, such as competitors trying to ruin the reputation of a business or companies attempting to promote their products dishonestly. The presence of fake reviews introduces noise and bias into sentiment analysis models, making it challenging to understand genuine customer opinions accurately.
Low-resource languages
This is a common problem in natural language processing, which only appears in pre-trained models. If the language is very rare, the model will not have enough data to learn that language well, and the accuracy of the sentiment analysis will suffer. If you have enough training data, this is usually fine for rule-based and classical machine learning approaches.
Misspellings
Humans are not perfect: we make spelling mistakes due to our language skills, which are not always perfect, or by accident. It is important to note that spelling mistakes are harmful to sentiment analysis.
Solving such mistakes is a challenging task. It is not always obvious which word should be placed instead of the misspelled one. Sometimes, you need help understanding if the word was really misspelled. Automatic spell-checkers and correction algorithms exist, but they are not foolproof. They heavily rely on predefined dictionaries or statistical models, which may not take into account uncommon or specialized vocabulary. This limitation becomes more evident when dealing with informal language, slang, or domain-specific jargon, where misspellings can be more frequent.
Sentiment analysis tools and resources
Today, data science provides many possibilities to perform sentiment analysis manually or using sentiment analysis APIs. This section shows you some tools that you can use in your project.
Open-source libraries
Rule-based
For rule-based analysis, you can use the vaderSentiment library. It provides you with an out-of-the-box system for applying sentiment analysis. It already has predefined dictionaries of positive and negative words and rules for prediction.
Machine learning
If you want to use pre-trained ML models, try using the spacytextblob component for spaCy to perform sentiment analysis easily. Follow the link to see the examples.
For machine learning solutions, you will need to leverage a whole bunch of libraries. A typical pipeline for building a machine learning sentiment analysis solution involves the following steps:
1. Get annotated data
2. Split your data into a train set and a test set
3. Preprocess the data
4. Train your model on the training set
5. Evaluate the performance of your model on the test set
6. The most famous library to train ML models is scikit-learn. You can also use their documentation as your data science handbook. It also allows you to perform train-test splitting, model evaluation, and some preprocessing.
For more complex preprocessing, use the Natural Language Toolkit (or more commonly NLTK). It provides functionality for smart text tokenization and normalization.
Deep learning
Today's standard for deep learning text analysis is the HuggingFace library. It provides many pre-trained models for sentiment analysis and allows you to finetune any model to use with your data easily. But if you would like to perform model training, remember to bring some GPUs since transformer models are resource-demanding.
SaaS (Software as a Service) sentiment analysis tools
IT giants can provide you with powerful tools out of the box. Amazon Comprehend is a good example. Aside from simple sentiment analysis, it can perform fine-grained sentiment analysis and much more.
Google's Cloud Natural Language provides approximately the same capabilities, so choose whose solution you like more and which suits your task. And remember that you will pay for each request, so pay attention to your budget.
What to choose?
The answer depends a lot on what you are trying to achieve. Suppose you want to spend less time on dataset collection, model training, and testing or need a budget to hire a team of data scientists to develop and support your sentiment analysis system. In that case, you should go for out-of-the-box sentiment analysis platforms unless your workload doesn't consist of hundreds of millions of requests per month.
If you can spend time writing, testing, and supporting your service, try going with pre-trained models from spaCy of HuggingFace. They provide decent performance but require more time before you can use them.
If your dataset is specific and pre-trained models do not meet your needs, you should train your own model.
Summary
Sentiment analysis is a powerful tool that tackles emotions in text and is used to understand public opinion, brand perception, market trends, and more. It can identify positive, negative, and neutral sentiments in text data and the intensity of those sentiments. This information can be used by businesses to make more informed decisions about product development, marketing, and customer service.
However, sentiment analysis faces challenges, such as irony and sarcasm, fake reviews, and misspellings, and how these challenges make the sentiment analysis process more challenging.
If you want to learn more about natural language processing, you can check out our in-depth article, which provides more context about various NLP tasks and ways to solve them.
