Question 1

What is The Data Structure of Batch Analysis Results for Opinion Mining

Accepted Answer

What is Aspect-based Sentiment Analysis

Aspect-based sentiment analysis refers to the task of identifying fine-grained opinion towards a specific topic in text. This text can take a variety of formats such as emails, product reviews, customer requests, social media posts, and any other user generated content. This approach to text analysis is considered the industry standard for analyzing customer reviews as seen in the image below.

https://malcolm-en-gb.s3.eu-west-1.amazonaws.com/instances/oiJS2eQZyE/resources/tIC9tp5U72/Amazon_review_absa.png (Amazon_review_absa.png)

How A Customer Review is Structured

A customer review consists of two main parts; the review text and review description or metadata such as name, date, rating, etc. This universally applies to most online reviews including Amazon product reviews, Google Maps company reviews, tweets, and more.

https://malcolm-en-gb.s3.eu-west-1.amazonaws.com/instances/oiJS2eQZyE/resources/l4s3zEcKGK/Amazon_review_detailed.png (Amazon_review_detailed.png)

Reviews tend to be dense with mentions of topics, emotions, and sentiments as customers are expressing their experience in a short message. You would hence find reviews that include multiple sentences. Therefore, we unpack a review to its core elements which we refer to as segments (eg, sentences). This way we ensure high accuracy and provide you with all the essential analysis you require to know about a review.

https://malcolm-en-gb.s3.eu-west-1.amazonaws.com/instances/oiJS2eQZyE/resources/mZxanVYcir/Review_text_segments.png (Review_text_segments.png)

To get fine-grained analysis the review is broken-down to 3 levels. Level 1 is the whole review message, Level 2 is a breakdown of the segments in the review text, and Level 3 are the topics and sentiments associated with each segment which can be none, one, or more. The diagram below illustrates this hierarchy.

https://malcolm-en-gb.s3.eu-west-1.amazonaws.com/instances/oiJS2eQZyE/resources/z4o5iQ1knZ/Absa_structure.png (Absa_structure.png)

Your analysis results will also include a unique ID for each level to give it a convenient structure and make working with it easy in your desired application.

https://malcolm-en-gb.s3.eu-west-1.amazonaws.com/instances/oiJS2eQZyE/resources/O1A0Q5wNuX/Review_absa_structure.png (Review_absa_structure.png)

How Our Analysis Results are Structured

It is common to upload a file with 100 rows (1 review per row) and download results with 300 rows for example. This is because each of these reviews would have multiple sentences that can mention 1 or more topics. To make results easy to work with, we add a new row for each topic identified in each segment.

For example, if you have a review consisting of 2 sentences and each sentence has 2 topics mentioned, you will end with 4 rows for this review. The review text will appear 4 times, each segment will appear twice, and each topic is considered unique to a segment. This way you get a layered/nested results ready for your following activities.

Example of input file with 50 rows (50 reviews)

https://malcolm-en-gb.s3.eu-west-1.amazonaws.com/instances/oiJS2eQZyE/resources/8FBcj8CvX2/Screen%20Shot%202021-10-30%20at%203.07.06%20PM.png (Screen Shot 2021-10-30 at 3.07.06 PM.png)

Example of output file with 280 rows (50 reviews)

Having more rows in the results is an indicator of the richness of the data. You will also notice some column headers are highlighted in blue color. These columns are generated by DeepOpinion Studio.

https://malcolm-en-gb.s3.eu-west-1.amazonaws.com/instances/oiJS2eQZyE/resources/3d2iVLcpzl/Screen%20Shot%202021-10-30%20at%2012.27.08%20PM.png (Screen Shot 2021-10-30 at 12.27.08 PM.png)

Results Data Dictionary

Here is a description of the columns you can expect from your analysis.

id [Level 1]: A system generated unique ID for each original row (document or review)

Text [Level 1]: The original text uploaded for automated analysis (document or review)

Segment id [Level 2]: A system generated unique ID for each segment in the original row (child of id)

Segment text [Level 2]: A segment of the original text uploaded for automated analysis (child of text)

Span start [Level 2]: The character count of where a segment starts (child of text)

Span end [Level 2]: The character count of where a segment ends (child of text)

Tag id [Level 3]: A system generated unique ID for each topic identified in a text segment (child of segment id)

Aspect [Level 3]: A topic identified in a text segment (child of segment text)

Sentiment [Level 3]: A sentiment category identified in a text segment (child of segment text)

Others [Level 1]: Any other column included in the original data such as dates, names, ratings, and source (metadata)

Why This Structure

This data structure has been carefully designed to make the results convenient to work with in downstream tasks such as in process automation or data visualization. For example, you might be interested in counting how often do your customers complain about battery life and auto-create a support ticket for each time this is mentioned.

https://malcolm-en-gb.s3.eu-west-1.amazonaws.com/instances/oiJS2eQZyE/resources/IGmg4MO3NV/Ticket_automation_freshdesk_animated.gif (Ticket_automation_freshdesk_animated.gif)

Contact us to know more about moving your customer review from static insights to dynamic and automated actions

Question 2

Data Labelling Guideline for Aspect-based Sentiment Analysis

Accepted Answer

Part: High-Quality Training Data Labelling for Opinion Mining Models

In this tutorial we present to you some tips on data labelling to ensure that your model can reach to high accuracy. We will start with the purpose of data labelling, cover 7 principles to follow, and tell you about our cool "Active Learning" feature that can help you automate part of this process.

The Purpose of Data Labelling

Humans learn best by examples. AI models learn the same way!

You can recall from your learning experience that you get to learn a concept by seeing example to experience this concept in its original context and then it is easy for you to make sense of it. The same thing applies to AI model training, you are showing the AI model examples of where a topic and a sentiment appear in text (it's original context). This way the AI model can pick up those and sentiments when appearing in similar sentences. The more examples the better the AI is learning and able to generalize what it learned.

https://malcolm-en-gb.s3.eu-west-1.amazonaws.com/instances/oiJS2eQZyE/resources/99HyRyLCDm/annotation%20picture.png (annotation picture.png)

Therefore the job of the data annotator is to help the AI model in associating sentences with concepts (topics and sentiments) as shown in this image below

https://malcolm-en-gb.s3.eu-west-1.amazonaws.com/instances/oiJS2eQZyE/resources/JFiS4b3p6C/Screen%20Shot%202021-10-14%20at%209.21.03%20PM.png (Screen Shot 2021-10-14 at 9.21.03 PM.png)

Labelling Data on DeepOpinion Studio

DeepOpinion Studio offers a user friendly interface that helps you to label data more efficiently and in an interactive way. Compared to other tools, DeepOpinion approach in creating a new model is 10X more efficient. When creating a new model, on step 3 "Training Data". Here are the data labelling steps:

Navigate to Step 3 "Training Data"
Click on "NEW LABELLING SESSION +" button on the top right
On the pop-up window choose session type to be "Annotation" and click on "CREATE LABELLING SESSION" button
A new session will be added to the page which you can click on to start labelling training data
The text box on the top right is what you have to label as shown on the below image
Read the sentence that is highlighted and select the topic and sentiment from the menu below the text
Once you finish labelling this session you will be notified and taken to the home page

Tip 1: You can label the same sentence with multiple topics and different sentiments if they appear in the text.

Tip 2: When starting, aim to label 10 sessions as a minimum and then train your first model.

https://malcolm-en-gb.s3.eu-west-1.amazonaws.com/instances/oiJS2eQZyE/resources/lV3rJqigKJ/annotation%20interface.svg (annotation interface.svg)

The 7 Principles for High-Quality Training Data

Involve domain experts in the process
Define topics to be comprehensive and non-overlapping in coverage
Write definitions for each topic and add examples
Annotate based on explicit mentions of topics with sentiments (= opinion)
Ignore implicit and subjective topics or factual statements
Focus on producing high quality labels, take breaks to refresh focus
Iterate between annotating and training to speed up the process

1. Involve domain experts

Involving domain experts is important to ensure that this expertise is reflected on the training data and hence giving the AI model a good chance to mimic the domain expert. The job title of this domain expert depends on the specific use case. For example when creating a customer review analysis model it is the Customer Experience Manager or for employee feedback it would be the HR Manager.

2. Define topics to be comprehensive and non-overlapping in coverage

Topics should be designed in a way that reflects the what is actually expressed in the text data. Doing this step well means that your list of topics mutually exclusive in meaning and collectively exhaustive in covering all the topics that appears in the text. Therefore, avoid having topics that are similar and can overlap in meaning. This illustration shows how you can cover the content of the text with topics.

https://malcolm-en-gb.s3.eu-west-1.amazonaws.com/instances/oiJS2eQZyE/resources/H6NjkhyFqr/Screen%20Shot%202021-10-14%20at%2010.09.53%20PM.png (Screen Shot 2021-10-14 at 10.09.53 PM.png)

3. Write definitions for each topic and add examples

It is a good practice to define topics and adding examples to understand how a topic appears in your text and label in a consistent way. When you label data and unsure if a topic is relevant or not you can refer to it. Additionally it helps other people in labelling the data in a consistent way too.

https://malcolm-en-gb.s3.eu-west-1.amazonaws.com/instances/oiJS2eQZyE/resources/1K2vp01qVv/Aspect%20Definition.svg (Aspect Definition.svg)

4. Annotate based on explicit mentions of topics with sentiments (= opinion)

Make sure that when labelling you only select a topic that is explicitly mentioned in the text. This is because AI models can only process and learn from information that is really present in the text.

https://malcolm-en-gb.s3.eu-west-1.amazonaws.com/instances/oiJS2eQZyE/resources/JFiS4b3p6C/Screen%20Shot%202021-10-14%20at%209.21.03%20PM.png (Screen Shot 2021-10-14 at 9.21.03 PM.png)

5. Ignore implicit and subjective topics or factual statements

The AI model can only learn from the text it sees and has no access to external knowledge. Therefore, avoid labelling with topics that do not appear in the text to avoid confusing the model. For example, if a customer review says "why didn't I see this product before" you shouldn't label "Marketing" with "Negative" sentiment as marketing is not referred to in the text.

https://malcolm-en-gb.s3.eu-west-1.amazonaws.com/instances/oiJS2eQZyE/resources/lhDKGzCDl6/Screen%20Shot%202021-10-14%20at%2010.33.30%20PM.png (Screen Shot 2021-10-14 at 10.33.30 PM.png)

6. Focus on producing high quality labels, take breaks to refresh focus

This point is self explanatory, taking breaks helps you to stay focused to produce consistent labels that the AI model can mimic.

7. Iterate between annotating and training to speed up the process

Iterate on this process by annotating ~100 examples at a time until you reach the desired performance. As a rule of thumb, aim to label at least 30 examples for each topic so that the AI model can see enough variety of example where such a topic appears. This way the AI model can reach a high accuracy. the data labelling and model training workflow is optimized to help you run through this process efficiently. You can expect to complete training a model from scratch in a matter of few hours. This is how the process looks like.

https://malcolm-en-gb.s3.eu-west-1.amazonaws.com/instances/oiJS2eQZyE/resources/dRKsa3NnnM/Workflow_Graphic_AutoNLU-2525951.png (Workflow_Graphic_AutoNLU-2525951.png)

Active learning is here to automate data labelling

Our goal is to minimize the manual effort in creating a model. Therefore, we released a new feature that helps in automating the data labelling step. In this case, once you label some data and train a model with at least 50% you can switch to "Active Learning". This is how it works:

The model go through all the unlabelled text and find the top examples that it is least confident about
It will predict the labels for topic and sentiment
You will then get to confirm the predicted labels by either ticking the checkbox of correcting the labels if the model got it wrong

This way you provide the model with the most impactful examples to improve its accuracy. Additionally this way you reduce your labelled effort by 5X compared to other conventional ways.

https://malcolm-en-gb.s3.eu-west-1.amazonaws.com/instances/oiJS2eQZyE/resources/lt1UJD4w5u/active%20learning.svg (active learning.svg)

As always, please remember to share your feedback and any questions you have.

Question 3

How to define aspects efficiently.

Accepted Answer

Summary

Defining the model’s aspects efficiently is essential for annotating data correctly and eventually building a state-of-the-art model. We can create the aspects for you. You might have them created or build them together. In any case, we share here seven tips for making the most pertinent aspects for your custom models!

Tip 1: Prominence

Try to capture topics that "stick out" and be more specific wherever this is possible. For example, in the software industry, we could use different aspects such as “app”, “desktop app” or “mobile app”, depending on how detailed the data is. On the other hand, if the information generally only mentions an app, we can’t be more specific, and we should call the aspect something like “app general”.

Tip 2: Overlapping

Be aware of overlapping meaning to avoid creating aspects corresponding to similar content. For example, the aspects “support” and “customer center” would be overlapping factors for customer data, as they would be used on the same text segments during labeling.

Tip 3: Removing an aspect

Make sure that a label is relevant to your data, and you use it during labeling before you add it to your aspects list. After spending some time on data annotation, you can permanently remove it from your aspects list if you realize that you never use an aspect. For example, if I have customer data and define the aspect “mobile app” then notice that the particular aspect is not used during labeling, I should remove it.

Tip 4: Adding an aspect

While annotating data, you might feel like an aspect is missing from the aspects list as you would like to use it during labeling. In this case, you can always add the needed aspect to your aspects list. Just make sure that you need this aspect and that there is more than one example where it appears.

Tip 5: Difficulty

Try to define an aspect with a simple word and not use complicated language or abstract concepts. For example, if you have to define aspects for a financial model, it is better to determine “expenses” than “deduction”.

Tip 6: Subjective opinion space

Keep in mind that we try to cover as much personal opinion space as we can. Meaning that people talk in a specific way, using particular words, and are interested in specific aspects of a product depending on the product’s industry. For example, people usually talk about the “engine”, the “wheels/tires”, etc. Our goal is to capture these aspects, which are essential for a product, to process as many text/reviews as possible. By capturing these aspects, we understand how the market “talks” about a product.

Tip 7: Annotators' cognitive effort

Lastly, since the data will be labeled manually by annotators, we should consider their cognitive effort. Creating many aspects that are difficult to understand will significantly increase the annotators’ work and lead to a more flawed result. For this reason, we should consider whether we would be happy with the existing aspects list as an annotator before we finalize our aspects list.