Question 1

Data Labelling Guideline for Aspect-based Sentiment Analysis

Accepted Answer

Part: High-Quality Training Data Labelling for Opinion Mining Models

In this tutorial we present to you some tips on data labelling to ensure that your model can reach to high accuracy. We will start with the purpose of data labelling, cover 7 principles to follow, and tell you about our cool "Active Learning" feature that can help you automate part of this process.

The Purpose of Data Labelling

Humans learn best by examples. AI models learn the same way!

You can recall from your learning experience that you get to learn a concept by seeing example to experience this concept in its original context and then it is easy for you to make sense of it. The same thing applies to AI model training, you are showing the AI model examples of where a topic and a sentiment appear in text (it's original context). This way the AI model can pick up those and sentiments when appearing in similar sentences. The more examples the better the AI is learning and able to generalize what it learned.

https://malcolm-en-gb.s3.eu-west-1.amazonaws.com/instances/oiJS2eQZyE/resources/99HyRyLCDm/annotation%20picture.png (annotation picture.png)

Therefore the job of the data annotator is to help the AI model in associating sentences with concepts (topics and sentiments) as shown in this image below

https://malcolm-en-gb.s3.eu-west-1.amazonaws.com/instances/oiJS2eQZyE/resources/JFiS4b3p6C/Screen%20Shot%202021-10-14%20at%209.21.03%20PM.png (Screen Shot 2021-10-14 at 9.21.03 PM.png)

Labelling Data on DeepOpinion Studio

DeepOpinion Studio offers a user friendly interface that helps you to label data more efficiently and in an interactive way. Compared to other tools, DeepOpinion approach in creating a new model is 10X more efficient. When creating a new model, on step 3 "Training Data". Here are the data labelling steps:

Navigate to Step 3 "Training Data"
Click on "NEW LABELLING SESSION +" button on the top right
On the pop-up window choose session type to be "Annotation" and click on "CREATE LABELLING SESSION" button
A new session will be added to the page which you can click on to start labelling training data
The text box on the top right is what you have to label as shown on the below image
Read the sentence that is highlighted and select the topic and sentiment from the menu below the text
Once you finish labelling this session you will be notified and taken to the home page

Tip 1: You can label the same sentence with multiple topics and different sentiments if they appear in the text.

Tip 2: When starting, aim to label 10 sessions as a minimum and then train your first model.

https://malcolm-en-gb.s3.eu-west-1.amazonaws.com/instances/oiJS2eQZyE/resources/lV3rJqigKJ/annotation%20interface.svg (annotation interface.svg)

The 7 Principles for High-Quality Training Data

Involve domain experts in the process
Define topics to be comprehensive and non-overlapping in coverage
Write definitions for each topic and add examples
Annotate based on explicit mentions of topics with sentiments (= opinion)
Ignore implicit and subjective topics or factual statements
Focus on producing high quality labels, take breaks to refresh focus
Iterate between annotating and training to speed up the process

1. Involve domain experts

Involving domain experts is important to ensure that this expertise is reflected on the training data and hence giving the AI model a good chance to mimic the domain expert. The job title of this domain expert depends on the specific use case. For example when creating a customer review analysis model it is the Customer Experience Manager or for employee feedback it would be the HR Manager.

2. Define topics to be comprehensive and non-overlapping in coverage

Topics should be designed in a way that reflects the what is actually expressed in the text data. Doing this step well means that your list of topics mutually exclusive in meaning and collectively exhaustive in covering all the topics that appears in the text. Therefore, avoid having topics that are similar and can overlap in meaning. This illustration shows how you can cover the content of the text with topics.

https://malcolm-en-gb.s3.eu-west-1.amazonaws.com/instances/oiJS2eQZyE/resources/H6NjkhyFqr/Screen%20Shot%202021-10-14%20at%2010.09.53%20PM.png (Screen Shot 2021-10-14 at 10.09.53 PM.png)

3. Write definitions for each topic and add examples

It is a good practice to define topics and adding examples to understand how a topic appears in your text and label in a consistent way. When you label data and unsure if a topic is relevant or not you can refer to it. Additionally it helps other people in labelling the data in a consistent way too.

https://malcolm-en-gb.s3.eu-west-1.amazonaws.com/instances/oiJS2eQZyE/resources/1K2vp01qVv/Aspect%20Definition.svg (Aspect Definition.svg)

4. Annotate based on explicit mentions of topics with sentiments (= opinion)

Make sure that when labelling you only select a topic that is explicitly mentioned in the text. This is because AI models can only process and learn from information that is really present in the text.

https://malcolm-en-gb.s3.eu-west-1.amazonaws.com/instances/oiJS2eQZyE/resources/JFiS4b3p6C/Screen%20Shot%202021-10-14%20at%209.21.03%20PM.png (Screen Shot 2021-10-14 at 9.21.03 PM.png)

5. Ignore implicit and subjective topics or factual statements

The AI model can only learn from the text it sees and has no access to external knowledge. Therefore, avoid labelling with topics that do not appear in the text to avoid confusing the model. For example, if a customer review says "why didn't I see this product before" you shouldn't label "Marketing" with "Negative" sentiment as marketing is not referred to in the text.

https://malcolm-en-gb.s3.eu-west-1.amazonaws.com/instances/oiJS2eQZyE/resources/lhDKGzCDl6/Screen%20Shot%202021-10-14%20at%2010.33.30%20PM.png (Screen Shot 2021-10-14 at 10.33.30 PM.png)

6. Focus on producing high quality labels, take breaks to refresh focus

This point is self explanatory, taking breaks helps you to stay focused to produce consistent labels that the AI model can mimic.

7. Iterate between annotating and training to speed up the process

Iterate on this process by annotating ~100 examples at a time until you reach the desired performance. As a rule of thumb, aim to label at least 30 examples for each topic so that the AI model can see enough variety of example where such a topic appears. This way the AI model can reach a high accuracy. the data labelling and model training workflow is optimized to help you run through this process efficiently. You can expect to complete training a model from scratch in a matter of few hours. This is how the process looks like.

https://malcolm-en-gb.s3.eu-west-1.amazonaws.com/instances/oiJS2eQZyE/resources/dRKsa3NnnM/Workflow_Graphic_AutoNLU-2525951.png (Workflow_Graphic_AutoNLU-2525951.png)

Active learning is here to automate data labelling

Our goal is to minimize the manual effort in creating a model. Therefore, we released a new feature that helps in automating the data labelling step. In this case, once you label some data and train a model with at least 50% you can switch to "Active Learning". This is how it works:

The model go through all the unlabelled text and find the top examples that it is least confident about
It will predict the labels for topic and sentiment
You will then get to confirm the predicted labels by either ticking the checkbox of correcting the labels if the model got it wrong

This way you provide the model with the most impactful examples to improve its accuracy. Additionally this way you reduce your labelled effort by 5X compared to other conventional ways.

https://malcolm-en-gb.s3.eu-west-1.amazonaws.com/instances/oiJS2eQZyE/resources/lt1UJD4w5u/active%20learning.svg (active learning.svg)

As always, please remember to share your feedback and any questions you have.

Question 2

How do I start with DeepOpinion Studio?

Accepted Answer

To get actionable insights from text data, you need to get familiar with DeepOpinion Studio.

Once you're logged in, you can see your models and all the available pre-trained models, which change to "My Models" when starring them. First, select a model you wish to use.

After choosing a model, you're in the demo menu of this particular model, where you can see a description and all pre-defined and pre-trained aspects (click on "classification categories"). You can select a demo document or insert your text example to convince yourself of the model’s capabilities in the section below.

In the next section of the menu, called "batch" you can start analyzing more extensive data sets to derive actionable insights. Simply upload your data set(s) in the "upload data" section and follow the instructions. The bell icon in the upper right corner will let you know once the analysis is completed and the results are ready to download.

Please keep in mind that the amount of data you can analyze depends on your subscribed plan (for more info, click "update" in the menu).

Question 3

How do I create a label model for customer support ticket categorization on DeepOpinion Studio

Accepted Answer

Create your own customer support ticket categorization AI model easily on DeepOpinion Studio in minutes! This tutorial will follow the steps shown in the diagram below. As we will be using historical data from customer support we will skip the "ANNOTATE" step.

https://malcolm-en-gb.s3.eu-west-1.amazonaws.com/instances/oiJS2eQZyE/resources/dRKsa3NnnM/Workflow_Graphic_AutoNLU-2525951.png (Workflow_Graphic_AutoNLU-2525951.png)

This short tutorial explains how to train an AI model on categorizing customer support tickets from the banking industry. For example, we want to determine what the request is about (eg, lost credit card, banking app not working, etc) so that it gets directly assigned to the responsible team to handle it. The same steps can be followed to train any label model where one 1 label is only assigned to a document.

Important: download to follow the tutorial and do it yourself.

Click Here to Download The Banking Customer Support Tickets Training Data

Follow this step-by-step guide to learn how:

Leave us a comment you would like more details.

Question 4

How do I create a custom model on DeepOpinion Studio?

Accepted Answer

Create your own custom text analysis model easily on DeepOpinion Studio in a few hours. or even minutes if you already have training data! This tutorial will follow the steps shown in the diagram below.

https://malcolm-en-gb.s3.eu-west-1.amazonaws.com/instances/oiJS2eQZyE/resources/dRKsa3NnnM/Workflow_Graphic_AutoNLU-2525951.png (Workflow_Graphic_AutoNLU-2525951.png)

This short tutorial explains how to a custom model for analyzing hotels reviews. The same steps can be followed to customize any model. You can download this Hotel Reviews Training Data to do it yourself.

Follow this step-by-step guide to learn how:

Leave us a comment you would like more details.

Question 5

How do I create a document information extraction model on DeepOpinion Studio

Accepted Answer

Create a document information extraction as a cognitive skill on DeepOpinion Studio then upload and label your documents to be able to train a model.

Question 6

How Do I Automate My Customer Service Requests on Freshdesk

Accepted Answer

Automate your Freshdesk tickets routing, tagging, and prioritization DeepOpinion Studio in 4 simple steps.

The 3 steps are:

Creat a free Freshdesk Support Account
Select your input trigger, which in this case is receiving a new ticket on freshdesk
Choose the model you want to perform the ticket categorization. You can also train a model on your freshdesk tickets historical data
Select your output, which in this case would be updating a field on the freshdesk ticket

Follow this step-by-step guide to learn how:

Leave us a comment you would like more details.

Question 7

How Do I Create A Freshdesk Ticket Automation on DeepOpinion Studio?

Accepted Answer

Automate your Freshdesk tickets routing, tagging, and prioritization DeepOpinion Studio in 3 simple steps.

The 3 steps are:

Select your input trigger, which in this case is receiving a new ticket on freshdesk
Choose the model you want to perform the ticket categorization. You can also train a model on your freshdesk tickets historical data
Select your output, which in this case would be updating a field on the freshdesk ticket

Follow this step-by-step guide to learn how:

Leave us a comment you would like more details.

Question 8

How can I evaluate my cognitive automation

Accepted Answer

What is Cognitive Automation

If you ever experienced a daunting task at work that you believe should be automated in the AI era, then you already know Cognitive Automation. Cognitive automation refers to the use of AI in automating repetitive tasks and processes that are alternatively performed manually be a person or even a whole team.

There are three component that come together to make this happen:

1. AI: To perform the core task that require human intelligence such as comprehending and analyzing text

2. Automation: To execute a task or a workflow based on the AI outcomes

3. Application: The system where this task or activity should be performed such as ERP, CRM, or a ticketing system

https://malcolm-en-gb.s3.eu-west-1.amazonaws.com/instances/oiJS2eQZyE/resources/tySaMt2Pwf/Automation%20pie.png (Automation pie.png)

How To Evaluate Your Cognitive Automation Use Case

The value of cognitive automation is clear in terms of completing tasks faster, saving time, and reducing operational costs. Evaluating your use case is a great way to measure your progress towards that goal and use it as a checkpoint to refine your approach further. Hence, it is best to do the evaluation based on the three components mention above; AI performance, Automation integration, and Application setup.

Evaluating the AI Performance

Evaluating the AI performance is the first step in your evaluation process. Let's take document classification as an example which is useful for many tasks such as customer service ticket routing. The task here is to assign the right category or label to a document like an email. The AI model would analyze the text and assign a category to it based on the text context as illustrated in the diagram below.

https://malcolm-en-gb.s3.eu-west-1.amazonaws.com/instances/oiJS2eQZyE/resources/YaL3PJYT2V/document%20classification.png (document classification (1).png)

Evaluating the AI model requires a very pragmatic approach. This is the case for a simple reason which is that AI model performance keeps increasing with more examples just like humans. Another point that AI models have in common with humans is performing a task with decent quality by seeing a few examples but it take significantly more time and effort to build that additional 10% knowledge and become the domain expert. With DeepOpinion proprietary technology an AI model requires significantly less examples but follows a similar trend.

https://malcolm-en-gb.s3.eu-west-1.amazonaws.com/instances/oiJS2eQZyE/resources/ZIX5ZCrRUt/AI%20Performance.png (AI Performance.png)

Therefore, the pragmatic metric of evaluation is when the AI model accuracy starts becoming useful to your application. This varies from one use case to the other. For example, at what AI accuracy would you speed up your resolution time by 70% or eliminate your mis-routed tickets by 50%. Once you reach to this point you can release a model and start realizing its value to your business process. From our experience, most applications can start realizing positive business value at a 70% accuracy. Our customer success team can work closely with you to define your go-live ready accuracy based on your application and business case.

An important question here is how do I get closer to a human-level accuracy or even a domain expert. DeepOpinion Studio is a learning system and continues to improve after going live in multiple ways. Some examples can be Active Learning where an AI model would ask you to validate its learning or by getting feedback from your users correcting its prediction on your application such as a ticketing system. This way the faster you go live the sooner your AI model can learn from new examples just like having an on-job training.

https://malcolm-en-gb.s3.eu-west-1.amazonaws.com/instances/oiJS2eQZyE/resources/lEgLjt4kBm/Learning.png (Learning.png)

Evaluating the Automation Integration

This part is straight forward. The thing to test here is that the integration works and executes the right task on your application. For example, you should see the ticket priority field gets assigned with a priority once you activate your automation and receive a new ticket. This way you can confirm that the setups works properly and the task is executed. It is advisable to do such tests in a sandbox environment so that you do not interfere with your current setup until you are ready to go-live.

https://malcolm-en-gb.s3.eu-west-1.amazonaws.com/instances/oiJS2eQZyE/resources/IGmg4MO3NV/Ticket_automation_freshdesk_animated.gif (Ticket_automation_freshdesk_animated.gif)

Evaluating the Application Setup

This point is specific to workflow automation. In some cases you might be performing a task manually while in others you might have a system in place that automates some of the tasks to a certain level. In this case you would ensure that the cognitive automation plays nicely with what you have in place already. For example, you might have 2 rules in place; the first one will search for the keyword "Delayed" in the body of an email and the second one will assign the priority to "Urgent". Once you integrate cognitive automation you would replace the first rule with a ticket priority classification AI model to have a better accuracy and then feed it to the second rule. In some cases, you might have a few dozen rules and it is important to configure them tightly so that your workflow can get the best of both and enhance your productivity.

https://malcolm-en-gb.s3.eu-west-1.amazonaws.com/instances/oiJS2eQZyE/resources/fH5f2SWkhA/Before%26After.png (Before&After.png)

We Are Here For You

Cognitive Automation is a new technology and you are one of the innovators and early adopter to harness this technology. We are here to guide you on your journey for any question or idea you might have and help you refine your cognitive automation use case. We have a shared goal here, which is to successfully implement your use case and improve your productivity. All you have to do is to get in-touch with our customer success team.

Question 9

What is The Data Structure of Batch Analysis Results for Opinion Mining

Accepted Answer

What is Aspect-based Sentiment Analysis

Aspect-based sentiment analysis refers to the task of identifying fine-grained opinion towards a specific topic in text. This text can take a variety of formats such as emails, product reviews, customer requests, social media posts, and any other user generated content. This approach to text analysis is considered the industry standard for analyzing customer reviews as seen in the image below.

https://malcolm-en-gb.s3.eu-west-1.amazonaws.com/instances/oiJS2eQZyE/resources/tIC9tp5U72/Amazon_review_absa.png (Amazon_review_absa.png)

How A Customer Review is Structured

A customer review consists of two main parts; the review text and review description or metadata such as name, date, rating, etc. This universally applies to most online reviews including Amazon product reviews, Google Maps company reviews, tweets, and more.

https://malcolm-en-gb.s3.eu-west-1.amazonaws.com/instances/oiJS2eQZyE/resources/l4s3zEcKGK/Amazon_review_detailed.png (Amazon_review_detailed.png)

Reviews tend to be dense with mentions of topics, emotions, and sentiments as customers are expressing their experience in a short message. You would hence find reviews that include multiple sentences. Therefore, we unpack a review to its core elements which we refer to as segments (eg, sentences). This way we ensure high accuracy and provide you with all the essential analysis you require to know about a review.

https://malcolm-en-gb.s3.eu-west-1.amazonaws.com/instances/oiJS2eQZyE/resources/mZxanVYcir/Review_text_segments.png (Review_text_segments.png)

To get fine-grained analysis the review is broken-down to 3 levels. Level 1 is the whole review message, Level 2 is a breakdown of the segments in the review text, and Level 3 are the topics and sentiments associated with each segment which can be none, one, or more. The diagram below illustrates this hierarchy.

https://malcolm-en-gb.s3.eu-west-1.amazonaws.com/instances/oiJS2eQZyE/resources/z4o5iQ1knZ/Absa_structure.png (Absa_structure.png)

Your analysis results will also include a unique ID for each level to give it a convenient structure and make working with it easy in your desired application.

https://malcolm-en-gb.s3.eu-west-1.amazonaws.com/instances/oiJS2eQZyE/resources/O1A0Q5wNuX/Review_absa_structure.png (Review_absa_structure.png)

How Our Analysis Results are Structured

It is common to upload a file with 100 rows (1 review per row) and download results with 300 rows for example. This is because each of these reviews would have multiple sentences that can mention 1 or more topics. To make results easy to work with, we add a new row for each topic identified in each segment.

For example, if you have a review consisting of 2 sentences and each sentence has 2 topics mentioned, you will end with 4 rows for this review. The review text will appear 4 times, each segment will appear twice, and each topic is considered unique to a segment. This way you get a layered/nested results ready for your following activities.

Example of input file with 50 rows (50 reviews)

https://malcolm-en-gb.s3.eu-west-1.amazonaws.com/instances/oiJS2eQZyE/resources/8FBcj8CvX2/Screen%20Shot%202021-10-30%20at%203.07.06%20PM.png (Screen Shot 2021-10-30 at 3.07.06 PM.png)

Example of output file with 280 rows (50 reviews)

Having more rows in the results is an indicator of the richness of the data. You will also notice some column headers are highlighted in blue color. These columns are generated by DeepOpinion Studio.

https://malcolm-en-gb.s3.eu-west-1.amazonaws.com/instances/oiJS2eQZyE/resources/3d2iVLcpzl/Screen%20Shot%202021-10-30%20at%2012.27.08%20PM.png (Screen Shot 2021-10-30 at 12.27.08 PM.png)

Results Data Dictionary

Here is a description of the columns you can expect from your analysis.

id [Level 1]: A system generated unique ID for each original row (document or review)

Text [Level 1]: The original text uploaded for automated analysis (document or review)

Segment id [Level 2]: A system generated unique ID for each segment in the original row (child of id)

Segment text [Level 2]: A segment of the original text uploaded for automated analysis (child of text)

Span start [Level 2]: The character count of where a segment starts (child of text)

Span end [Level 2]: The character count of where a segment ends (child of text)

Tag id [Level 3]: A system generated unique ID for each topic identified in a text segment (child of segment id)

Aspect [Level 3]: A topic identified in a text segment (child of segment text)

Sentiment [Level 3]: A sentiment category identified in a text segment (child of segment text)

Others [Level 1]: Any other column included in the original data such as dates, names, ratings, and source (metadata)

Why This Structure

This data structure has been carefully designed to make the results convenient to work with in downstream tasks such as in process automation or data visualization. For example, you might be interested in counting how often do your customers complain about battery life and auto-create a support ticket for each time this is mentioned.

https://malcolm-en-gb.s3.eu-west-1.amazonaws.com/instances/oiJS2eQZyE/resources/IGmg4MO3NV/Ticket_automation_freshdesk_animated.gif (Ticket_automation_freshdesk_animated.gif)

Contact us to know more about moving your customer review from static insights to dynamic and automated actions

How can we help?

Most popular

Data Labelling Guideline for Aspect-based Sentiment Analysis

How do I start with DeepOpinion Studio?

How do I create a label model for customer support ticket categorization on DeepOpinion Studio

How do I create a custom model on DeepOpinion Studio?

How do I create a document information extraction model on DeepOpinion Studio

How Do I Automate My Customer Service Requests on Freshdesk

How Do I Create A Freshdesk Ticket Automation on DeepOpinion Studio?

How can I evaluate my cognitive automation

What is The Data Structure of Batch Analysis Results for Opinion Mining

Contact Us