Question 1

Data Labelling Guideline for Aspect-based Sentiment Analysis

Accepted Answer

Part: High-Quality Training Data Labelling for Opinion Mining Models

In this tutorial we present to you some tips on data labelling to ensure that your model can reach to high accuracy. We will start with the purpose of data labelling, cover 7 principles to follow, and tell you about our cool "Active Learning" feature that can help you automate part of this process.

The Purpose of Data Labelling

Humans learn best by examples. AI models learn the same way!

You can recall from your learning experience that you get to learn a concept by seeing example to experience this concept in its original context and then it is easy for you to make sense of it. The same thing applies to AI model training, you are showing the AI model examples of where a topic and a sentiment appear in text (it's original context). This way the AI model can pick up those and sentiments when appearing in similar sentences. The more examples the better the AI is learning and able to generalize what it learned.

https://malcolm-en-gb.s3.eu-west-1.amazonaws.com/instances/oiJS2eQZyE/resources/99HyRyLCDm/annotation%20picture.png (annotation picture.png)

Therefore the job of the data annotator is to help the AI model in associating sentences with concepts (topics and sentiments) as shown in this image below

https://malcolm-en-gb.s3.eu-west-1.amazonaws.com/instances/oiJS2eQZyE/resources/JFiS4b3p6C/Screen%20Shot%202021-10-14%20at%209.21.03%20PM.png (Screen Shot 2021-10-14 at 9.21.03 PM.png)

Labelling Data on DeepOpinion Studio

DeepOpinion Studio offers a user friendly interface that helps you to label data more efficiently and in an interactive way. Compared to other tools, DeepOpinion approach in creating a new model is 10X more efficient. When creating a new model, on step 3 "Training Data". Here are the data labelling steps:

Navigate to Step 3 "Training Data"
Click on "NEW LABELLING SESSION +" button on the top right
On the pop-up window choose session type to be "Annotation" and click on "CREATE LABELLING SESSION" button
A new session will be added to the page which you can click on to start labelling training data
The text box on the top right is what you have to label as shown on the below image
Read the sentence that is highlighted and select the topic and sentiment from the menu below the text
Once you finish labelling this session you will be notified and taken to the home page

Tip 1: You can label the same sentence with multiple topics and different sentiments if they appear in the text.

Tip 2: When starting, aim to label 10 sessions as a minimum and then train your first model.

https://malcolm-en-gb.s3.eu-west-1.amazonaws.com/instances/oiJS2eQZyE/resources/lV3rJqigKJ/annotation%20interface.svg (annotation interface.svg)

The 7 Principles for High-Quality Training Data

Involve domain experts in the process
Define topics to be comprehensive and non-overlapping in coverage
Write definitions for each topic and add examples
Annotate based on explicit mentions of topics with sentiments (= opinion)
Ignore implicit and subjective topics or factual statements
Focus on producing high quality labels, take breaks to refresh focus
Iterate between annotating and training to speed up the process

1. Involve domain experts

Involving domain experts is important to ensure that this expertise is reflected on the training data and hence giving the AI model a good chance to mimic the domain expert. The job title of this domain expert depends on the specific use case. For example when creating a customer review analysis model it is the Customer Experience Manager or for employee feedback it would be the HR Manager.

2. Define topics to be comprehensive and non-overlapping in coverage

Topics should be designed in a way that reflects the what is actually expressed in the text data. Doing this step well means that your list of topics mutually exclusive in meaning and collectively exhaustive in covering all the topics that appears in the text. Therefore, avoid having topics that are similar and can overlap in meaning. This illustration shows how you can cover the content of the text with topics.

https://malcolm-en-gb.s3.eu-west-1.amazonaws.com/instances/oiJS2eQZyE/resources/H6NjkhyFqr/Screen%20Shot%202021-10-14%20at%2010.09.53%20PM.png (Screen Shot 2021-10-14 at 10.09.53 PM.png)

3. Write definitions for each topic and add examples

It is a good practice to define topics and adding examples to understand how a topic appears in your text and label in a consistent way. When you label data and unsure if a topic is relevant or not you can refer to it. Additionally it helps other people in labelling the data in a consistent way too.

https://malcolm-en-gb.s3.eu-west-1.amazonaws.com/instances/oiJS2eQZyE/resources/1K2vp01qVv/Aspect%20Definition.svg (Aspect Definition.svg)

4. Annotate based on explicit mentions of topics with sentiments (= opinion)

Make sure that when labelling you only select a topic that is explicitly mentioned in the text. This is because AI models can only process and learn from information that is really present in the text.

https://malcolm-en-gb.s3.eu-west-1.amazonaws.com/instances/oiJS2eQZyE/resources/JFiS4b3p6C/Screen%20Shot%202021-10-14%20at%209.21.03%20PM.png (Screen Shot 2021-10-14 at 9.21.03 PM.png)

5. Ignore implicit and subjective topics or factual statements

The AI model can only learn from the text it sees and has no access to external knowledge. Therefore, avoid labelling with topics that do not appear in the text to avoid confusing the model. For example, if a customer review says "why didn't I see this product before" you shouldn't label "Marketing" with "Negative" sentiment as marketing is not referred to in the text.

https://malcolm-en-gb.s3.eu-west-1.amazonaws.com/instances/oiJS2eQZyE/resources/lhDKGzCDl6/Screen%20Shot%202021-10-14%20at%2010.33.30%20PM.png (Screen Shot 2021-10-14 at 10.33.30 PM.png)

6. Focus on producing high quality labels, take breaks to refresh focus

This point is self explanatory, taking breaks helps you to stay focused to produce consistent labels that the AI model can mimic.

7. Iterate between annotating and training to speed up the process

Iterate on this process by annotating ~100 examples at a time until you reach the desired performance. As a rule of thumb, aim to label at least 30 examples for each topic so that the AI model can see enough variety of example where such a topic appears. This way the AI model can reach a high accuracy. the data labelling and model training workflow is optimized to help you run through this process efficiently. You can expect to complete training a model from scratch in a matter of few hours. This is how the process looks like.

https://malcolm-en-gb.s3.eu-west-1.amazonaws.com/instances/oiJS2eQZyE/resources/dRKsa3NnnM/Workflow_Graphic_AutoNLU-2525951.png (Workflow_Graphic_AutoNLU-2525951.png)

Active learning is here to automate data labelling

Our goal is to minimize the manual effort in creating a model. Therefore, we released a new feature that helps in automating the data labelling step. In this case, once you label some data and train a model with at least 50% you can switch to "Active Learning". This is how it works:

The model go through all the unlabelled text and find the top examples that it is least confident about
It will predict the labels for topic and sentiment
You will then get to confirm the predicted labels by either ticking the checkbox of correcting the labels if the model got it wrong

This way you provide the model with the most impactful examples to improve its accuracy. Additionally this way you reduce your labelled effort by 5X compared to other conventional ways.

https://malcolm-en-gb.s3.eu-west-1.amazonaws.com/instances/oiJS2eQZyE/resources/lt1UJD4w5u/active%20learning.svg (active learning.svg)

As always, please remember to share your feedback and any questions you have.

Question 2

How do I create a custom model on DeepOpinion Studio?

Accepted Answer

Create your own custom text analysis model easily on DeepOpinion Studio in a few hours. or even minutes if you already have training data! This tutorial will follow the steps shown in the diagram below.

https://malcolm-en-gb.s3.eu-west-1.amazonaws.com/instances/oiJS2eQZyE/resources/dRKsa3NnnM/Workflow_Graphic_AutoNLU-2525951.png (Workflow_Graphic_AutoNLU-2525951.png)

This short tutorial explains how to a custom model for analyzing hotels reviews. The same steps can be followed to customize any model. You can download this Hotel Reviews Training Data to do it yourself.

Follow this step-by-step guide to learn how:

Leave us a comment you would like more details.

Question 3

How do I create a label model for customer support ticket categorization on DeepOpinion Studio

Accepted Answer

Create your own customer support ticket categorization AI model easily on DeepOpinion Studio in minutes! This tutorial will follow the steps shown in the diagram below. As we will be using historical data from customer support we will skip the "ANNOTATE" step.

https://malcolm-en-gb.s3.eu-west-1.amazonaws.com/instances/oiJS2eQZyE/resources/dRKsa3NnnM/Workflow_Graphic_AutoNLU-2525951.png (Workflow_Graphic_AutoNLU-2525951.png)

This short tutorial explains how to train an AI model on categorizing customer support tickets from the banking industry. For example, we want to determine what the request is about (eg, lost credit card, banking app not working, etc) so that it gets directly assigned to the responsible team to handle it. The same steps can be followed to train any label model where one 1 label is only assigned to a document.

Important: download to follow the tutorial and do it yourself.

Click Here to Download The Banking Customer Support Tickets Training Data

Follow this step-by-step guide to learn how:

Leave us a comment you would like more details.

Question 4

How do I start with DeepOpinion Studio?

Accepted Answer

To get actionable insights from text data, you need to get familiar with DeepOpinion Studio.

Once you're logged in, you can see your models and all the available pre-trained models, which change to "My Models" when starring them. First, select a model you wish to use.

After choosing a model, you're in the demo menu of this particular model, where you can see a description and all pre-defined and pre-trained aspects (click on "classification categories"). You can select a demo document or insert your text example to convince yourself of the model’s capabilities in the section below.

In the next section of the menu, called "batch" you can start analyzing more extensive data sets to derive actionable insights. Simply upload your data set(s) in the "upload data" section and follow the instructions. The bell icon in the upper right corner will let you know once the analysis is completed and the results are ready to download.

Please keep in mind that the amount of data you can analyze depends on your subscribed plan (for more info, click "update" in the menu).

Question 5

How Do I Automate My Customer Service Requests on Freshdesk

Accepted Answer

Automate your Freshdesk tickets routing, tagging, and prioritization DeepOpinion Studio in 4 simple steps.

The 3 steps are:

Creat a free Freshdesk Support Account
Select your input trigger, which in this case is receiving a new ticket on freshdesk
Choose the model you want to perform the ticket categorization. You can also train a model on your freshdesk tickets historical data
Select your output, which in this case would be updating a field on the freshdesk ticket

Follow this step-by-step guide to learn how:

Leave us a comment you would like more details.

Question 6

How do I create a document information extraction model on DeepOpinion Studio

Accepted Answer

Create a document information extraction as a cognitive skill on DeepOpinion Studio then upload and label your documents to be able to train a model.

Question 7

How can I analyze text on DeepOpinion Studio?

Accepted Answer

Analyzing your own text can be easily done on DeepOpinion Studio in less than 1 minute!

This short tutorial explains how to analyze car reviews using our industry pre-trained model for "Automotives". The same steps can be followed to use any model. You can download this automative reviews sample data to do it yourself.

Follow this step-by-step guide to learn how:

Leave us a comment you would like more details.

Question 8

How can I evaluate my cognitive automation

Accepted Answer

What is Cognitive Automation

If you ever experienced a daunting task at work that you believe should be automated in the AI era, then you already know Cognitive Automation. Cognitive automation refers to the use of AI in automating repetitive tasks and processes that are alternatively performed manually be a person or even a whole team.

There are three component that come together to make this happen:

1. AI: To perform the core task that require human intelligence such as comprehending and analyzing text

2. Automation: To execute a task or a workflow based on the AI outcomes

3. Application: The system where this task or activity should be performed such as ERP, CRM, or a ticketing system

https://malcolm-en-gb.s3.eu-west-1.amazonaws.com/instances/oiJS2eQZyE/resources/tySaMt2Pwf/Automation%20pie.png (Automation pie.png)

How To Evaluate Your Cognitive Automation Use Case

The value of cognitive automation is clear in terms of completing tasks faster, saving time, and reducing operational costs. Evaluating your use case is a great way to measure your progress towards that goal and use it as a checkpoint to refine your approach further. Hence, it is best to do the evaluation based on the three components mention above; AI performance, Automation integration, and Application setup.

Evaluating the AI Performance

Evaluating the AI performance is the first step in your evaluation process. Let's take document classification as an example which is useful for many tasks such as customer service ticket routing. The task here is to assign the right category or label to a document like an email. The AI model would analyze the text and assign a category to it based on the text context as illustrated in the diagram below.

https://malcolm-en-gb.s3.eu-west-1.amazonaws.com/instances/oiJS2eQZyE/resources/YaL3PJYT2V/document%20classification.png (document classification (1).png)

Evaluating the AI model requires a very pragmatic approach. This is the case for a simple reason which is that AI model performance keeps increasing with more examples just like humans. Another point that AI models have in common with humans is performing a task with decent quality by seeing a few examples but it take significantly more time and effort to build that additional 10% knowledge and become the domain expert. With DeepOpinion proprietary technology an AI model requires significantly less examples but follows a similar trend.

https://malcolm-en-gb.s3.eu-west-1.amazonaws.com/instances/oiJS2eQZyE/resources/ZIX5ZCrRUt/AI%20Performance.png (AI Performance.png)

Therefore, the pragmatic metric of evaluation is when the AI model accuracy starts becoming useful to your application. This varies from one use case to the other. For example, at what AI accuracy would you speed up your resolution time by 70% or eliminate your mis-routed tickets by 50%. Once you reach to this point you can release a model and start realizing its value to your business process. From our experience, most applications can start realizing positive business value at a 70% accuracy. Our customer success team can work closely with you to define your go-live ready accuracy based on your application and business case.

An important question here is how do I get closer to a human-level accuracy or even a domain expert. DeepOpinion Studio is a learning system and continues to improve after going live in multiple ways. Some examples can be Active Learning where an AI model would ask you to validate its learning or by getting feedback from your users correcting its prediction on your application such as a ticketing system. This way the faster you go live the sooner your AI model can learn from new examples just like having an on-job training.

https://malcolm-en-gb.s3.eu-west-1.amazonaws.com/instances/oiJS2eQZyE/resources/lEgLjt4kBm/Learning.png (Learning.png)

Evaluating the Automation Integration

This part is straight forward. The thing to test here is that the integration works and executes the right task on your application. For example, you should see the ticket priority field gets assigned with a priority once you activate your automation and receive a new ticket. This way you can confirm that the setups works properly and the task is executed. It is advisable to do such tests in a sandbox environment so that you do not interfere with your current setup until you are ready to go-live.

https://malcolm-en-gb.s3.eu-west-1.amazonaws.com/instances/oiJS2eQZyE/resources/IGmg4MO3NV/Ticket_automation_freshdesk_animated.gif (Ticket_automation_freshdesk_animated.gif)

Evaluating the Application Setup

This point is specific to workflow automation. In some cases you might be performing a task manually while in others you might have a system in place that automates some of the tasks to a certain level. In this case you would ensure that the cognitive automation plays nicely with what you have in place already. For example, you might have 2 rules in place; the first one will search for the keyword "Delayed" in the body of an email and the second one will assign the priority to "Urgent". Once you integrate cognitive automation you would replace the first rule with a ticket priority classification AI model to have a better accuracy and then feed it to the second rule. In some cases, you might have a few dozen rules and it is important to configure them tightly so that your workflow can get the best of both and enhance your productivity.

https://malcolm-en-gb.s3.eu-west-1.amazonaws.com/instances/oiJS2eQZyE/resources/fH5f2SWkhA/Before%26After.png (Before&After.png)

We Are Here For You

Cognitive Automation is a new technology and you are one of the innovators and early adopter to harness this technology. We are here to guide you on your journey for any question or idea you might have and help you refine your cognitive automation use case. We have a shared goal here, which is to successfully implement your use case and improve your productivity. All you have to do is to get in-touch with our customer success team.

Data Labelling Guideline for Aspect-based Sentiment Analysis

How do I create a custom model on DeepOpinion Studio?

How do I create a label model for customer support ticket categorization on DeepOpinion Studio

How do I start with DeepOpinion Studio?

Contact Us

Let us know what you need!

How Do I Automate My Customer Service Requests on Freshdesk

How do I create a document information extraction model on DeepOpinion Studio

How can I analyze text on DeepOpinion Studio?

How can I evaluate my cognitive automation