Question 1

How can I analyze text on DeepOpinion Studio?

Accepted Answer

Analyzing your own text can be easily done on DeepOpinion Studio in less than 1 minute!

This short tutorial explains how to analyze car reviews using our industry pre-trained model for "Automotives". The same steps can be followed to use any model. You can download this automative reviews sample data to do it yourself.

Follow this step-by-step guide to learn how:

Leave us a comment you would like more details.

Question 2

Data Labelling Guideline for Aspect-based Sentiment Analysis

Accepted Answer

Part: High-Quality Training Data Labelling for Opinion Mining Models

In this tutorial we present to you some tips on data labelling to ensure that your model can reach to high accuracy. We will start with the purpose of data labelling, cover 7 principles to follow, and tell you about our cool "Active Learning" feature that can help you automate part of this process.

The Purpose of Data Labelling

Humans learn best by examples. AI models learn the same way!

You can recall from your learning experience that you get to learn a concept by seeing example to experience this concept in its original context and then it is easy for you to make sense of it. The same thing applies to AI model training, you are showing the AI model examples of where a topic and a sentiment appear in text (it's original context). This way the AI model can pick up those and sentiments when appearing in similar sentences. The more examples the better the AI is learning and able to generalize what it learned.

https://malcolm-en-gb.s3.eu-west-1.amazonaws.com/instances/oiJS2eQZyE/resources/99HyRyLCDm/annotation%20picture.png (annotation picture.png)

Therefore the job of the data annotator is to help the AI model in associating sentences with concepts (topics and sentiments) as shown in this image below

https://malcolm-en-gb.s3.eu-west-1.amazonaws.com/instances/oiJS2eQZyE/resources/JFiS4b3p6C/Screen%20Shot%202021-10-14%20at%209.21.03%20PM.png (Screen Shot 2021-10-14 at 9.21.03 PM.png)

Labelling Data on DeepOpinion Studio

DeepOpinion Studio offers a user friendly interface that helps you to label data more efficiently and in an interactive way. Compared to other tools, DeepOpinion approach in creating a new model is 10X more efficient. When creating a new model, on step 3 "Training Data". Here are the data labelling steps:

Navigate to Step 3 "Training Data"
Click on "NEW LABELLING SESSION +" button on the top right
On the pop-up window choose session type to be "Annotation" and click on "CREATE LABELLING SESSION" button
A new session will be added to the page which you can click on to start labelling training data
The text box on the top right is what you have to label as shown on the below image
Read the sentence that is highlighted and select the topic and sentiment from the menu below the text
Once you finish labelling this session you will be notified and taken to the home page

Tip 1: You can label the same sentence with multiple topics and different sentiments if they appear in the text.

Tip 2: When starting, aim to label 10 sessions as a minimum and then train your first model.

https://malcolm-en-gb.s3.eu-west-1.amazonaws.com/instances/oiJS2eQZyE/resources/lV3rJqigKJ/annotation%20interface.svg (annotation interface.svg)

The 7 Principles for High-Quality Training Data

Involve domain experts in the process
Define topics to be comprehensive and non-overlapping in coverage
Write definitions for each topic and add examples
Annotate based on explicit mentions of topics with sentiments (= opinion)
Ignore implicit and subjective topics or factual statements
Focus on producing high quality labels, take breaks to refresh focus
Iterate between annotating and training to speed up the process

1. Involve domain experts

Involving domain experts is important to ensure that this expertise is reflected on the training data and hence giving the AI model a good chance to mimic the domain expert. The job title of this domain expert depends on the specific use case. For example when creating a customer review analysis model it is the Customer Experience Manager or for employee feedback it would be the HR Manager.

2. Define topics to be comprehensive and non-overlapping in coverage

Topics should be designed in a way that reflects the what is actually expressed in the text data. Doing this step well means that your list of topics mutually exclusive in meaning and collectively exhaustive in covering all the topics that appears in the text. Therefore, avoid having topics that are similar and can overlap in meaning. This illustration shows how you can cover the content of the text with topics.

https://malcolm-en-gb.s3.eu-west-1.amazonaws.com/instances/oiJS2eQZyE/resources/H6NjkhyFqr/Screen%20Shot%202021-10-14%20at%2010.09.53%20PM.png (Screen Shot 2021-10-14 at 10.09.53 PM.png)

3. Write definitions for each topic and add examples

It is a good practice to define topics and adding examples to understand how a topic appears in your text and label in a consistent way. When you label data and unsure if a topic is relevant or not you can refer to it. Additionally it helps other people in labelling the data in a consistent way too.

https://malcolm-en-gb.s3.eu-west-1.amazonaws.com/instances/oiJS2eQZyE/resources/1K2vp01qVv/Aspect%20Definition.svg (Aspect Definition.svg)

4. Annotate based on explicit mentions of topics with sentiments (= opinion)

Make sure that when labelling you only select a topic that is explicitly mentioned in the text. This is because AI models can only process and learn from information that is really present in the text.

https://malcolm-en-gb.s3.eu-west-1.amazonaws.com/instances/oiJS2eQZyE/resources/JFiS4b3p6C/Screen%20Shot%202021-10-14%20at%209.21.03%20PM.png (Screen Shot 2021-10-14 at 9.21.03 PM.png)

5. Ignore implicit and subjective topics or factual statements

The AI model can only learn from the text it sees and has no access to external knowledge. Therefore, avoid labelling with topics that do not appear in the text to avoid confusing the model. For example, if a customer review says "why didn't I see this product before" you shouldn't label "Marketing" with "Negative" sentiment as marketing is not referred to in the text.

https://malcolm-en-gb.s3.eu-west-1.amazonaws.com/instances/oiJS2eQZyE/resources/lhDKGzCDl6/Screen%20Shot%202021-10-14%20at%2010.33.30%20PM.png (Screen Shot 2021-10-14 at 10.33.30 PM.png)

6. Focus on producing high quality labels, take breaks to refresh focus

This point is self explanatory, taking breaks helps you to stay focused to produce consistent labels that the AI model can mimic.

7. Iterate between annotating and training to speed up the process

Iterate on this process by annotating ~100 examples at a time until you reach the desired performance. As a rule of thumb, aim to label at least 30 examples for each topic so that the AI model can see enough variety of example where such a topic appears. This way the AI model can reach a high accuracy. the data labelling and model training workflow is optimized to help you run through this process efficiently. You can expect to complete training a model from scratch in a matter of few hours. This is how the process looks like.

https://malcolm-en-gb.s3.eu-west-1.amazonaws.com/instances/oiJS2eQZyE/resources/dRKsa3NnnM/Workflow_Graphic_AutoNLU-2525951.png (Workflow_Graphic_AutoNLU-2525951.png)

Active learning is here to automate data labelling

Our goal is to minimize the manual effort in creating a model. Therefore, we released a new feature that helps in automating the data labelling step. In this case, once you label some data and train a model with at least 50% you can switch to "Active Learning". This is how it works:

The model go through all the unlabelled text and find the top examples that it is least confident about
It will predict the labels for topic and sentiment
You will then get to confirm the predicted labels by either ticking the checkbox of correcting the labels if the model got it wrong

This way you provide the model with the most impactful examples to improve its accuracy. Additionally this way you reduce your labelled effort by 5X compared to other conventional ways.

https://malcolm-en-gb.s3.eu-west-1.amazonaws.com/instances/oiJS2eQZyE/resources/lt1UJD4w5u/active%20learning.svg (active learning.svg)

As always, please remember to share your feedback and any questions you have.

Question 3

How can I evaluate my cognitive automation

Accepted Answer

What is Cognitive Automation

If you ever experienced a daunting task at work that you believe should be automated in the AI era, then you already know Cognitive Automation. Cognitive automation refers to the use of AI in automating repetitive tasks and processes that are alternatively performed manually be a person or even a whole team.

There are three component that come together to make this happen:

1. AI: To perform the core task that require human intelligence such as comprehending and analyzing text

2. Automation: To execute a task or a workflow based on the AI outcomes

3. Application: The system where this task or activity should be performed such as ERP, CRM, or a ticketing system

https://malcolm-en-gb.s3.eu-west-1.amazonaws.com/instances/oiJS2eQZyE/resources/tySaMt2Pwf/Automation%20pie.png (Automation pie.png)

How To Evaluate Your Cognitive Automation Use Case

The value of cognitive automation is clear in terms of completing tasks faster, saving time, and reducing operational costs. Evaluating your use case is a great way to measure your progress towards that goal and use it as a checkpoint to refine your approach further. Hence, it is best to do the evaluation based on the three components mention above; AI performance, Automation integration, and Application setup.

Evaluating the AI Performance

Evaluating the AI performance is the first step in your evaluation process. Let's take document classification as an example which is useful for many tasks such as customer service ticket routing. The task here is to assign the right category or label to a document like an email. The AI model would analyze the text and assign a category to it based on the text context as illustrated in the diagram below.

https://malcolm-en-gb.s3.eu-west-1.amazonaws.com/instances/oiJS2eQZyE/resources/YaL3PJYT2V/document%20classification.png (document classification (1).png)

Evaluating the AI model requires a very pragmatic approach. This is the case for a simple reason which is that AI model performance keeps increasing with more examples just like humans. Another point that AI models have in common with humans is performing a task with decent quality by seeing a few examples but it take significantly more time and effort to build that additional 10% knowledge and become the domain expert. With DeepOpinion proprietary technology an AI model requires significantly less examples but follows a similar trend.

https://malcolm-en-gb.s3.eu-west-1.amazonaws.com/instances/oiJS2eQZyE/resources/ZIX5ZCrRUt/AI%20Performance.png (AI Performance.png)

Therefore, the pragmatic metric of evaluation is when the AI model accuracy starts becoming useful to your application. This varies from one use case to the other. For example, at what AI accuracy would you speed up your resolution time by 70% or eliminate your mis-routed tickets by 50%. Once you reach to this point you can release a model and start realizing its value to your business process. From our experience, most applications can start realizing positive business value at a 70% accuracy. Our customer success team can work closely with you to define your go-live ready accuracy based on your application and business case.

An important question here is how do I get closer to a human-level accuracy or even a domain expert. DeepOpinion Studio is a learning system and continues to improve after going live in multiple ways. Some examples can be Active Learning where an AI model would ask you to validate its learning or by getting feedback from your users correcting its prediction on your application such as a ticketing system. This way the faster you go live the sooner your AI model can learn from new examples just like having an on-job training.

https://malcolm-en-gb.s3.eu-west-1.amazonaws.com/instances/oiJS2eQZyE/resources/lEgLjt4kBm/Learning.png (Learning.png)

Evaluating the Automation Integration

This part is straight forward. The thing to test here is that the integration works and executes the right task on your application. For example, you should see the ticket priority field gets assigned with a priority once you activate your automation and receive a new ticket. This way you can confirm that the setups works properly and the task is executed. It is advisable to do such tests in a sandbox environment so that you do not interfere with your current setup until you are ready to go-live.

https://malcolm-en-gb.s3.eu-west-1.amazonaws.com/instances/oiJS2eQZyE/resources/IGmg4MO3NV/Ticket_automation_freshdesk_animated.gif (Ticket_automation_freshdesk_animated.gif)

Evaluating the Application Setup

This point is specific to workflow automation. In some cases you might be performing a task manually while in others you might have a system in place that automates some of the tasks to a certain level. In this case you would ensure that the cognitive automation plays nicely with what you have in place already. For example, you might have 2 rules in place; the first one will search for the keyword "Delayed" in the body of an email and the second one will assign the priority to "Urgent". Once you integrate cognitive automation you would replace the first rule with a ticket priority classification AI model to have a better accuracy and then feed it to the second rule. In some cases, you might have a few dozen rules and it is important to configure them tightly so that your workflow can get the best of both and enhance your productivity.

https://malcolm-en-gb.s3.eu-west-1.amazonaws.com/instances/oiJS2eQZyE/resources/fH5f2SWkhA/Before%26After.png (Before&After.png)

We Are Here For You

Cognitive Automation is a new technology and you are one of the innovators and early adopter to harness this technology. We are here to guide you on your journey for any question or idea you might have and help you refine your cognitive automation use case. We have a shared goal here, which is to successfully implement your use case and improve your productivity. All you have to do is to get in-touch with our customer success team.

Question 4

Which are the different types of models that I can build in the Studio?

Accepted Answer

Summary

You can create your Class models, Label models & ABSA models on Studio to execute different Natural Language Processing tasks such as the Aspect Based Sentiment Analysis task. Find here the analytical descriptions of these models with exemplary Natural Language Processing tasks.

Model types

Label Model & Sentiment Analysis

Imagine that you have many fruits in one big basket and you want to learn more about them. The first thing that comes naturally to you is to separate them based on some characteristics. Let’s say that we pick their color as a criterion for separation. You can split the fruits in your basket into three groups: green, red, and yellow. The process you just conducted is called classification, and it is what a Label Model does: it separates objects into categories based on the objects' characteristics. Each object can belong to only one group as a fruit cannot be green and red simultaneously.

A Label Model is forced to choose precisely one class name, even when it is not clear in which class it should be assigned to, e.g., an orange fruit would be tagged as “Red” in the example of the fruits above.

https://malcolm-en-gb.s3.eu-west-1.amazonaws.com/instances/oiJS2eQZyE/resources/E3B9c750Ek/Screenshot%20from%202021-05-18%2018-23-24.png (Screenshot from 2021-05-18 18-23-24.png)

In Natural Language Processing, you can identify the above task in Sentiment Analysis, as the process where a Label Model classifies text into the following three sentiments: positive, negative, or neutral, based on the context of the words in the text. For example, a Label Model used for Sentiment Analysis would assign the label “Positive” to the text: “I am pleased with this product.”

Class Model - Aspect Analysis

Imagine that you have to analyze a fruit to its nutrient components, and you are particularly interested in identifying if your fruit has sugars, vitamin A, and protein. A Class Model is a model that explicitly knows a collection of labels, and it tries to identify these labels on a particular object.

The output of a Class Model can be none, one or more labels. For example, lime does not contain any sugars, vitamin A, or protein, so the corresponding Class Model would not assign any labels to a lime. On the other hand, an apple contains all three nutrients; hence the same Class Model would give the labels “sugars,” “vitamin A,” and “protein” to it.

A Class Model assigns a label to an object when a label's definition matches the object explicitly. If the object’s characteristics do not match any known description to the model labels, then no label is assigned to the object.

https://malcolm-en-gb.s3.eu-west-1.amazonaws.com/instances/oiJS2eQZyE/resources/rIfbj05G5c/Screenshot%20from%202021-05-18%2018-23-43.png (Screenshot from 2021-05-18 18-23-43.png)

In Natural Language Processing, you can identify the above model in the Aspect Analysis task. Here, we use a Class Model to assign none, one or more of the predefined labels (aspects) to a piece of text.

For example, a Class Model that is used for Aspect Analysis and "knows" the following aspects (labels):

Aspect Based Sentiment Analysis Model - ClassLabel Model

After understanding the two different models above (Label & Class), it will be straightforward for you to grasp their “combination,” the ClassLabel Model. So, a ClassLabel Model does nothing more than using a Class Model firstly on a piece of text and secondly a Label Model on the first model’s output.

More explicitly, when a ClassLabel "sees" a piece of text, it first tries to identify any predefined labels. We saw before that none, one or more labels can be assigned to the text at this stage. If no tags are assigned to the text, then a ClassLabel Model stops, and nothing has been set to the text example. In the last case, the Model's final output is the label None.

If at least one label is assigned to the provided text, then the Label Model is activated, and each label produces a class name. Each label is paired with the corresponding class, and the final output of the ClassLabel Model is one or more (label, class) pairs, depending on the number of the identified labels.

https://malcolm-en-gb.s3.eu-west-1.amazonaws.com/instances/oiJS2eQZyE/resources/fwT7UXRyFT/Screenshot%20from%202021-05-18%2018-23-57.png (Screenshot from 2021-05-18 18-23-57.png)

In Natural Language Processing, you can identify the above model type in the Aspect Based Sentiment Analysis task. We currently support in Studio only the ABSA task regarding the ClassLabel Model type.

Let’s see three text examples and the corresponding output of an ABSA model with the following predefined labels:

and the following sentiment classes:

Example 1: Input: “I currently live in Austria.” ---> Output pairs: None
Example 2: Input: “The customer support was amazing!” ---> Output pairs: (“Service”, “Positive”)
Example 3: Input: “The product was very easy to use, even though it was expensive” ---> Output pairs: (“Ease of use”, “Positive”), (“Money”, “Negative”)

Question 5

How Do I Automate My Customer Service Requests on Freshdesk

Accepted Answer

Automate your Freshdesk tickets routing, tagging, and prioritization DeepOpinion Studio in 4 simple steps.

The 3 steps are:

Creat a free Freshdesk Support Account
Select your input trigger, which in this case is receiving a new ticket on freshdesk
Choose the model you want to perform the ticket categorization. You can also train a model on your freshdesk tickets historical data
Select your output, which in this case would be updating a field on the freshdesk ticket

Follow this step-by-step guide to learn how:

Leave us a comment you would like more details.

Question 6

How do I create a label model for customer support ticket categorization on DeepOpinion Studio

Accepted Answer

Create your own customer support ticket categorization AI model easily on DeepOpinion Studio in minutes! This tutorial will follow the steps shown in the diagram below. As we will be using historical data from customer support we will skip the "ANNOTATE" step.

https://malcolm-en-gb.s3.eu-west-1.amazonaws.com/instances/oiJS2eQZyE/resources/dRKsa3NnnM/Workflow_Graphic_AutoNLU-2525951.png (Workflow_Graphic_AutoNLU-2525951.png)

This short tutorial explains how to train an AI model on categorizing customer support tickets from the banking industry. For example, we want to determine what the request is about (eg, lost credit card, banking app not working, etc) so that it gets directly assigned to the responsible team to handle it. The same steps can be followed to train any label model where one 1 label is only assigned to a document.

Important: download to follow the tutorial and do it yourself.

Click Here to Download The Banking Customer Support Tickets Training Data

Follow this step-by-step guide to learn how:

Leave us a comment you would like more details.

Question 7

How to define aspects efficiently.

Accepted Answer

Summary

Defining the model’s aspects efficiently is essential for annotating data correctly and eventually building a state-of-the-art model. We can create the aspects for you. You might have them created or build them together. In any case, we share here seven tips for making the most pertinent aspects for your custom models!

Tip 1: Prominence

Try to capture topics that "stick out" and be more specific wherever this is possible. For example, in the software industry, we could use different aspects such as “app”, “desktop app” or “mobile app”, depending on how detailed the data is. On the other hand, if the information generally only mentions an app, we can’t be more specific, and we should call the aspect something like “app general”.

Tip 2: Overlapping

Be aware of overlapping meaning to avoid creating aspects corresponding to similar content. For example, the aspects “support” and “customer center” would be overlapping factors for customer data, as they would be used on the same text segments during labeling.

Tip 3: Removing an aspect

Make sure that a label is relevant to your data, and you use it during labeling before you add it to your aspects list. After spending some time on data annotation, you can permanently remove it from your aspects list if you realize that you never use an aspect. For example, if I have customer data and define the aspect “mobile app” then notice that the particular aspect is not used during labeling, I should remove it.

Tip 4: Adding an aspect

While annotating data, you might feel like an aspect is missing from the aspects list as you would like to use it during labeling. In this case, you can always add the needed aspect to your aspects list. Just make sure that you need this aspect and that there is more than one example where it appears.

Tip 5: Difficulty

Try to define an aspect with a simple word and not use complicated language or abstract concepts. For example, if you have to define aspects for a financial model, it is better to determine “expenses” than “deduction”.

Tip 6: Subjective opinion space

Keep in mind that we try to cover as much personal opinion space as we can. Meaning that people talk in a specific way, using particular words, and are interested in specific aspects of a product depending on the product’s industry. For example, people usually talk about the “engine”, the “wheels/tires”, etc. Our goal is to capture these aspects, which are essential for a product, to process as many text/reviews as possible. By capturing these aspects, we understand how the market “talks” about a product.

Tip 7: Annotators' cognitive effort

Lastly, since the data will be labeled manually by annotators, we should consider their cognitive effort. Creating many aspects that are difficult to understand will significantly increase the annotators’ work and lead to a more flawed result. For this reason, we should consider whether we would be happy with the existing aspects list as an annotator before we finalize our aspects list.

Question 8

How can I understand better my custom model?

Accepted Answer

Model Explainability

AI Models are powerful tools as they can save us time from manual work, and they often predict labels more efficiently than a human, as they can process a large amount of data automatically. On the other hand, model explainability is one of the biggest problems in data science, but do not worry as we have a plan!

We present below several steps to follow when you want to investigate a model’s quality. These are the exact steps that our data science team would follow to examine your model.

Step 1 - Understand your goal
Step 2 - Understand the training dataset
Step 3 - Understand one aspect’s distribution
Step 4 - Understand the distribution of all the aspects
Step 5 - Model training & sample size
Step 6 - When it feels wrong
Step 7 - Evaluation scenarios & none labels

Step 1 - Understand your goal

Today, decision-making in business is based on data. The challenge there is to find good quality data to make the best decisions. Till recently, we could process massively only structured data, i.e., numbers, but what about all the hidden gems in the unstructured data?

Imagine that you have a hotel and you want to know more about your guests’ experience. How could you ever achieve that efficiently without considering their emails, reviews, and voice under consideration? How can you understand if they feel positive or negative about your business? On top of that, wouldn’t you like to know which aspects of your business are doing well and which ones you should improve on?

I remember a Hotel manager who could see low ratings of her hotel on TripAdvisor, but she could not comprehend the issue of those low ratings. She could read some of the thousand reviews online. Still, she knew nothing about the overall picture of the hotel as she could not transform the unstructured data (customer reviews) into structured ones. If she could manage to achieve that, she would be able to efficiently monitor the hotel’s performance on different aspects such as food service, food quality, hotel’s location, room cleanliness, etc. Imagine if she had a model on which she could feed all of the written reviews and receive as an outcome the total number of positive or negative reviews per aspect. This would help her gain a high-level view of her business and identify the particularly nasty aspects. An AI model that deals with text could help her prioritize the hotel’s needs and act more efficiently as a hotel manager.

We assume that you already have a few thousands of unstructured data examples, e.g., product reviews, consumer complaints, emails, or any other text you would like to extract insights from. This is a challenging and time-consuming process, as you already know because tasks like manual labeling need a lot of sources to be executed.

So, you decided to build a model that will automatically transform this data into structured ones (labeled data) that you can measure and use to make data-driven decisions, right?

In the process of building a custom model, you had to collect your data, label a few examples, and then use DeepOpinion’s Text Intelligence Platform to train your custom model. So, here our goal is to understand the best model for YOUR business’s current needs to help you automate your processes, add new features to them, and better approach your users.

The power of these models is that they are tireless, fast, accurate, and easy to use or update when you want to. Their only weakness is that they cannot guess how you would like them to label your data, so we highly suggest you have a clear idea of what you would like to achieve and how you should teach your model.

Step 2 - Understand the training dataset

Let’s continue with something that you are already familiar with, or at least you took a taste of it during annotation: your labeled data or, in other words, what the model knows as training data.

During manual labeling, and probably without even noticing it, you also better understand your data variation. What is the data variation, and how does it affect your model’s performance?

I would like you to keep in mind one specific scenario, feature, aspect, label, or whatever you want to predict with your model until the end of this section. Do not focus on how many of them you want your model to recognize in total, e.g., 23 unique aspects, as we will talk about this type of variety later. Let’s focus for now on one specific case.

Imagine that hotel visitors wrote the following reviews.

Example - Hotel reviews

The service in this restaurant is terrible.
The waiter treated me like I am a dog.
I never go back as the way they offer food to people is hideous.
The service was fantastic.
Service here felt like Christmas.
Do you expect to have customers with your current service?

https://malcolm-en-gb.s3.eu-west-1.amazonaws.com/instances/oiJS2eQZyE/resources/FYoONI0rWU/Screenshot%20from%202021-05-18%2018-02-44.png (Screenshot from 2021-05-18 18-02-44.png)

This hotel faces a problem because it has visitors coming from different places globally, and they all have diverse educational backgrounds. This leads to having reviews expressed with a different vocabulary, other words, and different tones.

There are many ways to say that something is excellent or nasty as we do not all have the same vocabulary or manner of expressing things. Especially when humor, sarcasm, or proverbs are used, it is sometimes difficult even for humans to understand a text (this is why we prefer to call someone to clarify things). Above, you see how our AI Model which is trained on hotel data, was able to correctly label the sentiment to all cases except for the last one where it produced no label (we will come back to this later).

We talk about homogeneity when there are only a few ways of saying something, and we do not have many variations of the same thing. While working with language models, it is crucial to understand that if a specific word appears with many different synonyms in our training data, the sample is not homogenous for this word. The sample’s inhomogeneity results in making it more difficult for the model to learn this word in the first place. After adding more labeled data (more synonyms), we would have a more potent model to recognize all the different terms used for a specific concept.

Another example is to imagine what would happen to our data if we were translating it to another language. Imagine that you are a tourist and want to write a review about the hotel you just visited. Your mother tongue is Chinese, and you do not speak English. To make sure that the hotel manager will understand your critique so she will take action on the matter that left you disappointed, you use Google Translate to translate your review into English and then post it online.

Passing text through a translator could lead to simplification of the language. Hence training a model on native data (heterogeneous dataset with a flatter data distribution) would lead to a model that can generalize with a minor error than a model trained on translated data (homogeneous dataset with a steeper bell curve). The last option might be faster if we do not have native data but have similar data in another language, leading to a more limited version of the actual data model.

Step 3 - Understand one aspect’s distribution

The crowd’s wisdom is the collective opinion of a group of individuals rather than a single expert!

In our hotel example, the above concept could be translated as: the majority of the words used in the customers’ reviews to describe an aspect, e.g., cleanliness, will be commonly used by the future reviewers to describe the same aspect.

In other words, if some words/phrases are frequently used to describe cleanliness in general, we expect to see these words appear more frequently in our data sample. On the other hand, if a specific word/phrase is not present (or it is present but in a minimal frequency) in the cleanliness examples, then it is probably true that this word/phrase is not used in general for describing the aspect of cleanliness.

The range and the frequency of the different words used for describing one aspect describe an aspect’s distribution.

Comparing the two examples below, we can see that we have a wider variety of words used for cleanliness in the second example. In both cases, the words beautiful and costly are not saying anything about the aspect of cleanliness. Since we did not teach our model through annotation that these two words talk about cleanliness, in an example with only these (or other similar) words, the aspect of cleanliness would not have been detected.

Example 1: Variety & frequency of words that describe Cleanliness in a Hotel Model

Example 2: Variety & frequency of words that describe Cleanliness in a Hotel Model

The number of the different scenarios and their frequency in a dataset describes the dataset’s variety or, in other words, its distribution. The normal distribution is depicted as a bell curve, which is steeper (minor standard deviation) when the examples are not spread apart (less diversity in the ways that someone can describe a specific scenario, aspect, etc.).

From our experience on opinion mining, having approximately 40 examples for a particular scenario, aspect, case, label (depending on your model) is usually enough to train a finetuned language model. For a not finetuned language model, we would need more: approximately 70 examples.

Step 4 - Understand the distribution of all the aspects

In this section, we will not focus on a specific scenario, class, aspect, label, etc., as we did before, but we will keep an eye on the high-level view of our use case. For example, in the hotel’s case, we would like to know how many different aspects of the hotel we should examine. What are the visitors talking about? Knowing which aspects are essential, the hotel manager can decide which aspects should be included in her AI model.

Now, we understand why it is essential to know: how many classes and which ones we want our model to be able to predict. This number will affect the size of the training dataset.

We have a finetuned on the Hotel industry English model (which means that it understands English words in the Hotels domain), and we want to train it on the ABSA task (detect aspects & sentiments). If we want the model to know 6 aspects and 3 sentiment labels, we will need approximately 6340 = 720 annotations for training based on what we have said so far.

How do we decide on the aspects when we have unlabeled data?

How do we decide on the aspects when we have labeled data?

Example 3: Imbalanced dataset

The dataset that is depicted in the above bar plot is imbalanced because the class functionality has significantly fewer examples than the other three classes. If we sample from it before balancing it, then it is highly probable that we will now “catch” many functionality examples in our sample.

Step 5 - Model training & sample size

After annotating data, we learned more about the samples’ distribution and their homogeneity. Now, we understand part of the results and the performance metrics, but these are still not enough to explain:

Why, even though I provided some labeled data for a particular aspect, the model still does not know this aspect and returns a None label?
How many annotations are considered as enough for teaching something to the model?
Why does it seem like my model can’t learn from the training data?

After answering these last questions, you will have a 360 degrees understanding of your training data and how your model learns from it!

An essential thing that will help you understand what and why should someone take under consideration to address all the above three questions is to keep in mind that the annotated data is split into training and testing data, i.e., data for teaching the model and data for testing if the model learned what it was supposed to know. The annotated data is randomly split into training and testing data. We usually keep the majority for training, e.g., 80% of all annotated data is used for training, while less for testing, e.g., 20%.

Now imagine that only one visitor wrote one review regarding the food in the hotel, and the visitor was happy about it. Hence his review was positive. He wrote the following review:

“The food was tasty, cheap & healthy in the Hotel’s restaurant.”

If we add this example to our annotations with the labels (“Food”,”Pos”) and we retrain our model, then we will have the following cases:

After the random split, this sentence falls in the training set; hence the model learns about this case, but since this sentence is not present in the testing set, we will not see any increase in the model’s performance.
After the random split, this sentence falls in the testing set; hence the model does not learn about this sentence. When later, we test it on this sentence, it will probably return a None label (no prediction for the aspect or the sentiment) as it had never seen this case before (the case was not present in the training set).

Someone can now better understand why the model doesn’t necessarily learn about it even by providing an example. To enhance learning, we offer more than one example per case. The last step will assure that we have better chances of including the case example in the training set after splitting, and our model will be able to learn from it.

Step 6 - When it feels wrong

We would like to explain here one tool that can help us better investigate a “strange” situation and be persuaded that our model is robust and trust it. Sometimes what feels normal and logical is not the same as what our model predicts but this does not mean that the model is wrong. This only means that we either did not provide enough information to the model to “see” things the way we see them or that sometimes we tend to generalize and focus on a specific tree rather than seeing the whole forest. Our model achieves keeping holistic and closer to reality view by using statistics. To analyze a bit further the last two bits, I will refer to 2 different business cases.

A few months back, a customer came to us by sharing their concerns that their model was not “smart” enough as it kept predicting some reviews as negative ones even though they had a 5-star rating. In our customer mind, the assumption was that a 5-star review must be positive, while the model that did not even know if 5-star was a high score or not (how could it after all since no one explained it to the model) kept doing what it knew: judge a review from its content. It turned out that 5-star reviews can have negative comments included (additionally to positive ones); hence the model was working correctly. Additionally, we learned that even if the metadata paints a specific picture like a 5-star rating, the model only learns from the text input (e.g., reviews) included in the training data and might give different output than the metadata.

Another customer felt that even though their two different models had the same recall score (a performance metric that affects the number of None labels), batch analysis results from model A had many more None labels than model B. The exciting word to focus on here was the word “felt”. We trust the statistics, e.g., the ratio of the total number of None labels to the total number of all predicted labels, because as human beings, we tend to hook on a feeling and miss the objective truth.

Let’s see what happened with the above case. The two models had the same recall score, but this means nothing about the training data of the two models (if they were different, then the models knew different things) or the datasets that were used for batch analysis. Now, let’s investigate all the different cases here based on what we learned so far:

i) the customer compared the performance of model A & B on different batch data (size and content)

If the dataset that was batch analyzed by model B was more extensive, e.g., 10 times bigger, then it is expected to have more None labels, 10 times more respectively, so it could create a feeling of having a higher recall.
If models A & B were trained on different training sets, they knew different things. If the batch dataset analyzed by model A included many “unknown” cases, then model A would return many None labels. If the dataset that was batch analyzed by model B included many “known” for model B cases, then model B would not return many None labels. In this case, the customer had correctly felt that this is the case, and it would mean that model A had to make predictions on a different data distribution than the one that it was trained on, while model B had a better performance (smaller recall score) as the training set and the batch set had similar distributions.

ii) the customer compared the performance of model A & B on the same batch data (size and content)

If models A & B were not trained on the same training set, it is customary to expect to know different things and react differently on different datasets.
If models A & B were trained on the same training set, we should look at the order of the batch examples, e.g., all the None labels were present in the first rows of model A. This could lead us to feel that there are more None labels for model A. Here we would have the same recall score on the two different batch datasets.

Step 7 - Evaluation scenarios & none labels

In the last case, we tasted what can happen if we train our model on a different area and then use it on a different one. To provide more insights on this issue, I will introduce here the transfer learning concept, which is the usage of the gained from one thing knowledge on learning something else, though a simple example.

Let’s assume that we have 5.000 labeled reviews for restaurants and 5.000 labeled reviews for laptops, and we want to build a model. We understand that if we use the labeled data coming from the restaurant reviews, we will construct a specialized model, or in other words, finetuned, on restaurant reviews. If we were using only laptop reviews, then we specialized in reading the laptop reviews model. But what would happen if we were using all of our data, and how well would a laptop model react to restaurant reviews and the opposite?

When someone trains a model on a domain-specific dataset, e.g., restaurant reviews, then uses it to do batch analysis (predict labels) on the same domain. We call this process In Domain Training.

Now assume that you train a model on a domain-specific dataset, e.g., restaurant reviews, and evaluate it (check its performance) on datasets from a different domain, e.g., laptop reviews. This process is called Cross-Domain Training.

In the first case, the model’s performance is expected to be higher as the model is evaluated on the same domain that it has been trained. If someone wants to create a model that performs well in different fields, then Cross-Domain Training would make more sense.

Let’s see now what will happen if we use datasets from different domains for training. The process of training a model on a downstream task such as the ABSA task on a training dataset created by the join of datasets coming from different domains, e.g., laptop and restaurant reviews, and evaluating it on other datasets independently is called Joint-Domain Training.

In the last case, someone might expect that a Joint-Domain Training model is “wiser” than an In-Domain Trained model because it has “seen” more. This is not always true, as it depends on how close the different domains are. For example, if we train a model on data coming from a mobile bank app and a food mobile and then evaluate it on data coming from the first domain (mobile bank app), we would expect a better performance than training a model on bank mobile app data and on hotel data.

We use a pre-trained language model as a base because it has been trained on knowledge-based corpora like Wikipedia, which means that the model already knows things about the language syntax and grammar and understands the context of some words based on how they are used in this corpora. When we finetune a model on an opinions’ corpora, e.g., reviews, coming from a specific domain, we show the model how to interpret words based on the special meanings within the particular discipline.

For example, the word cheap can have a positive or a negative meaning.

To sum up sometimes less is more! If we want our model to perform well on a specific domain, we should train it and use it on data from that domain (In-Domain Training). If we use data sources from fields that are not similar and some of their words have opposite meanings, then we will rather “confuse” our model than improve it. To understand why, imagine that if the model has seen examples only from the first case, then it would be confident to predict the Sentiment Label as Positive. On the other hand, if the model has seen examples from both domains (hotel, banking), it knows that “cheap” can be interpreted positively or negatively. Then to avoid making a wrong prediction, it will take a step back, and it will not predict any label. Having many None labels (no predicted labels) leads to a high recall metric.

I will leave you with this, in data science, we make decisions based on our data. Data collection is not a big issue today. On the contrary, the volume and the velocity of Big Data force us to use powerful AI models to process information faster and better by using brutal computational force and smart algorithms to achieve data transformation and knowledge extraction. Having this heavy task automated by cutting-edge software is impressive. It saves us a lot of time and time that we can use to understand these tools and then use them to process information massively. Information leads to knowledge, and we can use it to respond better and faster to changes, understand our business and make better decisions.

Question 9

How can I annotate my data efficiently?

Accepted Answer

Summary

We made the building of an AI model super easy by creating Studio. You have probably already seen on Studio that this is a 4 simple steps process:

Upload Data ---> Define Aspects ---> Annotate Data ---> Start training

After automating the training process, we left only one task for the user: to label, or in other words to annotate, data.

Nevertheless, manual annotation is still a step that demands:

some background knowledge
some of your time for its completeness

In this article, you will learn everything you need to know for annotating data quickly, efficiently & correctly.

Text Annotation Guidelines

Annotators help AI models associate text segments to tags (topics & sentiments). To achieve that more efficiently, they should:

Involve domain experts in the process
Define topics to be comprehensive and non-overlapping in coverage
Write definitions for each tag and add examples
Annotate based on explicit mentions of topics with sentiments (= opinion)
Ignore implicit and subjective topics and factual statements
Focus on producing high-quality data, take breaks to refresh focus
Iterate between annotating and training to speed up the process To shed more light on the process, we mention a few examples

Below you can see how someone can define some aspects in a gsheet efficiently. Firstly, create larger groups to categorize the aspects, and then determine the aspects by providing an example and a description of each case.

https://malcolm-en-gb.s3.eu-west-1.amazonaws.com/instances/oiJS2eQZyE/resources/wPQ58HpA2Y/Screenshot%20from%202021-05-25%2010-12-37.png (Screenshot from 2021-05-25 10-12-37.png)

To shed more light on the process, we mention below two good practices along with exemplary cases.

Be careful only to annotate explicitly mentioned topics with sentiments. AI models can only process and learn from the information that is present in the text.
Try to ignore implicit and subjective or factual segments. Ignoring subjectivity and factual statements improves the quality of the opinion mining model.

Example 1: Facts have no sentiment, so they shouldn’t be annotated.

Example 2: No opinion was explicitly expressed about Delivery.

Example 3 - Not being able to open a bottle makes it not easy to use. It might have a nice design.

Example 4 - The customer received damaged quality. No mention of packaging.

Question 10

What is AI?

Accepted Answer

Artificial Intelligence is the science of making machines "bright". It is based on the principle that human intelligence can be defined in a way that a machine can easily mimic "cognitive" behavior such as problem-solving or learning. The fields of application are endless. Goals include learning, reasoning, and perception.