eGitty

Discover The Most Popular Algorithms

Limitations of Attention Mechanism in Text Classification

Attention mechanism is widely used in deep learning. It has been proved that it can improve the performance of deep learning model. However, it also has some limitations.

We will use a paper to illustrate it.

This paper is: Relational Graph Attention Network for Aspect-based Sentiment Analysis

In this paper, the limitation of attention method is proposed in aspect-based sentiment analysis.

For example:

Limitations of Attention Mechanism in Text Classification

In the first example, the word like is used as a verb and it expresses a positive sentiment towards the aspect recipe, which is successfully attended by the attention-based LSTM model. However, when it is used as a preposition
in the second example, the model still attends to it with a high weight, resulting in a wrong prediction. The third example shows a case where there are two aspects in a single sentence with different sentiment polarities. For the aspect chicken, the LSTM model mistakenly assigns high attention weights to the words but and dried, which leads to another prediction mistake.

How to solve this problem?

We can use word part-of-speech or syntax information.

Leave a Reply

Your email address will not be published. Required fields are marked *