eGitty

Discover The Most Popular Algorithms

An Introduction to Topic Attention in Deep Learning

Topic attention is proposed in paper Aspect Category Detection via Topic-Attention Network, it can incorporate topic information in self-attention mechanism. It is an improvement for self-attention.

Why use topic attention?

In a same sentence, topic attention can compute different attention score for the same words based on different topic. For example, as to sentence:

It is very overpriced and not very tasty

As to topic food and price, tasty and overpriced will get different attention score.

How to implement topic attention?

Topic attention can be computed as follows:

An Introduction to Topic Attention in Deep Learning

Here \(h_t\) can be computed by LSTM, GRU or BiLSTM. \(T_i\) is the embedding of the \(i\)-th topic.

\(e_{it}\) can be viewed as the correlation of word \(h_t\) and topic \(T_i\). We should notice it is linear.

By this method, \(a_{it}\) will be different as to different topic.

Here is an effect.

the effect of topic attention in deep learning

Leave a Reply

Your email address will not be published. Required fields are marked *