Topic attention is proposed in paper Aspect Category Detection via Topic-Attention Network, it can incorporate topic information in self-attention mechanism. It is an improvement for self-attention.
Why use topic attention?
In a same sentence, topic attention can compute different attention score for the same words based on different topic. For example, as to sentence:
It is very overpriced and not very tasty
As to topic food and price, tasty and overpriced will get different attention score.
How to implement topic attention?
Topic attention can be computed as follows:
Here \(h_t\) can be computed by LSTM, GRU or BiLSTM. \(T_i\) is the embedding of the \(i\)-th topic.
\(e_{it}\) can be viewed as the correlation of word \(h_t\) and topic \(T_i\). We should notice it is linear.
By this method, \(a_{it}\) will be different as to different topic.
Here is an effect.