eGitty

Discover The Most Popular Algorithms

Using IndRNN to Handle Longer Sequence than LSTM in TensorFlow

LSTM usually can handle about 200 length sequence effectively. However, if you should handle more than 200, for example 2000 length, how to do?

How Long Sequence Can be Processed Effectively by LSTM? – LSTM Tutorial

In this tutorial, we will introduce IndRNN, which is the tensorflow implementation of the paper: Independently Recurrent Neural Network (IndRNN): Building A Longer and Deeper RNN.

How is IndRNN better than LSTM?

The IndRNN model modifies the computation of hidden state and output, it is:

The structure of IndRNN

The performance of IndRNN

IndRNN can handle longer sequence than LSTM, here is a comparative result.

The performance of IndRNN

From the result, we can find: LSTM perform worse when the sequence length is more than 200. However, IndRNN has a good performance. Even if the sequence length is 5000, IndRNN is a suitable performance.

How to use IndRNN

Step 1: Copy ind_rnn_cell.py into your project

Step 2: Use code below

from ind_rnn_cell import IndRNNCell

# Regulate each neuron's recurrent weight as recommended in the paper
recurrent_max = pow(2, 1 / TIME_STEPS)

cell = MultiRNNCell([IndRNNCell(128, recurrent_max_abs=recurrent_max),
                     IndRNNCell(128, recurrent_max_abs=recurrent_max)])
output, state = tf.nn.dynamic_rnn(cell, input_data, dtype=tf.float32)
...

Leave a Reply

Your email address will not be published. Required fields are marked *