Recurrent Neural Networks architectures excel at processing sequences by
 modelling dependencies over different timescales. The recently introduced
 Recurrent Weighted Average (RWA) unit captures long term dependencies
 far better than an LSTM on several challenging tasks. The RWA achieves
 this by applying attention to each input and computing a weighted average
 over the full history of its computations. Unfortunately, the RWA cannot
 change the attention it has assigned to previous timesteps, and so struggles
 with carrying out consecutive tasks or tasks with changing requirements.
 We present the Recurrent Discounted Attention (RDA) unit that builds on
 the RWA by additionally allowing the discounting of the past.
 We empirically compare our model to RWA, LSTM and GRU units on
 several challenging tasks. On tasks with a single output the RWA, RDA and
 GRU units learn much quicker than the LSTM and with better performance.
 On the multiple sequence copy task our RDA unit learns the task three
 times as quickly as the LSTM or GRU units while the RWA fails to learn at
 all. On the Wikipedia character prediction task the LSTM performs best
 but it followed closely by our RDA unit. Overall our RDA unit performs
 well and is sample efficient on a large variety of sequence tasks.
We introduce the Recurrent Discounted Unit which applies attention to any length sequence in linear time This paper proposes the Recurrent Discounted Attention (RDA), an extension to Recurrent Weighted Average (RWA) by adding a discount factor. Extends the recurrent weight average to overcome the limitation of the original method while maintaining its advantage and proposes the method of using Elman nets as the base RNN