have been reading up a bit on LSTM's and their use for time series and its been interesting but difficult at the same time. One thing I have had difficulties with understanding is the approach to adding additional features to what is already a list of time series features. Assuming you have your dataset up like this:


Now lets say you know you have a feature that does affect the output but its not necessarily a time series feature, lets say its the weather outside. Is this something you can just add and the LSTM will be able to distinguish what is the time series aspect and what isnt?

  • $\begingroup$ I like your question. However, can you elaborate how this non-time series feature influence the output at time t, based on subject matter knowledge. $\endgroup$
    – horaceT
    Feb 23 '17 at 20:28
  • $\begingroup$ The title was misleading, I thought it was adding new features to a model that had already been trained and then continuing with the training. $\endgroup$
    – Joke Huang
    Feb 19 at 5:55


# Input tensor to RNN
    # Timestep 1
    [ temperature_in_paris, value_of_nasdaq, unemployment_rate ],
    # Timestep 2
    [ temperature_in_paris, value_of_nasdaq, unemployment_rate ],
    # Timestep 3
    [ temperature_in_paris, value_of_nasdaq, unemployment_rate ],




AUXILIARY_INPUTS --> [do something] --/




In other cases, as @horaceT points out, you may want to condition the LSTM on non-temporal data. For example, predict the weather tomorrow, given location. In this case, here are three suggestions, each with positive/negatives:




    1. For each training sample, take your condition variables $\vec{x}$.
    2. Transform/reshape your condition variables with an affine transformation to get it into the right shape as the internal state of the RNN: $\vec{v} = \mathbf{W} \vec{x} + \vec{b}$ (these $\mathbf{W}$ and $\vec{b}$ are trainable weights). You can obtain it with a Dense layer in keras.
    3. For the very first timestep, add $\vec{v}$ to the hidden state of the RNN when calculating its value.


  • 1
    $\begingroup$ Good suggestion, but what if the output of the LSTM has structural dependence on a non-time series predictor. $\endgroup$
    – horaceT
    Feb 23 '17 at 18:26
  • 1
    $\begingroup$ Could you give an example? $\endgroup$ Feb 23 '17 at 18:27
  • 8
    $\begingroup$ OK, here is a very artificial example. Say you're trying to predict weather at time t, based on obs from last n time steps. Weather depends on the part of the world you're in. If it's summer in northern hemisphere, it's winter in southern hemisphere. So this north/south factor should be taken into account. Can you incorp it into LSTM? $\endgroup$
    – horaceT
    Feb 23 '17 at 18:32
  • 1
    $\begingroup$ Great question! I've included edits to address this. $\endgroup$ Feb 24 '17 at 16:16
  • $\begingroup$ @AdamSypniewski, are you sure that recommendation is what those papers are doing? My read is that Karpathy & Fei-Fei are encoding an image to h-space via one model (a CNN), and encoding a sentence into h-space via a separate model (an LSTM), and then they infer from there; I think Vinyals et al are feeding in their conditioning variable at time '-1' before the start of the sequence. Am I wrong? $\endgroup$ Jan 9 '18 at 16:52

Based on all the good answers of this thread, I wrote a library to condition on auxiliary inputs. It abstracts all the complexity and has been designed to be as user-friendly as possible:

https://github.com/philipperemy/cond_rnn/ (tensorflow)

Hope it helps!


Adam's answer does seem to make the most sense, however, I am not sure about the second statement "Polluting sequential data with non-sequential information".

So recently I trained a character-level LSTM model, in which I just appended a non-sequential feature in the end of the sequential features. The model learned how to differentiate that pretty well.

The question if the model will perform better had I done it Adam's way, is still to be tested. But for people who don't want to go the extra mile, appending non-sequential features to sequential ones works just fine.


There is a function in keras LSTM reset_states(states).

However the parameter states is the concatination of two states, hidden state h and cell state.

States = [h, c]

it would be interesting to know if you should initialize h or c according to the approaches in the above mentioned papers.


This is probably not the most efficient way, but the static variables could be repeated to timeseries length using tf.tile().


Your Answer

