<p>In case someone is having a similar issue with future sequential (or temporal) data, University of Oxford and Google Cloud AI have come up with a new architecture to handle all three types of input (past temporal, future temporal as well as static). It is called <strong>Temporal Fusion Transformer</strong> and, at least from reading the <a href="https://arxiv.org/abs/1912.09363" rel="nofollow noreferrer">paper</a>, looks like a neat fit. However, I have yet to implement and test it. There is also a <a href="https://pytorch-forecasting.readthedocs.io/en/latest/tutorials/stallion.html" rel="nofollow noreferrer">PyTorch Tutorial</a> available.</p>
How to inject future sequential input for multi-step time series forecasts in LSTM networks
<p>I am trying to do multi-step (i.e., sequence-to-sequence) forecasts for product sales using both (multivariate) sequential and non-sequential inputs.</p>
<p>Specifically, I am using sales numbers as well as some other sequential inputs (e.g., price, is day before holiday, etc...) of the past n days to predict the sales for future m days. Additionally, I have some non-sequential features characterizing the product itself.</p>
<p>Definitions:</p>
<ul>
<li>n_seq_features &lt;- number of sequential features (in the multivariate time-series) including sales</li>
<li>n_non_seq_features &lt;- number of non-sequential features characterizing a product</li>
</ul>
<p>I got as far as building a hybrid-model, where first the sequential input is passed through some LSTM layers. The output of the final LSTM layer is then concatenated with the non-sequential features and fed into some dense layers.</p>
<p>What I can't quite get my head around, though, is how to input future sequntial input (everything except sales numbers for the following m days) in a way that efficiently utilizes the sequential information (i.e., causality, etc...). For m=1, I can simply input the sequential data for this one day together with the non-sequential input after the LSTM layers, however as soon as m becomes greater than 1 this appears to be a waste of causal information.</p>
<p>The only ways I could think of were:</p>
<ul>
<li>to incorporate the sequential information for future m days as features in the LSTM input blowing up the input shape from (..., n, n_seq_features) to (..., n, n_seq_features + m*(n_seq_features-1))</li>
<li>add a separate LSTM branch handling the future data, the output of which is then 'somehow' fed into the dense layers at the last stage of the model</li>
</ul>
<p>I only started using LSTM networks a while ago so I unfortunately have only limited intuition on how they are best utilized (especially in hybrid approaches). For this reason, I would like to ask:</p>
<ol>
<li>Is the general approach of injecting sequential and non-sequential input at different stages of the same model (i.e., trained concurrently) useful or would one rather split it into separate models which can be trained independently for more fine-grained control?</li>
<li>How is future sequential input injected into an LSTM network to preserve causal information? Can this be achieved with a high-level frontend like KERAS or does it require a very deep dive into the tensorflow backend?</li>
<li>Are LSTM networks not the way to go for this specific problem in the first place?</li>
</ol>
<p>Cheers and thanks in advance for any advice, resources or thoughts on the matter. :)</p>