<p>I assume you want to forecast 50 time steps with the 125 previous ones (as an example). I give you the most basic Encoder-Decoder Structure for time Series but it can be improved (with <a href="https://arxiv.org/abs/1508.04025" rel="nofollow noreferrer">Luong Attention</a> for instance).</p>
<pre><code>from tensorflow.keras import layers,models

input_timesteps=125
input_features=2
output_timesteps=50
output_features=2
units=100

#Input
encoder_inputs = layers.Input(shape=(input_timesteps,input_features))

#Encoder
encoder = layers.LSTM(units, return_state=True, return_sequences=False)
encoder_outputs, state_h, state_c = encoder(encoder_inputs) # because return_sequences=False =&gt; encoder_outputs=state_h

#Decoder
decoder = layers.RepeatVector(output_timesteps)(state_h)
decoder_lstm = layers.LSTM(units, return_sequences=True, return_state=False)
decoder = decoder_lstm(decoder, initial_state=[state_h, state_c])


#Output
out = layers.TimeDistributed(Dense(output_features))(decoder)

model = models.Model(encoder_inputs, out)
</code></pre>
<p>So the core idea here is :</p>
<ol>
<li>Encode the time series into two states : <code>state_h</code> and <code>state_c</code>. Check <a href="https://colah.github.io/posts/2015-08-Understanding-LSTMs/" rel="nofollow noreferrer">this</a> to understand the work of LSTM cells.</li>
<li>Repeat <code>state_h</code> the number of time steps you want to forecast</li>
<li>Decode using an LSTM with initial states calculated by the encoder</li>
<li>Use a Dense layer to shape the number of needed features for each time steps</li>
</ol>
<p>I advise you to test our achtecture and visualize them with <code>model.summary()</code> and <code>tf.keras.utils.plot_model(mode,show_shapes=True)</code>. It gives you good representations like, for the summary :</p>
<pre><code>Layer (type)                    Output Shape         Param #     Connected to                     
==================================================================================================
input_5 (InputLayer)            [(None, 125, 2)]     0                                            
__________________________________________________________________________________________________
lstm_8 (LSTM)                   [(None, 100), (None, 41200       input_5[0][0]                    
__________________________________________________________________________________________________
repeat_vector_4 (RepeatVector)  (None, 50, 100)      0           lstm_8[0][1]                     
__________________________________________________________________________________________________
lstm_9 (LSTM)                   (None, 50, 100)      80400       repeat_vector_4[0][0]            
                                                                 lstm_8[0][1]                     
                                                                 lstm_8[0][2]                     
__________________________________________________________________________________________________
time_distributed_4 (TimeDistrib (None, 50, 2)        202         lstm_9[0][0]                     
==================================================================================================
Total params: 121,802
Trainable params: 121,802
Non-trainable params: 0
__________________________________________________________________________________________________
</code></pre>
<p>and the model plotted :</p>
<p><a href="https://i.stack.imgur.com/IDpvr.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/IDpvr.png" alt="enter image description here" /></a></p>
Encoder-Decoder for Trajectory Prediction
<p>I need to use encoder-decoder structure to predict 2D trajectories. As almost all available tutorials are related to NLP -with sparse vectors-, I couldn't be sure about how to adapt the solutions to a continuous data.</p>
<p>In addition to my ignorance in seqence-to-sequence models, <code>embedding</code> process for words confused me more. I have a dataset that consists of 3,000,000 samples each having <code>x-y</code> coordinates (-1, 1) with <code>125</code> observations, which means the shape of each sample is <code>(125, 2)</code>. I thought I could think of this as 125 words with 2 dimensional already embedded words, but the encoder and the decoder in this <a href="https://blog.keras.io/a-ten-minute-introduction-to-sequence-to-sequence-learning-in-keras.html" rel="nofollow noreferrer">Keras Tutorial</a> expect 3D arrays as <code>(num_pairs, max_english_sentence_length, num_english_characters)</code>.</p>
<p>I doubt I need to train each sample <code>(125, 2)</code> separately with this model, as the way Google's search bar does with only one word written.</p>
<p>As far as I understood, an encoder is <code>many-to-one</code> type model and a decoder is <code>one-to-many</code> type model. I need to get a memory state <code>c</code> and a hiddenstate <code>h</code> as vectors(?). Then I should use those vectors as input to decoder and extract predictions in the shape of (x,y) as many as I determine as encoder output.</p>
<p>I'd be so thankful if someone could give an example of an encoder-decoder LSTM architecture over the shape of my dataset, especially in terms of dimensions required for encoder-decoder inputs and outputs, particulary on Keras model if possible.</p>