<p>This problem has been tackled with both Generative Adverserial Networks (GAN) and RNNs with decent degree of success. </p>

<p>One fairly successful approach using GANs, introduced by Facebook AI Research <a href="https://arxiv.org/pdf/1511.05440.pdf" rel="noreferrer">here</a>, uses a multi-scale generator. Basically, you sample down the image at various scales, and then predict the next frame for that particular lower resolution. Then using the upsampled predicted frame and original frame at next higher resolution you predict the next frame for that higher scale and the process continues. Multi-scale aggregation helps prevent blurriness and retains details. You can find the code <a href="https://github.com/coupriec/VideoPredictionICLR2016" rel="noreferrer">here</a></p>

<p>A more recent approach uses a combination of Autoencoder and GANs, Basically a Variational encoder recurrently compresses the entire video stream into a latent space and you have different networks to predict flow and the next frame from the latent space representation. It then fuses next frame along with information from predicted flow. You can read the paper <a href="http://openaccess.thecvf.com/content_ICCV_2017/papers/Liang_Dual_Motion_GAN_ICCV_2017_paper.pdf" rel="noreferrer">here</a> for details.</p>

<p>Another approach from Cornell, does this without GANs. It's rather complex, but in simple terms they use Stacked LSTMs, and propagate error signals to further LSTM layers whose job is then to predict the error for next frame. Here's the <a href="https://arxiv.org/pdf/1605.08104.pdf" rel="noreferrer">paper</a></p>

<p>There are other approaches too, but these seem to be widely cited and have code available online.</p>
Image sequence prediction with keras
<p>I'm having fun with keras lately and i would like to know how one would approach this problem.</p>

<p>I have a sequence of 100 images. They are daily images of a radar map, for 100 consecutive days. I would like to predict the image for the next day.</p>

<p>Theses images can be interpreted as matrices of n x m dimensions ( not square ) .</p>

<p>Can this be adapted to a lstm nn? How would you approach this problem? </p>

<p>Thanks for sharing ideas! </p>