patch-by-patch. Specifically, we use a deep encoder-decoder network to predict the patch-wise momentum. As shown in Fig. \[fig:1\], in 3D the inputs are two layers of $15 \times 15 \times 15$ image patches ($15 \times 15$ in 2D), where the two layers are from the source and target images respectively. Two patches are taken at the same position by two parallel encoders, which learn features independently. The output is the predicted initial momentum in the $x$, $y$ and $z$ directions (obtained by numerical optimization on the training samples). Basically, the network is split into an encoder and a decoder part. An *encoder* consists of 2 blocks of three 3 $\times$ 3 $\times$ 3 convolutional layers with PReLU activations, followed by another 2 $\times$ 2 $\times$ 2 convolution+PReLU with a stride of two, serving as a “pooling” operation. The number of features in the first convolutional layer is 64 and increases to 128 in the second. In the *decoder*, three parallel decoders share the same input generated from the encoder. Each decoder is the inverse of the encoder except for using 3D transposed convolution layers with a stride of two to perform “unpooling”, and no non-linearity at the end. To speed 