Unsupervised Learning for Optical Flow Estimation Using Pyramid Convolution LSTM
Shuosen Guan Haoxin Li Wei-Shi Zheng
cs.CV















let@tokenonedot
onedotlet@token..

e.g E.g
i.e I.e
c.f C.f
etc vs
w.r.t d.o.f
et al


𝐱
ŁL







Unsupervised Learning for Optical Flow Estimation Using Pyramid Convolution LSTM




Shuosen Guan, Haoxin Li, Wei-Shi Zheng





Code is available at: 

            <https://github.com/Kwanss/PCLNet>




For reference of this work, please cite:



Shuosen Guan, Haoxin Li, Wei-Shi Zheng.
“Unsupervised Learning for Optical Flow Estimation Using Pyramid Convolution LSTM.”
In Proceedings of IEEE International Conference on Multimedia and Expo(ICME). 2019.




Bib:

@inproceedings{PCLNet-icme2019,

     title={Unsupervised Learning for Optical Flow Estimation Using Pyramid Convolution LSTM},

    author={Shuosen Guan, Haoxin Li, Wei-Shi Zheng},

    booktitle={Proceedings of IEEE International Conference on Multimedia and Expo(ICME)},

    year={2019}

}











Shuosen Guan1,3 Haoxin Li2,3 Wei-Shi Zheng1,3,** Corresponding author
1School of Data and Computer Science, Sun Yat-sen University, China 

2School of Electronics and Information Technology, Sun Yat-sen University, China

3Key Laboratory of Machine Intelligence and Advanced Computing, Ministry of Education, China 

,  ,  

Unsupervised Learning for Optical Flow Estimation Using

Pyramid Convolution LSTM
    Christopher J. Moore
    December 30, 2023
=================================================================================




Most of current Convolution Neural Network (CNN)
based methods for optical flow estimation focus on learning
optical flow on synthetic datasets with groundtruth, which is
not practical. In this paper, we propose an unsupervised optical
flow estimation framework named PCLNet. It uses pyramid
Convolution LSTM (ConvLSTM) with the constraint of
adjacent frame reconstruction, which allows flexibly estimating
multi-frame optical flows from any video clip. Besides,
by decoupling motion feature learning and optical flow representation,
our method avoids complex short-cut connections used in existing frameworks while improving accuracy of optical flow estimation. Moreover, different from those
methods using specialized CNN architectures for capturing motion, our framework directly learns optical
flow from the features of generic CNNs and thus can be easily
embedded in any CNN based frameworks for other tasks.
Extensive experiments have verified that our method not only estimates
optical flow effectively and accurately, but also obtains
comparable performance on action recognition.



 


Optical flow estimation, motion representation, unsupervised learning




§ INTRODUCTION


As a key problem in video analysis, optical flow estimation is widely used in lots of fields such as visual SLAM, autonomous driving, action recognition, . Traditional optical flow estimation methods (TVL-1 <cit.>) treat the estimation problem as a energy function minimization problem, making strong assumptions on the pixel-level information such as brightness and spatial smoothness <cit.>. These methods are time-consuming and are hard to use for large scale video data. 

With the remarkable success of CNN in image understanding, the optical flow estimation problem  has been promoted by recently proposed deep learning based methods such as FlowNet <cit.>, FlowNet2.0 <cit.>, PWC-Net <cit.>, . These methods are end-to-end trainable and meanwhile achieve better accuracy and lower time consumption than the traditional methods. However, most of these CNN based methods need groundtruth optical flow for supervision during learning, which is almost unavailable in lots of real-world applications, such as action recognition on website videos, pedestrian tracking on surveillance videos, .

In addition, these methods generally adopt a “coarse-to-fine" manner to predict optical flow by utilizing  auxiliary short-cut connections among high-level semantic layers. This brings the following shortcomings: First, it increases the complexity on the short-cut connections handling; Second, with this coupled architecture binding motion feature learning and optical flow representation, these frameworks make it hard to extend to other tasks. Moreover, these methods tend to customize  architectures for optical flow estimation and thus generally need training from scratch on some synthetic datasets, which leads to low compatibility with currently mature CNN backbones (VGG, ResNet, ).


In addition, these method generally use a special layer named “correlation" to  optical flow, that is , takes a patch of the second frame as a convolution kernel to conduct convolution on the first frame. It can be seem as a matching schema to find the similar texture between these two frames. Theoretically, to learn large displacement motion between two frames.  Each patches from the second frames should travel all patches on the first frames, which however yields a large result and makes intractable efficient for training. As a result, these method generally use a limitation of maximum displacement for comparisons, which  inevitably introduce restriction for the network.



We address the above issues in this work from three aspects: First, for the sake of unsupervised learning on real-world videos, we formulate the optical flow estimation as a reconstruction problem using adjacent frames and employ the reconstruction loss <cit.> to guide our network training. Second, our framework decouples motion capturing and optical flow representation by first applying spatial pyramid pooling to aggregate multi-scale motion information into low dimension motion features and then decoding optical flow directly from these features. As a result, it avoids complex connections while improving estimation accuracy. Last, our method constructs optical flow learning on generic CNNs and thus can be flexibly embedded in other CNN based architectures for other tasks, meanwhile, significantly reduce the time for network training with ImageNet pre-training. Different from other unsupervised learning methods <cit.>, our method performs optical flow learning using ConvLSTM rather than merely CNN. Therefore, it is able to  simultaneously handle multi-frames on video sequences. 



In summary, the contributions of our work are of three folds: First,  we propose an unsupervised  approach to simultaneously learn multi-frame optical flow on real-world videos without groundtruth. Second, our method decouples motion learning and optical flow representation. In this way, it avoids time consumption on complex short-cut connections among high-level semantic layers while improving estimation accuracy. Last, our model is compatible with generic CNNs and thus can be  embedded in general CNN based frameworks.


In summary, the contributions of our work are as follows. 

    
  * We propose an unsupervised  approach to  simultaneously learn multi-frame optical flow on real-world videos without groundtruth.
    
  * Our method decouples motion learning and optical flow representation. By this way, our model avoids time consumption on short-cut connections between high-level semantic layers, meanwhile, learns low dimension motion feature which can be used for other motion modeling tasks.
    
  * The model we proposed is compatible with generic CNNs, hence can be applied to general CNN based frameworks.









§ RELATED WORK



Traditional methods for flow estimation are mainly based on variational approach. The most representative one  is the method proposed by Horn and Schunck <cit.>.  It estimates optical flow by minimizing an energy function with some photometry assumptions such as brightness consistency and spatial smoothness. However, these assumptions  could not stand in some realistic cases. Other methods make improvements by choosing approximative but looser constraints,  <cit.>. Nevertheless, these methods prone to be  restricted due to the hand-crafted design of these assumptions.

FlowNet <cit.> takes an significant step to first deal with this problem by using a simple U-Net <cit.> architecture to learn optical flow layer by layer in a refinement way. After that, many other CNN based optical flow estimation methods were proposed. Among them, FlowNet2.0 pushes this task forward by stacking several sub-models in FlowNet (FlowNetS and FlowNetC) into a large model and obtains great improvement on estimation. SpyNet <cit.> only calculates optical flow residual at each layer and thus obtains  smaller model size and faster inference speed than previous methods. PWC-Net <cit.> constructs optical flow following three principles: pyramidal processing, warping and cost volume. By applying these principles at each CNN layers, PWC-Net achieved much better results than previous ones with less parameters. To capture motion information, these methods generally compute cost volume at different feature levels using a special CNN layer named “Correlation", which takes a patch of the second feature as a convolution kernel to conduct convolution on the first feature. It can be viewed as a matching scheme to find regions with spatial similarities between two feature representations.

In general, these CNN based methods for optical flow estimation mainly focus on learning optical flow on the videos with groundtruth and use complex short-cut connections for refinement. By contrast, our method explores to learn optical flow on real-world videos without groundtruth data. Moreover, by applying ConvLSTMs to capture motion information on multi-scale and aggregate motion information before optical flow estimation, our framework is capable of learning  multi-frame optical flows without these complex  connections. Besides, different from those methods using “correlation" layer to capture motion between two frames, we use ConvLSTM to implicitly learn the motion dynamics and decode it as optical flow. Thanks to the compatibility with generic CNNs,  our framework can be flexibly embedded in other CNN based networks.









§ APPROACH
 
Our framework mainly consists of three modules: the generic CNN that used for appearance feature extraction, the motion concentration module that learns multi-scale motion representation and the optical flow reconstruction module that estimates optical flows from the motion features (see Figure <ref>). We use the generic ResNet18 <cit.> as our feature extraction network and take the output of  its 4 blocks for the following processing. In the remaining of this section, we present the details about the other two modules:  motion concentration module and the optical flow reconstruction module. 





 §.§ Motion Concentration Module


For completeness, we first give a brief introduction to ConvLSTM that we use for capturing motion dynamics on CNN feature pyramid. Like the original LSTM, ConvLSTM  uses four “gates" to control data flowing and self-updating including input gate i_t, forget gate f_t, cell gate c_t and output gate o_t.  It takes 2D feature maps as input and performs convolution operation  between them. The overall operation can be formulated as:

i_t    = σ(W_xi∗X_t + W_hi∗H_t-1 + W_ci∘c_t-1 + b_i)

f_t    = σ(W_xf∗X_t + W_hf∗H_t-1 + W_cf∘c_t-1 + b_f)

c_t    = f_t∘c_t-1 + i_t ∘tanh(W_xc∗X_t + W_hc∗H_t-1 + b_c)

o_t    = σ(W_xo ∗X_t + W_ho∗H_t-1 + W_co∘c_t + b_0)

H_t    = o_t ∘tanh(c_t)

where ∗ denotes convolution operation, ∘ denotes Hadamard product and σ denotes sigmoid operation. 

We apply a ConvLSTM layer to each output layer of the feature extraction CNN. Suppose that the length of each input clip is l. Then the output hidden state of each ConvLSTM layer has the shape of l×c_i×h_i×w_i, where  c_i,h_i,w_i is the output channel, height and width of the corresponding i^th layer respectively. We take the latter l-1 hidden states as the learned motion expression among these l frames. After that, we use several convolution layers to refine the learned hidden features and pass them to the next layer. Finally, to capture the motion information on different scales, we apply Spatial Pyramid Pooling <cit.> to the output of all layers to generate fixed-length low-dimension motion features.





 §.§ Optical Flow Reconstruction Module


To constrain the model to learn motion dynamics, we reconstruct optical flows among adjacent frames of each input clip from the learned motion features. Since in most cases, the groundtruth optical flow is not available for realistic tasks, inspired by <cit.>, we formulate the optical flow estimation problem as an image reconstruction problem.

Let I={I_1, I_2} be two adjacent frames as input, and then the transformation from I_1 to I_2 can be formulated as I_2=W(I_1, F), where F is the optical flow from I_1 to I_2, and W means warping operation that maps each point of I_1 to I_2 according to the flow F. Conversely, given the estimated flow F̂, we can reconstruct I_1 as Î_̂1̂ = W^-1(I_2, F̂) using the inverse warping operation W^-1 <cit.>. To obtain the estimated optical flow F̂, we minimize the similarity error between Î_̂1̂ and I_1, which can be formulated as

    F̂ = min  L_sim(Î_̂1̂, I_1),

where L_sim is similarity loss function. Here, we use three kinds  of similarity loss function as follows. 

Pixelwise Similarity Loss. It directly computes the pixelwise error between two frames using the generalized Charbonnier penalty function to alleviate the effects of outliers:

    L_diff =1/N∑_i,j^N( (Î_̂1̂(i,j) - I_1(i,j))^2 +ϵ)^α,

where α is the Charbonnier penalty factor.

 Modified Peak Signal-to-Noise Ratio (PSNR) Loss. PSNR  measures the peak error between two images. It is a common pixel difference-based measure of picture degradation. Here, we modified the original PSNR to be greater than 0 :

    L_prsn =  -10 ·log_10(1/1 + MSE(Î_̂1̂, I_1)),

where MSE denotes the mean square error measure. 

Mean Structural Similarity (MSSIM) Loss.  Structural Similarity (SSIM) <cit.> considers image degradation as perceived change of three aspects: luminance, contrast and structure information. It is based on the assumption that closer regions in space have stronger inter-dependencies and provide structure information of image appearance. Overall similarity error is computed by first splitting the reference and target images into small windows and then computing the mean SSIM (MSSIM) error over all windows: 


    L_ssim = 1- 1/M∑_m^MSSIM(Î_̂1̂_m, I_1_m),
 
where SSIM(.) is the standard Structural Similarity function and M denotes the number of windows we split.

Finally, we take the weighted sum of these three similarity loss functions as the final reconstruction loss:

    L_rec = β_1 · L_diff + β_2 · L_prsn  + β_3 · L_ssim
 
Here, β_1, β_2 and β_3 are the balanced weights of these three similarity loss, respectively.

To reconstruct optical flow in higher resolution, we upscale the predicted flow by stacking five Optical Flow Estimation (OFE) blocks (bottom right in Figure <ref>) from bottom to top. Here, the “Flow Prediction" in OFE blocks is a simple convolution layer that shrinks the feature channel to 2 without changing the height and width. We compute the reconstruction loss at each layer and add up to the final unsupervised reconstruction losses. Besides, to capture spatial context for finer flow representation, we apply  a “Context Block" consisting of 7 dilated convolution layers on the top as described in <cit.>. Note that we refine the optical flows merely from the low dimension motion features without utilizing  auxiliary short-cut connections from high-level features. 




§ EXPERIMENTS



 §.§ Datasets

Datasets without groundtruth. We investigate the performance of optical flow estimation on two real-world action recogniton datasets: UCF101 <cit.> and HMDB51 <cit.>. UCF101 consists of 101 action categories and 13,320 videos. HMDB51 contains 6766 videos clips from 51 action classes. 

Datasets with groundtruth. We perform experiments on two synthetic optical flow datasets: FlyingChairs <cit.> and MPI-Sintel <cit.>. FlyingChairs consists of 22,872 frame pairs and is split into 22,232 training examples and 640 testing examples. MPI-Sintel consists of 1628 frames with higher resolution and diverse motion patterns. It is split into training set and testing set. Both of them are divided into three “passes": Albedo, Clean and Final. To supervise our training, we further split the training set into training split and validation split as <cit.>.  





 §.§ Training Details

We first pretrain our network on FlyingChairs and MPI-Sintel dataset. Specifically, We initialize the feature extraction network (ResNet18 in our experiments) with the weights pretrained from ImageNet. In this stage, we use the average end-point-error (EPE) as the training loss. Here, we use single GPU and set batch size as 32 for training. We start training with an initial learning rate of 1e-4 and then decay the learning by 0.5 at step 60K, 80K, 100K and 120K.  For data augmentation, we apply random crop, random scale, horizontal flipping and vertical flipping in the optical flow estimation task. Note that, we perform augmentation jointly for each input clip to keep the optical flows coherent.

Next, we train our model on HMDB51 and UCF101 dataset without groundtruth optical flow supervision. We also apply data augmentation as in the pretraining stage. We split each video into multiple clips. The size of each input clip is set to 6× 224× 224. Note that in this stage, we only use  reconstruction loss for training and set the α as 0.4 (Eq. <ref>), M=7 (Eq. <ref>), and β_1 = 1.0, β_2 = 0.2, β_3 = 0.5 (Eq. <ref>) based on validation. Here, we set batch size as 48 and start training with a learning rate as 1e-3, and then decay the learning by 0.1 when the training loss plateaus. 
All of the above experiments were conducted on NVIDIA Titan V GPUs.








 §.§ Ablation Study



To investigate the impact on our “decouple" design and  compare with those methods using short-cut connections, we design another U-net like PCLNet, which is illustrated in Figure <ref>. We denote this network as “PCLNetC" in the following section. In this model, we add a couple connection between each layer in motion concentration module and the corresponding layer in the optical flow reconstruction module. The remaining part of this model is the same as in PCLNet.


We evaluate these two kinds of models on the training set of MPI-Sintel dataset. The result is shown in Table <ref>. Surprisingly, even using lighter framework, the PCLNet without “couple connections" achieves better result than the coupled one. We own this improvement to the employment of  spatial pyramid pooling: by integrating multi-scale information into low-dimension features before decoding as optical flows, it gains even better performance without using auxiliary information from high-level semantic layers. Moreover, this decoupled design reduces time consumption to a small degree.



 §.§ Comparison Results


Obviously, our framework can be extended for entirely supervised learning by simply replacing the reconstruction loss (Eq. <ref>) to EPE loss. For comparison, we first conduct an experiment to perform optical flow estimation on MPI-Sintel dataset with merely EPE loss. The results are shown in Table <ref>. As shown, our method achieves better performance than traditional TVL-1 on both accuracy and efficiency. As for CNN based optical flow estimation methods, even though our model gains slightly worse accuracy, it achieves promising runtime performance comparing with other methods. It shows that although we estimate optical flow from the low-dimension motion features without high-level short-cut connections, our method still achieves a good trade-off between accuracy and efficiency. The visualized results of the estimated optical flows on both PCLNet and PCLNetC are shown in Figure <ref>. We can see that both of these two kinds of  models are able to estimate clear optical flows while the PCLNet is slightly better on capturing details. 





As for unsupervised learning on videos without groundtruth, we evaluate our method on HMDB51 and UCF101 using PCLNet. The estimated optical flows are visualized in Figure <ref>. As shown, our framework is able to estimate crisp optical flow merely using the reconstruction constraint. To quantitatively verify our method for optical flow estimation on real-world videos, we further perform an experiment on action recognition task on these datasets compared with other optical flow estimation methods including  TVL-1 <cit.>, DIS-Fast <cit.>, Deepflow <cit.>, FlowNet <cit.> and  FlowNet2.0 <cit.>. Specifically, we first pre-compute the optical flows on these two datasets, and then we take them as input and use a ResNet50 network to  perform action recognition. The result is shown in Table <ref>. Although our result is slightly worse than the results of TV-L1, we enjoy higher accuracy than other specialized motion descriptors.














§ CONCLUSIONS

In this paper, we present a novel end-to-end trainable framework for optical flow estimation. By utilizing reconstruction constraint as supervision, our framework is able to efficiently learn optical flow on real-world videos without groundtruth. In addition, we decouple motion feature learning and optical flow reconstruction by applying ConvLSTMs on multi-scale to capture motion dynamics and then decode the learned motion features into optical flows. In this way, our method simultaneously estimates multi-frame optical flow without complex short-cut connections. Furthermore, this framework is compatible with generic CNNs and thus can be easily embedded in CNN based Networks for other video analysis tasks. Experiments verified that our approach achieves  promising results for optical flow estimation on both synthetic and real-world video data.




 
 Acknowledgement This work was supported partially by the National Key Research and Development Program of China (2018YFB1004903), NSFC (61522115,U1811461), Guangdong Province Science and Technology Innovation Leading Talents (2016TX03X157), and the Royal Society Newton Advanced Fellowship (NA150459).















IEEEbib


