weakness and issues which should be clarified: 1) The novelty of this paper is incremental. The proposed method is developed based on the MDNet framework. It seems that the only difference is that the proposed method further incorporate the attention regularization for backward propagation. 2) The regularization term seems a bit ad-hoc. Although the author has provided some intuitive explanation of the regularization, it seems lack of theoretical support. There are some other statistics which may be used to replace role of the mean and standard derivation in the regularization. Why they are not adapted in the regularization? For example, the median which is not sensitive to outlier value of data can be used to replace mean value. 3) The author claims that the proposed method can enable the classifier attend to temporal invariant motion patterns. It seems that no explanation is provided about what motion patterns mean in this paper. Although some figures show the evolvement of attention during training, no motion pattern is illustrated. In addition, some large variations may happen during the tracking process, such as out-plane-rotation, how can the proposed method ensure that the temporal motion invariant pattern can be found and the classifiers can attend to them? [POST-REBUTTAL COMMENTS] I have read the rebuttal and still have the concerns on the theoretical support for the regularization term. I keep my rating.
2) The regularization term seems a bit ad-hoc. Although the author has provided some intuitive explanation of the regularization, it seems lack of theoretical support. There are some other statistics which may be used to replace role of the mean and standard derivation in the regularization. Why they are not adapted in the regularization? For example, the median which is not sensitive to outlier value of data can be used to replace mean value.