video [@Oneata:ICCV13; @Tang:ICCV13; @Yuan:CVPR16], which leads to huge computations for further classification. In contrast, content-dependent proposal methods, e.g., [@Buch:CVPR17; @Escorcia:ECCV16; @Gao:ECCV18; @Gao:ICCV17; @Lin:ECCV18], utilize the label of action proposals during training. For instance, Escorcia *et al.* [@Escorcia:ECCV16] leverage Long Short-Term Memory cells to learn an appropriate encoding of a video sequence as a set of discriminative states to indicate proposal scores. Though the method avoids running sliding windows of multiple scales, there is still the need of executing an overlapping sliding window that is inapplicable when the video duration is long. To address this problem, Single Stream Temporal proposal (SST) [@Buch:CVPR17] generates proposals with only one single pass by utilizing a recurrent GRU-based model, and Temporal Unit Regression Network (TURN) [@Gao:ICCV17] builds video units in a pyramid manner to avoid window overlapping. Different from the above methods which generate proposals in a fixed multi-scale manner, Boundary Sensitive Network (BSN) [@Lin:ECCV18] localizes the action boundaries based on three actionness curves in a more flexible way. Nevertheless, such actionness-based methods may fail in locating dense and short actions because of the difficulty to discriminate between very close starting and ending peaks in the curve.

Once the localization of action proposals completes, the 