19\
& & & & & 35\
& & & & & 67\
& & & & & 131\
& & & & & 259\
& & & & & 515\
& & & & & 1027\

\[table3:1\]

**Evaluation Metrics.** We follow the official evaluation metrics in each dataset for action detection task. On ActivityNet v1.3, the mean average precision (mAP) values with IoU thresholds between $0.5$ and $0.95$ (inclusive) with a step size $0.05$ are exploited for comparison. On THUMOS14, the mAP with IoU threshold $0.5$ is measured. We evaluate performances on top-100 and top-200 returned proposals in ActivityNet v1.3 and THUMOS14, respectively.

Evaluation on Temporal Action Proposal
--------------------------------------

We first examine the performances on temporal action proposal task, which is to only assess the boundary quality of action proposals, regardless of action classes. We compare the following advanced approaches: (1) Structure Segment Network (SSN) [@Xiong:ICCV17] generates action proposals by temporal actionness grouping. (2) Single Shot Action Detection (SSAD) [@Lin:MM17] is the 1D variant version of Single Shot Detection [@Liu:ECCV16], which generates action proposals by multiple temporal anchor layers. (3) Convolution-De-Convolution Network (CDC) [@Shou:CVPR17] builds a 3D Conv-Deconv network to precisely localize the boundary of action instances at frame 