deep convolutional neural networks (CNNs) which could generate high quality candidate boxes. Then at the second stage, a region-wise subnetwork is designed to classify and refine these candidate boxes. Using very deep CNNs [@resnet][@vgg16], the Fast R-CNN pipeline [@frcnn][@fasterrcnn] has recently shown high accuracy on mainstream object detection benchmarks [@pascalvoc][@imagenet][@coco]. The region proposal stage could reject most of the background samples, thus the searching space for object detection is largely reduced [@mcg][@edgeboxes]. Multi-stage training process is usually developed for joint optimization of region proposal generation and post detection (e.g., [@rfcn][@fasterrcnn][@hypernet]). In Fast R-CNN [@frcnn], the region-wise subnetwork repeatedly evaluates thousands of region proposals to generate detection scores. Under Fast R-CNN pipeline, Faster R-CNN shares full-image convolutional features with the detection network to enable nearly cost-free region proposals. Recently, R-FCN [@rfcn] tries to make the unshared per-RoI computation of Faster R-CNN to be sharable by adding position-sensitive score maps. Nevertheless, R-FCN still needs region proposals generated from region proposal networks [@fasterrcnn]. To ensure detection accuracy, all methods resize the image to a large enough size (usually with the shortest side of 600 pixels). It is somewhat resource/time consuming when feeding the image into deep networks, both in training and inference time. 