Aspects of the present disclosure are directed to techniques that improve performance of CNN systems through the effect of improved memory efficiencies for CNNs operating on GPUs. Aspects of the disclosure demonstrate that off-chip memory in such CNN systems is underutilized due to at least three characteristics namely, data layout, data locality and inter-kernel redundancy. Aspects of the disclosure examine the performance impact of different data layouts and then describe a method to produce data layout selection for various layers of the CNN including a fast transformation implementation. Disclosed are improvements to data locality from working set expansion, elimination of inter-kernel redundancy and increase of TLP using kernel reconstruction techniques including kernel fusion and thread injection. Disclosed experimental results show that our optimizations are very effective to boost the performance of CNNs by amounts up to 9.76 times for a single kernel and 2.05 times for a network.
Please help me write a proper abstract based on the patent claims. CLAIM: 1. A method for improving memory efficiency of a convolutional neural network (CNN) operating on a graphics processing unit (GPU), the method comprising the computer implemented steps of: generating layout transformation for data operated on by the CNN; enabling efficient utilization of on-chip memory resource(s) in the GPU such that data locality in a pooling layer of the CNN is exploited; and reducing any inter-kernel redundancy while increasing a number of threads employed through the effect of kernel fusion and thread injection. 2. The method of claim 1 wherein the layout transformation comprises the computer implemented steps of: degrading in the CNN wherein data are organized using four dimensional arrays, the four dimensional arrays into two dimensional arrays by combining dimensions; partitioning each two dimensional array into multiple sub-blocks, and applying a transformation on each sub-block; and optimizing memory patterns to use 8-byte access mode(s) such that a shared memory bandwidth is maximized by providing 256 byte per cycle bandwidth; 3. The method of claim 1 wherein the enablement of efficient utilization of on-chip memory resources in the GPU comprises the computer implemented steps of: based on a stride, partition a pooling window into segments whose size is equal to a number of overlapped elements; expand a working set of any threads employed, by producing multiple outputs for each; and aggregating a set of intermediate results in a register block such that a set of output elements are generated such that consecutive neurons in the CNN use intermediate results directly through low-latency thread-level register tiling. 4. The method of claim 1 wherein inter-kernel redundancy is reduced by the computer implemented steps of: applying kernel fusion optimization and thread injection optimization by fusing multiple kernels into a single kernel without modifying thread block configuration such that communication between the two kernels becomes an inner-thread communication and any data used for inner-thread communication may be cached in register file or shared memory of the GPU and after kernel fusion any intermediate/temporary global memory accesses are eliminated; and parallelizing all the inner loop communications