A new approach is proposed to support efficient convolution for deep learning by vectorizing multi-dimensional input data for multi-dimensional fast Fourier transform (FFT) and direct memory access (DMA) for data transfer. Specifically, a deep learning processor (DLP) includes a plurality of tensor engines each configured to perform convolution operations by applying one or more kernels on multi-dimensional input data for pattern recognition and classification based on a neural network, wherein each tensor engine includes, among other components, one or more vector processing engines each configured to vectorize the multi-dimensional input data at each layer of the neural network to generate a plurality of vectors and to perform multi-dimensional FFT on the generated vectors and/or the kernels to create output for the convolution operations. Each tensor engine further includes a data engine configured to prefetch the multi-dimensional data and/or the kernels to both on-chip and external memories via DMA.
Please help me write a proper abstract based on the patent claims. CLAIM: 1. A hardware-based programmable deep learning processor (DLP), comprising: an on-system memory (OSM) and one or more controllers configured to access a plurality of external memory resources via direct memory access (DMA); a plurality of programmable tensor engines configured to perform a plurality of convolution operations by applying one or more kernels on multi-dimensional input data to generate deep learning processing results for pattern recognition and classification based on a neural network, wherein each of the tensor engines further comprises: a data engine configured to prefetch the multi-dimensional input data and/or the kernels from the OSM and/or the external memory resources for the convolution operations; one or more vector processing engines each configured to vectorize the multi-dimensional input data at each layer of the neural network to generate a plurality of vectors; perform multi-dimensional fast Fourier transform (FFT) on the generated vectors and/or the kernels to create output for the convolution operations. 2. The processor of claim 1, wherein: each of the plurality of tensor engines further includes a programmable CPU having its own instruction cache and data cache configured to store instructions from a host and the retrieved data from the OSM and/or the external memory resources, respectively. 3. The processor of claim 2, wherein: the DLP is configured to accept a plurality of instructions from the host and submit the instructions to the tensor engines and their respective components in the DLP, wherein the instructions are stored in the instruction cache of the tensor engines. 4. The processor of claim 2, wherein: the deep learning processing results by the DLP back to the host. 5. The processor of claim 1, wherein: the DLP is configured to partition each convolution operation for pattern classification among the plurality of vector processing engines, wherein each vector processing engine is configured to perform a sub-task of the convolution operation in parallel. 6. The processor of claim 1, wherein: each kernel is a multi-dimensional matrix having its own values for elements in the matrix, wherein the dimensions represent (x, y, time) coordinates as well as depth of the elements of the kernel. 7. The processor of claim 1, wherein: each tensor engine further includes an online memory (OLM) configured to store the prefetched input data from the OSM and/or the external memory resources. 8. The processor of claim 7, wherein: each vector processing engine is configured to retrieve a vector of input data across V out of N rows column-wise from each column of the multi-dimensional input data, one column at a time, wherein the multiple dimensional input data is stored in the OSM or the OLM in column major. 9. The processor of claim 8, wherein: each vector processing engine is configured to perform divide and conquer to partition the retrieved vectors based on a primary factor; perform a butterfly operation on the partitioned vectors, wherein V number of FFT operations are performed on each of the V rows of retrieved input data in parallel and only the primary factor number of partitioned vectors need to be loaded from the memory for the FFT; store output from the FFT back to the same location in the memory as the input data is retrieved. 10. The processor of claim 9, wherein: multiple set of vector processing engines each having the primary factor number of vector processing engines are applied to perform the FFT operations on the partitioned vectors simultaneously in parallel. 11. The processor of claim 1, wherein: the vector processing engines are configured to perform the FFT on the input data having more than two-dimensions. 12. The processor of claim 11, wherein: each vector processing engine is configured to retrieve the vectors contiguously from each row of the multi-dimensional input data for FFT except for the first dimension; perform vectorization on a single row of the first dimension by performing a divide and conquer partition on the row until a granularity of S elements is reached; perform the FFT on vectors of elements across the row of the first dimension. 13. The processor of claim 12, wherein: each vector processing engine is configured to perform the divide and conquer partition on the row to register and/or sub-register level after the granularity of S elements is reached by utilizing a plurality of registers associated with each vector processing engine. 14. A method to support vectorized fast Fourier transform (FFT) for multi-dimensional convolution, comprising: prefetching multi-dimensional input data and/or one or more kernels from an on-system memory (OSM) and/or a plurality of external memory resources via direct memory access (DMA); vectorizing the multi-dimensional input data to generate a plurality of vectors at each layer of a neural network used for a plurality of convolution operations; performing multi-dimensional fast Fourier transform (FFT) on the generated vectors and/or the kernels to create output for the convolution operations; outputting deep learning processing results for pattern recognition and classification to a host based on the output for the convolution operations. 15. The method of claim 14, further comprising: accepting a plurality of instructions from the host and submitting the instructions to a plurality of vector processing engines, wherein the instructions are stored in an instruction cache of the vector processing engines. 16. The method of claim 15, further comprising: partitioning each convolution operation for pattern classification among the plurality of vector processing engines, wherein each vector processing engine is configured to perform a sub-task of the convolution operation in parallel. 17. The method of claim 14, wherein: each kernel is a multi-dimensional matrix having its own values for elements in the matrix, wherein the dimensions represent (x, y, time) coordinates as well as depth of the elements of the kernel. 18. The method of claim 14, further comprising: retrieving a vector of input data across V out of N rows column-wise from each column of the multi-dimensional input data, one column at a time, wherein the multiple dimensional input data is stored in the OSM or an online memory (OLM) in column major. 19. The method of claim 18, further comprising: performing divide and conquer to partition the retrieved vectors based on a primary factor; performing a butterfly operation on the partitioned vectors, wherein V number of FFT operations are performed on each of the V rows of retrieved input data in parallel and only the primary factor number of partitioned vectors needs to be loaded from the memory for the FFT; storing output from the FFT back to the same location in the memory as the input data is retrieved. 20. The method of claim 19, further comprising: applying multiple set of vector processing engines each having the primary factor number of vector processing engines to perform the FFT operations on the partitioned vectors simultaneously in parallel. 21. The method of claim 14, further comprising: performing the FFT on the input data having more than two-dimensions. 22. The method of claim 21, further comprising: retrieving the vectors contiguously from each row of the multi-dimensional input data for FFT except for the first dimension; performing vectorization on a single row of the first dimension by performing a divide and conquer partition on the row until a granularity of S elements is reached; performing the FFT on vectors of elements across the row of the first dimension. 23. The method of claim 22, further comprising: performing the divide and conquer partition on the row to register and/or sub-register level after the granularity of S elements is reached by utilizing a plurality of registers.