A hardware-based programmable deep learning processor (DLP) is proposed, wherein the DLP comprises with a plurality of accelerators dedicated for deep learning processing. Specifically, the DLP includes a plurality of tensor engines configured to perform operations for pattern recognition and classification based on a neural network. Each tensor engine includes one or more matrix multiplier (MatrixMul) engines each configured to perform a plurality of dense and/or sparse vector-matrix and matrix-matrix multiplication operations, one or more convolutional network (ConvNet) engines each configured to perform a plurality of efficient convolution operations on sparse or dense matrices, one or more vector floating point units (VectorFPUs) each configured to perform floating point vector operations, and a data engine configured to retrieve and store multi-dimensional data to both on-chip and external memories.
Please help me write a proper abstract based on the patent claims. CLAIM: 1. A hardware-based programmable deep learning processor (DLP), comprising: an on-system memory (OSM) and one or more controllers configured to access a plurality of external memory resources via direct memory access (DMA); a plurality of programmable tensor engines configured to perform a plurality of operations on input data to generate deep learning processing results for pattern recognition and classification based on a neural network, wherein each of the tensor engines further comprises a plurality types of hardware engines to accelerate the operations on data at each layer of the neural network, wherein the types of hardware engines include: one or more matrix multiplier engines each configured to perform a plurality of dense and/or sparse vector-matrix and matrix-matrix multiplication operations; one or more convolutional network engines each configured to perform a plurality of convolution operations by exploring sparsity of the vectors and/or matrices; one or more vector floating point units each configured to perform floating point vector operations of the operations; a data engine configured to prefetch the input data from the OSM and/or the external memory resources. 2. The processor of claim 1, wherein: The DLP is configured to multiplex the data prefetched from the OSM and/or the external memory resources to each of the tensors engines via a crossbar. 3. The processor of claim 2, wherein: each of the plurality of tensor engines further includes a programmable CPU having its own instruction RAM and data RAM configured to store instructions from a host and the retrieved data from the OSM and/or the external memory resources, respectively. 4. The processor of claim 3, wherein: the DLP is configured to accept a plurality of instructions from the host and submit the instructions to the tensor engines and their respective components in the DLP via a DLP interface, wherein the instructions are stored in the instruction RAM of the tensor engines. 5. The processor of claim 3, wherein: the DLP is also configured to provide the deep learning processing results by the DLP back to the host via the DLP interface. 6. The processor of claim 1, wherein: the configuration of the neural network is dynamically adjusted based on current deep learning application of the DLP. 7. The processor of claim 1, wherein: the neural network includes a plurality of layers each having a plurality of neurons connecting to neurons on a neighboring layer, wherein data processed progresses from one layer to the next in sequence along a processing pipeline. 8. The processor of claim 7, wherein: the DLP is configured to trim the neural network by pruning the neurons at each layer of the neural network as well as edges connecting the neurons of different layers to create a compact neural network while maintaining accuracy of the neural network to reduce size of the vectors and/or the matrices to be multiplied by the matrix multiplier engines and the data that needs to be read from the memory. 9. The processor of claim 7, wherein: the neural network utilized for convolution operations has three types of layers: one or more convolutional layers, each of which is configured to apply one or more local filters and/or a non-linear activation function to data from the input layer, one or more sub-sampling layers, each of which is configured to aggregate information amongst a set of neighbors of a neuron of the layer; one or more classification layers, each of which is configured to perform a linear or multi-layer perceptron (MLP) operation on the neural network and apply a non-linear activation function to output from the neuron. 10. The processor of claim 9, wherein: one or more kernels are applied to source pixels in an image for image classification, wherein a center element of each kernel is placed over a source pixel to replace the source pixel with a weighted sum of itself and its neighboring pixels. 11. The processor of claim 10, wherein: each kernel is a multi-dimensional matrix having its own values for elements in the matrix, wherein the dimensions represent (x, y, time) coordinates as well as depth of the elements of the kernel. 12. The processor of claim 1, wherein: the DLP is configured to partition each operation for pattern classification among the plurality of tensor engines, wherein each tensor engine is configured to perform a sub-task of the operation in parallel. 13. The processor of claim 12, wherein: the DLP is configured to replicate a sub-task among multiple tensor engines or move a sub-task from one tensor engine to another for efficient use of compute resources. 14. The processor of claim 1, wherein: each of the vector floating point units is a simplified arithmetic-logic unit (ALU) that handles on vector operations only and does not handle loops, branches, and branch predictions. 15. The processor of claim 1, wherein: each of the matrix multiplier engines is configured to perform one or more of: multiplication between a dense vector or matrix and a dense matrix, multiplication between a sparse vector and a dense matrix, and multiplication between a sparse vector and a sparse matrix, wherein a sparse vector or matrix has most of its elements as zero, while a dense vector or matrix has most of its elements as nonzero. 16. The processor of claim 1, wherein: each of the matrix multiplier engines is configured to minimize the number of times the input data and a weight matrix need to be read at each layer of the neural network and the number of times the output matrix needs to be written at each layer of the neural network. 17. The processor of claim 1, wherein: each of the matrix multiplier engines is configured to minimize data movement for multiplication between a sparse vector and a dense or sparse matrix, wherein only data that corresponds to non-zero values in the sparse vector is loaded into the memory of the tensor engine upon request. 18. The processor of claim 1, wherein: the tensor engine is configured to reuse data in the memory across one or more of the convolutional network engines efficiently to avoid data movement by unnecessary read and/or write to memory during the convolution operations. 19. The processor of claim 18, wherein: each of the convolutional network engines is configured to keep and repeatedly apply a same kernel on different parts of the input data at each layer of the neural network wherein the kernel is loaded into the memory only once during the convolution operations. 20. The processor of claim 18, wherein: each of the convolutional network engines is configured to apply different kernels to the same portion of the input data at each layer of the neural network, wherein that specific portion of the input data has already been loaded into the memory and does not need to be reloaded again during the convolution operations. 21. The processor of claim 18, wherein: each of the convolutional network engines is configured to save and reuse convolution output by a kernel on an overlapping part of two portions of the input data without calculating the output again at each convolution layer when the kernel is applied to the two portions of the data in stride and the two data portions overlap. 22. A method to support hardware-based deep learning processing, comprising: prefetching input data from an on-system memory (OSM) and/or a plurality of external memory resources via direct memory access (DMA); performing a plurality of operations on input data to generate deep learning processing results for pattern recognition and classification based on a neural network, wherein the convolution operations on data at each layer of the neural network are accelerated by a plurality of types of hardware engines to: perform a plurality of dense and/or sparse vector-matrix and matrix-matrix multiplication operations; perform a plurality of convolution operations by exploring sparsity of the vectors and/or matrices; perform floating point vector operations of the operations; outputting deep learning processing results for pattern recognition and classification to a host. 23. The method of claim 22, wherein: the neural network includes a plurality of layers each having a plurality of neurons connecting to neurons on a neighboring layer, wherein data processed progresses from one layer to the next in sequence along a processing pipeline. 24. The method of claim 23, further comprising: trimming the neural network by pruning the neurons at each layer of the neural network as well as edges connecting the neurons of different layers to create a compact neural network while maintaining accuracy of the neural network to reduce the size of the vectors and/or the matrices to be multiplied and the data that needs to be read from the memory. 25. The method of claim 22, further comprising: applying one or more local filters and/or a non-linear activation function to data from an input layer of the neutral network; aggregating information amongst a set of neighbors of a neuron of the layer; performing a linear or multi-layer perceptron (MLP) operation on the neural network and apply a non-linear activation function to output from the neuron. 26. The method of claim 25, further comprising: applying one or more kernels to source pixels in an image for image classification, wherein the center element of each kernel is placed over a source pixel to replace the source pixel with a weighted sum of itself and its neighboring pixels and each kernel is a multi-dimensional matrix having its own values for elements in the matrix, wherein the dimensions represent (x,y, time) coordinates as well as depth of the elements of the kernel. 27. The method of claim 22, further comprising: partitioning each operation for pattern classification among the plurality of hardware engines, wherein each hardware engine is configured to perform a sub-task of the operation in parallel. 28. The method of claim 27, further comprising: replicating a sub-task among multiple tensor engines or moving a sub-task from one tensor engine to another for efficient use of compute resources. 29. The method of claim 22, further comprising: performing one or more of: multiplication between a dense vector or matrix and a dense matrix, multiplication between a sparse vector and a dense matrix, and multiplication between a sparse vector and a sparse matrix, wherein a sparse vector or matrix has most of its elements as zero, while a dense vector or matrix has most of its elements as nonzero. 30. The method of claim 22, further comprising: minimizing the number of times the input data and a weight matrix need to be read at each layer of the neural network and number of times the output matrix needs to be written at each layer of the neural network. 31. The method of claim 22, further comprising: minimizing data movement for multiplication between a sparse vector and a dense or sparse matrix, wherein only data that corresponds to non-zero values in the sparse vector is loaded into the memory of the tensor engine upon request. 32. The method of claim 22, further comprising: reusing data in the memory across one or more of the convolutional network engines efficiently to avoid data movement by unnecessary read and/or write to memory during the convolution operations. 33. The method of claim 32, further comprising: keeping and repeatedly applying a same kernel on different parts of the input data at each layer of the neural network wherein the kernel is loaded into the memory only once during the convolution operations. 34. The method of claim 32, further comprising: applying different kernels to the same portion of the input data at each layer of the neural network, wherein that specific portion of the input data has already been loaded into the memory and does not need to be reloaded again during the convolution operations. 35. The method of claim 32, further comprising: saving and reusing convolution output by a kernel on an overlapping part of two portions of the input data without calculating the output again at each convolution layer when the kernel is applied to the two portions of the data in stride and the two data portions overlap.