We investigate the numerical approximation of the semiconductor Boltzmann transport equation using an expansion of the distribution function in spherical harmonics. A complexity analysis shows that traditional implementations of higher order spherical harmonics expansions suffer from huge memory requirements, especially for two and three dimensional devices. To overcome these complexity limitations, a compressed matrix storage scheme using Kronecker products is proposed, which reduces the memory requirements for the storage of the system matrix significantly. Furthermore, the total memory requirements are asymptotically dominated only by the memory required for the unknowns. We discuss the increased importance of the selection of an appropriate linear solver and show that execution times for matrix-vector multiplications using the compressed matrix scheme are even smaller than those for an uncompressed system matrix. Numerical results demonstrate the applicability of our method and confirm our theoretical results.