Aiming at the problems of long detection time and low detection accuracy in the existing coal gangue recognition, this paper proposes a method to collect the multispectral images of coal gangue using spectral technology and match with the improved YOLOv5s (You Only Look Once Version-5s) neural network model to apply it to coal gangue target recognition and detection, which can effectively reduce the detection time and improve the detection accuracy and recognition effect of coal gangue. In order to take the coverage area, center point distance and aspect ratio into account at the same time, the improved YOLOv5s neural network replaces the original GIou Loss loss function with CIou Loss loss function. At the same time, DIou NMS replaces the original NMS, which can effectively detect overlapping targets and small targets. In the experiment, 490 sets of multispectral data were obtained through the multispectral data acquisition system. Using the random forest algorithm and the correlation analysis of bands, the spectral images of the sixth, twelfth and eighteenth bands from twenty-five bands were selected to form a pseudo RGB image. A total of 974 original sample images of coal and gangue were obtained. Through two image noise reduction methods, namely, Gaussian filtering algorithm and non-local average noise reduction, 1948 images of coal gangue were obtained after preprocessing the dataset. This was divided into a training set and test set according to an 8:2 ratio and trained in the original YOLOv5s neural network, improved YOLOv5s neural network and SSD neural network. By identifying and detecting the three neural network models obtained after training, the results can be obtained, the loss value of the improved YOLOv5s neural network model is smaller than the original YOLOv5s neural network and SSD neural network, the recall rate is closer to 1 than the original YOLOv5s neural network and SSD neural network, the detection time is the shortest, the recall rate is 100% and the average detection accuracy of coal and gangue is the highest. The average precision of the training set is increased to 0.995, which shows that the improved YOLOv5s neural network has a better effect on the detection and recognition of coal gangue. The detection accuracy of the improved YOLOv5s neural network model test set is increased from 0.73 to 0.98, and all overlapping targets can also be accurately detected without false detection or missed detection. At the same time, the size of the improved YOLOv5s neural network model after training is reduced by 0.8 MB, which is conducive to hardware transplantation.