It is important to detect and classify foreign fibers in cotton, especially white and transparent foreign fibers, to produce subsequent yarn and textile quality. There are some problems in the actual cotton foreign fiber removing process, such as some foreign fibers missing inspection, low recognition accuracy of small foreign fibers, and low detection speed. A polarization imaging device of cotton foreign fiber was constructed based on the difference in optical properties and polarization characteristics between cotton fibers. An object detection and classification algorithm based on an improved YOLOv5 was proposed to achieve small foreign fiber recognition and classification. The methods were as follows: (1) The lightweight network Shufflenetv2 with the Hard-Swish activation function was used as the backbone feature extraction network to improve the detection speed and reduce the model volume. (2) The PANet network connection of YOLOv5 was modified to obtain a fine-grained feature map to improve the detection accuracy for small targets. (3) A CA attention module was added to the YOLOv5 network to increase the weight of the useful features while suppressing the weight of invalid features to improve the detection accuracy of foreign fiber targets. Moreover, we conducted ablation experiments on the improved strategy. The model volume, mAP@0.5, mAP@0.5:0.95, and FPS of the improved YOLOv5 were up to 0.75 MB, 96.9%, 59.9%, and 385 f/s, respectively, compared to YOLOv5, and the improved YOLOv5 increased by 1.03%, 7.13%, and 126.47%, respectively, which proves that the method can be applied to the vision system of an actual production line for cotton foreign fiber detection.