Intelligent detection is vital for achieving the intelligent picking operation of daylily, but complex field environments pose challenges due to branch occlusion, overlapping plants, and uneven lighting. To address these challenges, this study selected an intelligent detection model based on YOLOv5s for daylily, the depth and width parameters of the YOLOv5s network were optimized, with Ghost, Transformer, and MobileNetv3 lightweight networks used to optimize the CSPDarknet backbone network of YOLOv5s, continuously improving the model’s performance. The experimental results show that the original YOLOv5s model increased mean average precision (mAP) by 49%, 44%, and 24.9% compared to YOLOv4, SSD, and Faster R-CNN models, optimizing the depth and width parameters of the network increased the mAP of the original YOLOv5s model by 7.7%, and the YOLOv5s model with Transformer as the backbone network increased the mAP by 0.2% and the inference speed by 69% compared to the model after network parameter optimization. The optimized YOLOv5s model provided precision, recall rate, mAP, and inference speed of 81.4%, 74.4%, 78.1%, and 93 frames per second (FPS), which can achieve accurate and fast detection of daylily in complex field environments. The research results can provide data and experimental references for developing intelligent picking equipment for daylily.