Deep learning has been widely used for plant disease recognition in smart agriculture and has proven to be a powerful tool for image classification and pattern recognition. However, it has limited interpretability for deep features. With the transfer of expert knowledge, handcrafted features provide a new way for personalized diagnosis of plant diseases. However, irrelevant and redundant features lead to high dimensionality. In this study, we proposed a swarm intelligence algorithm for feature selection [salp swarm algorithm for feature selection (SSAFS)] in image-based plant disease detection. SSAFS is employed to determine the ideal combination of handcrafted features to maximize classification success while minimizing the number of features. To verify the effectiveness of the developed SSAFS algorithm, we conducted experimental studies using SSAFS and 5 metaheuristic algorithms. Several evaluation metrics were used to evaluate and analyze the performance of these methods on 4 datasets from the UCI machine learning repository and 6 plant phenomics datasets from PlantVillage. Experimental results and statistical analyses validated the outstanding performance of SSAFS compared to existing state-of-the-art algorithms, confirming the superiority of SSAFS in exploring the feature space and identifying the most valuable features for diseased plant image classification. This computational tool will allow us to explore an optimal combination of handcrafted features to improve plant disease recognition accuracy and processing time.