Spectral filters are an important part of a multispectral acquisition system, and the selection of suitable filters can improve the spectral recovery accuracy. In this paper, we propose an efficient human color vision-based method to recover spectral reflectance by the optimal filter selection. The original sensitivity curves of the filters are weighted using the LMS cone response function. The area enclosed by the weighted filter spectral sensitivity curves and the coordinate axis is calculated. The area is subtracted before weighting, and the three filters with the smallest reduction in the weighted area are used as the initial filters. The initial filters selected in this way are closest to the sensitivity function of the human visual system. After the three initial filters are combined with the remaining filters one by one, the filter sets are substituted into the spectral recovery model. The best filter sets under L-weighting, M-weighting, and S-weighting are selected according to the custom error score ranking. Finally, the optimal filter set is selected from the three optimal filter sets according to the custom error score ranking. The experimental results demonstrate that the proposed method outperforms existing methods in spectral and colorimetric accuracy, which also has good stability and robustness. This work will be useful for optimizing the spectral sensitivity of a multispectral acquisition system.