Exploring a cost-effective and high-accuracy optical detection method is of great significance in promoting fruit quality evaluation and grading sales. Apples are one of the most widely economic fruits, and a qualitative and quantitative assessment of apple quality based on soluble solid content (SSC) was investigated via visible (Vis) spectroscopy in this study. Six pretreatment methods and principal component analysis (PCA) were utilized to enhance the collected spectra. The qualitative assessment of apple SSC was performed using a back-propagation neural network (BPNN) combined with second-order derivative (SD) and Savitzky–Golay (SG) smoothing. The SD-SG-PCA-BPNN model’s classification accuracy was 87.88%. To improve accuracy and convergence speed, a dynamic learning rate nonlinear decay (DLRND) strategy was coupled with the model. After that, particle swarm optimization (PSO) was employed to optimize the model. The classification accuracy was 100% for testing apples via the SD-SG-PCA-PSO-BPNN model combined with a Gaussian DLRND strategy. Then, quantitative assessments of apple SSC values were performed. The correlation coefficient (r) and root-square-mean error for prediction (RMSEP) in testing apples were 0.998 and 0.112 °Brix, surpassing a commercial fructose meter. The results demonstrate that Vis spectroscopy combined with the proposed synthetic model has significant value in qualitative and quantitative assessments of apple quality.