To predict oil and phenol concentrations in olive fruit, the combination of back propagation neural networks (BPNNs) and contact-less plant phenotyping techniques was employed to retrieve RGB image-based digital proxies of oil and phenol concentrations. Fruits of cultivars (×3) differing in ripening time were sampled (~10-day interval, ×2 years), pictured and analyzed for phenol and oil concentrations. Prior to this, fruit samples were pictured and images were segmented to extract the red (R), green (G), and blue (B) mean pixel values that were rearranged in 35 RGB-based colorimetric indexes. Three BPNNs were designed using as input variables (a) the original 35 RGB indexes, (b) the scores of principal components after a principal component analysis (PCA) pre-processing of those indexes, and (c) a reduced number (28) of the RGB indexes achieved after a sparse PCA. The results show that the predictions reached the highest meanR2values ranging from 0.87 to 0.95 (oil) and from 0.81 to 0.90 (phenols) across the BPNNs. In addition to theR2, other performance metrics were calculated (root mean squared error and mean absolute error) and combined into a general performance indicator (GPI). The resulting rank of the GPI suggests that a BPNN with a specific topology might be designed for cultivars grouped according to their ripening period. The present study documented that an RGB-based image phenotyping can effectively predict key quality traits in olive fruit supporting the developing olive sector within a digital agriculture domain.