} }}%
    \caption{Examples of explanations for deep models generated using LIME (a) and SHAP (b).}%
    \label{fig:lime-shap}%
\end{figure*}

\subsection{Contributions}
In our experiments, we show that all three of the selected neural models are capable of converging to very high (and, in some cases, close-to-perfect) accuracy in a relatively short amount of training epochs. This leads us to question the difficulty of the task at hand and the actual need for such powerful, yet black-box, methods. Unfortunately, as we will showcase, the explanation methods that were supposed to clarify the input-to-output mapping of the DL models by feature attribution seem to provide uninformative explanations by highlighting apparently random image patches.
 
As results suggest the task is somewhat easy, it is legitimate to try to tackle it with a simpler strategy. In particular, we propose to use one of the simplest and most transparent models (\textit{i.e.}, a decision tree trained on the average RGB color of each image) to achieve excellent accuracy while being able to provide informative and understandable explanations. To this aim, we map each leaf of our decision tree to a portion of the RGB color gamut space, with each portion being defined by the threshold dictated by the nodes on the path to the leaf.

Lastly, we perform a pilot user study involving a sample of the main stakeholders of the classifiers to verify their attitude towards the various models and their respective explanations.

In summary, our contributions are the following:
\begin{itemize}
    \item 
    We propose a thorough analysis of a selection of DL methods in terms of performance and explainability, utilizing relevant models that offer a wide panoramic of the task;
   
   
    \item We show that the same classification task can be solved effectively and efficiently by a much simpler and more transparent model, a decision tree, with minimum effort of feature engineering. More specifically, we constrain the number of features to the three channels in the RGB color space, which are easy to represent in the three-dimensional space;
    \item We conduct a user study to collect the preferences of stakeholders in regard to which explanations best suit their needs; 
    \item We release our code\footnote{Anonymous for peer review} and self-collected dataset\footnote{Anonymous for peer review} for reproducibility and possibly extension of our experiments.
\end{itemize}

\section{Related Work}

Our work relates to two main paths of research; (i) the optimization of fruit ripeness grading, and (ii) the advocating for more focus on explainability in the Artificial Intelligence (AI) community.

Grading the ripeness of the fruit is a long-studied problem for whom strategies based on statistics (\textit{e.g.}, \cite{Mendoza-2006-BananaRipening,Olarewaju-2016-NondestructiveAvocados}), traditional machine learning (\textit{e.g.}, \cite{Xueping-2020-BlueberrySeg,Septiarini-2020-OilPalmSeg}), and DL (\textit{e.g.}, \cite{Saranya-2022-BananaRipenessDL,Inkyu-2016-DeepFruits}) have been proposed. The top-performing methods are those based on DL, which, aside from reaching astonishing accuracy, do away with the complex and error-prone task of feature engineering. However, the literature lacks extensive comparisons among the three noted strategies. For more information on the fruit ripeness grading problem and solutions, 
a recent survey was authored by Rizzo et al. \cite{Rizzo-2022-Fruit}.  

A common problem associated with DL models is their inner opacity. In fact, providing meaningful explanations for a DL model's prediction is an arduous task. Much research has gone towards the extraction of explanations by
using, for instance, information from gradients \cite{Selvaraju-2016-GradCAM}, attention scores \cite{Bahdanau-2014-Attention2}, surrogate models \cite{Ribeiro-2018-WhyTrust}, and latent prototypes \cite{Chen-2018-DLIntepretableImage}. Some methods have been proposed with the promise of being model-agnostic, \textit{i.e.}, to explain the prediction of any classifier. Prominent examples are the aforementioned LIME and SHAP methods \cite{Ribeiro-2018-WhyTrust,Lundberg-2017-UnifiedInterPred}. However, the proposed explanations have been challenged (\textit{e.g.}, \cite{Adebayo-2018-SaliencySanity,Serrano-2019-IsAttInterpret,Garreau-2021-WhatDoesLIMESee,Nauta-2021-ExplainingPrototypes}) and proved to be unreliable in multiple scenarios. 

Despite this, the focus of much of recent research still seems to be on scraping a few decimals of task accuracy, while still not accounting for explainability. On this topic, works such as the one by Rudin \cite{Rudin-2019-StopExplaining} discuss the necessity of more carefully gauging the tasks being solved and, whenever possible (or necessary due to high stakes), using more transparent models rather than black box ones. In the same vein, we advocate for the use of simpler and more explainable models when the task at hand is particularly simple.


\section{Preliminaries}

\subsection{Task definition}
As mentioned, the specific task we deal with in this work is that of banana ripeness stage classification. From a more generic perspective, this can be interpreted as a multiclass image classification task; each image comprises a picture of banana bunches within a crate. There are four grades of increasing ripeness, with crates classified as 1 being less ripe while crates classified as 4 being riper. Such a classifier can be used in fruit wholesale markets to aid operators in the labeling process of large numbers of incoming crates. This helps deliver the best possible product to customers and avoid waste. Data and expert knowledge have been gathered from the wholesale market of the city of Treviso, Italy, which aims to propose the output of this project for practical use.

\subsection{Dataset}

To train our selected models to accomplish this task, we collected an \emph{ad-hoc} dataset. Pictures of crates filled with various bunches of bananas were acquired at a native resolution of 4160 x 3120 pixels, and then manually labeled by experts at the Treviso public fruit market. Pictures were taken with a CZUR Shine Ultra scanner, with an effort made to achieve consistent lighting. The dataset consists of a total of 887 acquisitions, split among classes in a fairly balanced way. We detail the class distribution in the supplementary material.


To conduct the experiments we repeatedly split the dataset into 5 folds, accounting for class balance. As per the standard cross-validation pattern, each iteration of the procedure sets one fold apart as testing data, while the rest of the dataset is used for training (80\% of the remaining 4 folds) and validation (20\% of the remaining 4 folds). To strengthen the findings, each 5-fold cross-validation is run using ten different random seeds. 

\subsection{Data Processing}
To obtain a reasonable inference time, each image was resized to 224 x 224 pixels. This resizing was also chosen as it is fairly standard in many pre-trained models, allowing us to easily make use of modern transfer learning approaches. The dataset was augmented with random transformations, including rotation, affine transforms, elastic morphology transforms, random location crop, gaussian blur, as well as the erasure of patches of the image and changes in perspective. The latter transformations were applied to account for different angles and accidental occlusions, which are likely to happen when non-expert users are taking pictures with a smartphone (which is one potential end use of this classifier). The dataset is augmented in an ``offline'' manner, \textit{i.e.}, generated once before all training procedures.
All original pictures are maintained, and roughly 50\% of the images go through transformations and are then added to each respective class. Exact parameters utilized in the augmentation process may be found in the supplementary material, which also details the statistics for the augmented dataset.



Manual visual inspection of the dataset reveals that pictures are noisy in that parts of the crate are being captured in the overall image (mostly crate boundaries, as well as, sometimes, its bottom). A way to circumvent this problem is to perform semantic segmentation of the images to filter out the background of banana bunches. Having no manually segmented images, an unsupervised approach was the only feasible way to achieve this. After trying several segmentation algorithms implemented in the popular Python image processing package scikit-image\footnote{https://scikit-image.org/docs/dev/api/skimage.segmentation.html}, we selected the SLIC algorithm \cite{SLIC} for this task. In our experiments, all methods benefit from the inclusion of segmentation as a pre-processing step. As such, we only report results on segmented images.

\subsection{Feature Extraction}
DL models have the notable ability to extract features automatically (in this case, through convolution- or attention-based operations). Conversely, traditional machine learning methods such as a decision tree require a set of hand-crafted features to operate correctly. In both cases, methods are applied to the pre-processed and augmented dataset.

The feature extraction