# An Explainable Regression Framework for Predicting Remaining Useful Life of
Machines

Talhat Khan1, Kashif Ahmad2, Jebran Khan3, Imran Khan1, Nasir Ahmad 1

1 Department of Computer Systems Engineering, University of Engineering and
Technology, Peshawar, Pakistan.
<EMAIL_ADDRESS>2 Department of Computer Science, Munster
Technological University, Cork, Ireland<EMAIL_ADDRESS>3 Department of
Artificial Intelligence, AJOU University, Suwon, South Korea,
<EMAIL_ADDRESS>
###### Abstract

Prediction of a machine’s Remaining Useful Life (RUL) is one of the key tasks
in predictive maintenance. The task is treated as a regression problem where
Machine Learning (ML) algorithms are used to predict the RUL of machine
components. These ML algorithms are generally used as a black box with a total
focus on the performance without identifying the potential causes behind the
algorithms’ decisions and their working mechanism. We believe, the performance
(in terms of Mean Squared Error (MSE), etc.,) alone is not enough to build the
stakeholders’ trust in ML prediction rather more insights on the causes behind
the predictions are needed. To this aim, in this paper, we explore the
potential of Explainable AI (XAI) techniques by proposing an explainable
regression framework for the prediction of machines’ RUL. We also evaluate
several ML algorithms including classical and Neural Networks (NNs) based
solutions for the task. For the explanations, we rely on two model agnostic
XAI methods namely Local Interpretable Model-Agnostic Explanations (LIME) and
Shapley Additive Explanations (SHAP). We believe, this work will provide a
baseline for future research in the domain.

###### Index Terms:

Explainability, Interpretability, Predictive Maintenance, Regression,
Remaining Useful Life, LIME, SHAP.

##  I Introduction

In the modern world, the scope of industries has expanded a lot. These days
industries are generally equipped with a large number of modern machines
resulting in a significant increase in production. However, the performance of
these machines may degrade over time if proper care is not taken, thus, they
need a continuous monitoring and maintenance process. Maintenance of machines
in industries is a tedious and time-consuming process and generally needs to
take different factors into account. However, thanks to the recent advancement
in technology, industry 4.0 have opened new opportunities for predictive
maintenance [1].

Predictive maintenance is one of the key aspects of modern industries
especially after the fourth revolution of industry. It allows for monitoring,
analyzing, and determining the condition of machine components for early
detection of potential faults. The process generally involves data
acquisition, data processing, and making intelligent decisions on the basis of
the collected data to improve and optimize maintenance processes. Predictive
maintenance generally involves different tasks. One of these tasks is the
prediction of remaining useful life (RUL) of machines installed in a factory.

Thanks to the recent advancement in Machine Learning (ML) and sensor
technology, it is possible to automate the prediction of RUL of machines by
training ML algorithms on the data collected through a diversified set of
sensors installed in the machines. The literature already reports the
effectiveness of a wide range of ML algorithms in predictive maintenance in
general and in predicting the RUL of machines in particular. However, these ML
algorithms are used as a black box with total focus on their performance
(i.e., accuracy and Mean Squared Error, etc.,) without providing any insights
on the working mechanism and cause behind the decisions of these algorithms.
In such a critical application, accuracy/MSE alone is not enough to build the
stakeholders’ trust in ML prediction rather more insights on the causes behind
the predictions are needed [2].

In this paper, we propose an explainable ML framework incorporating a couple
of model agnostic explainable AI techniques for the prediction of the RUL of
machines. The proposed framework provides insights into the ML model’s
predictions allowing the stakeholders to analyze the main causes of machine
degradation. These insights not only build stockholders’ trust in the
framework but also allow them to tune the model for better predictive
performance.

The key contributions of the work can be summarized as follows:

  * •

We propose an explainable framework incorporating multiple regression and
explainability methods for predicting the RUL of machines.

  * •

We also evaluate several regression methods including multiple classical and
Neural Networks (NNs) based techniques. Moreover, we analyze the results of
two model agnostic methods for the explanation of our regression algorithms.

  * •

In extensive experimental setup, we evaluate the potential and applicability
of explainable AI methods in predictive maintenance.

The rest of the paper is organized as follows. Section II provides an overview
of the related work. Section III describes the methodology of the proposed
framework. Section IV provides a detailed description of the dataset,
experiments, and experimental results. Finally, Section V concludes the work.

##  II Related Work

In this section, we provide an overview of the existing literature on both
predictive maintenance and explainable AI. In the first part, we focus on the
literature on predictive maintenance by highlighting some recent works in the
domain. In the second part, we provide an overview of explainable AI
techniques and key applications where AI could be beneficial.

###  II-A Predictive Maintenance

The literature reports several interesting works on predictive maintenance,
where different aspects of predictive maintenance are explored [1, 3].
Predictive maintenance generally involves three activities namely (i) data
acquisition, (ii) data processing, and (iii) maintenance decision making, and
the research in the domain mainly focuses on these areas [4]. For data
acquisition, the predictive maintenance techniques heavily depend on sensor
technologies, which provide relevant and useful information on the machine
conditions. Based on the nature of sensors, predictive maintenance techniques
can be roughly divided into three categories including (i) existing sensor-
based maintenance, (ii) test-sensor-based maintenance, and (iii) test signal-
based maintenance techniques [5].

The literature also reports several interesting frameworks of intelligent data
processing and handling for predictive maintenance. For instance, Shcherbakov
et al. [6] provides an overview of several data processing techniques and
pipelines for data handling and processing for cyber-physical systems
maintenance. Similarly, Yan et al. [7] provide a detailed overview of the
challenges associated with heterogeneous industrial data handling and
processing for predictive maintenance.

Predictive decision-making is one of the most explored topics in predictive
maintenance. To this aim, several interesting solutions have been proposed
over the years. The majority of the initial efforts in this direction rely on
conventional/statistical ML algorithms, such as Random Forst, Support Vector
Machines (SVMs), and decision trees [3]. For instance, Kusiak et al. [8]
relied on two classical ML algorithms namely decision trees and SVMs, which
were trained on feature vectors composed of 60 sensor readings, for the
predictive maintenance of wind turbines. Other classical ML algorithms that
are widely used for predictive maintenance include RF and Naive Bayes. These
algorithms are normally trained raw sensor data or handcrafted features
depending on the nature of the data [3]. A vast majority of the literature
also relies on fuzzy logic and Hidden Markov Models (HMMs) for predictive
maintenance. For instance, Zaki et al. [9] and Omoregbee et al. [10] employed
Fuzzy logic and HMMs for predictive maintenance of renewable energy systems,
respectively. However, recently the trend shifted towards the use NNs, and the
majority of the recently proposed solutions rely on different types of NNs,
such as MLP, Convolutional Neural Networks (CNNs), and Long short-term memory
(LSTM) [3]. The choice of these algorithms mainly depends on the nature of the
data. For example, CNNs are mostly used for predictive maintenance using
visual content [11, 12]. LSTM-based solutions, on the other hand, are more
effective for the analysis of sequential/time series data [13, 14].

###  II-B Explainable AI

Over the last few years, explainable/interpretable AI got the attention of the
research community. The literature reports several studies where it is
demonstrated that in critical applications, such as healthcare, education,
defense, and transportation, predictive capabilities of ML algorithms alone
are not enough rather the algorithms should be interpretable [2, 15].
Explainability/interpretability, which aims at highlighting the causes behind
the AI models’ predictions, could be obtained either by developing explainable
AI algorithms or providing an explanation of the so-called black-box AI
algorithms [16]. However, there is a trade-off between accuracy/performance of
AI algorithms and interpretation [17]. Therefore, a majority of the
explainable AI frameworks rely on model agnostics methods for the
interpretation of AI models [2, 18]. To this aim, several interesting
techniques are proposed. Some most commonly used techniques include LIME [19],
SHAP [20], Grad-CAM [21], and DiCE [22].

Some key applications in which explainable AI has been widely explored include
healthcare [23], education [24], security [25], and other smart cities
applications [2]. The applications of explainable AI have been recently also
introduced in industry [26]. However, most of the literature aims to analyze
its applicability, challenges, and advantages in different industrial
applications [26]. For instance, Shukla et al. [27] analyzed the opportunities
of explainable AI in aerospace predictive maintenance. The authors also
provide a detailed overview of the challenges associated with predictive
maintenance in the domain. In contrast to most of the works reported in the
literature, in this work, we propose an explainable regression framework for
one of the most crucial applications of predictive maintenance namely the
prediction of the remaining useful life of a machine.

##  III Methodology

Figure 1 provides the block diagram of the proposed explainable predictive
maintenance framework. The framework is mainly composed of two components
namely (i) features selection and ML-based prediction, and (ii) explanation of
the predictions. In the first part, we rely on several ML algorithms for the
prediction of the remaining useful life of the machine components. In the
second part, two different algorithms are used for the explanation of the
model’s predictions. We note that the main contribution of the work lies in
the explanation part. In the next subsections, we provide a detailed
description of each of the phases.

Figure 1: Block diagram of the proposed methodology.

###  III-A ML-based Prediction

For the prediction of the useful remaining life of the machine components, we
rely on several algorithms including (i) Random Forest (RF), (ii) ElasticNet
with Generalized Linear Models (GLMs), (iii) Gradient Boosting, (iv) Support
Vector Machines (SVMs), and a (v) Neural Network (NNs) model. A description of
each of the methods is provided below.

  * •

RF-based Prediction: RF is one of the most widely used methods. In this work,
we use it as one of our baseline methods. In this approach, as a first step, a
shallow RF model is used to identify more important/influential features by
plotting a chart of feature ranking. After the plotting feature ranking, the
less important features (i.e., sensors values) are dropped. An RF model is
then trained on the selected features. Table I provides the values of
hyperparameters of the model used in the experiments.

  * •

SVM-based Prediction: Our second baseline method is based on SVMs. SVMs are
one of the most widely used algorithms for classification problems. However,
it is rarely used for regression. It follows the same rules and criteria for
regression tasks where the aim is to identify a function approximating the
mapping from the input to real numbers based on training samples. One of the
key processes in SVM-based prediction is the selection of hyperparameters
values. To this aim, we rely on a grid-search algorithm to find the best
combination of SVM hyperparameters.

  * •

Gardient Boosting-based Prediction: Our third method is based on Gradient
Boosting [28], which is also one of the most widely used algorithms for
classification and regression tasks. It is an ensemble method where multiple
learning algorithms, which are also called weak learners, are combined to
obtain better predictive performance. In our case, the weak learners are based
on decision trees. Similar to the other models, we rely on the grid-search
approach for the selection of hyperparameters of the model.

  * •

ElasticNetGLM-based Prediction: Elastic net regularization pairing with GLMs
is one of the widely used regularization methods. It allows to filter out
unimportant and highly correlated features and helps to improve the
performance of the model. In this work, we use an ML library namely Scikit-
learn for the implementation of the ElasticNetGLM model. For tuning the
hyperparameters of the model, we rely on the grid-search algorithm that allows
us to find the best combination of the hyperparameters. Table II provides the
summary of the parameters used in the model.

  * •

NNs-based Prediction: Based on the proven performances in other applications,
we also propose an NNs-based solution for the prediction of the useful
remaining life of the machine components. To this aim, we propose an MLP
regressor model. Our MLP regressor is composed of a total of 50 hidden layers,
which are trained using backpropagation without an activation function in the
output layer. Moreover, we used the square error as the loss function
resulting in continuous values as an output.

Table I: Parameters setting of the FR model. Attribute | Value
---|---
max-depth | 9
max-features | auto
min-samples-leaf | 10
min-samples-split | 2
n-estimators | 10
Table II: Parameters setting of the ElasticNetGLM model. Attribute | Value
---|---
Alpha | 0.01
l1-ratio | 0.01
copy-X | True
fit-intercept | True
selection | Cyclic
tol | True 0.0001
Table III: Parameters setting of the MLP model. Attribute | Value
---|---
Model | MLP Regressor
Hidden Layers | 50
Max Iteration | 1000
Learning Rate | Adaptive

###  III-B Model’s Explanation

For the models’ explanation, we mainly rely on two methods namely (i) LIME,
and (ii) SHAP. In the next subsections, we provide a detailed description of
each of the methods.

####  III-B1 LIME-Local Interpretable Model-Agnostic Explanations

LIME [19] is one of the most commonly used methods for the interpretation of
ML models’ predictions. One of the key advantages of LIME is that it is a
model agnostics method and could be used for the explanation/interpretation of
any model. For the explanation of an ML model, LIME perturbs the input samples
and analyzes the changes in the prediction of the model. This simple working
mechanism makes it a preferable choice for model interpretation compared to
model-specific methods, which require a deeper understanding of the underlying
models.

LIME provides local interpretation, which means the model’s behavior is
described by analyzing the response of a model to changes in a single data
sample. Here the intuition is to analyze causing behind a particular
prediction by answering questions like ”why was this prediction made?” or
”which features caused the prediction?”. It produces results in the form of a
list of explanations highlighting the contribution of the individual feature
as detailed in Section IV. We note that the idea and working mechanism of LIME
is different from a related concept of ”feature importance”, which is
generally conducted over the entire datasets.

LIME could be used for the explanations of models deployed in different
application domains including textual, tabular (i.e., sensor data), and visual
content. The literature already reports the effectiveness of the method in
several interesting human-centric applications, such as healthcare and other
smart cities applications [2].

####  III-B2 SHAPE-Shapley Additive Explanations

Our second explanation approach is based on another state-of-the-art technique
namely SHAPE. The method was introduced by Lundberg et al. [20] to cope with
the limitations of the existing methods. Similar to LIME, SHAP provides
explanations of individual predictions and could be used for the explanation
of any model. However, in contrast to LIME, SHAP provides both local and
global explanations. The main difference between global and local explanations
lies in the level/scope of explanation. The global
interpretations/explanations include complete insights into the
general/overall behavior of the model. The local explanations/interpretations
describe the causes behind a decision on an individual data sample.

Similar to LIME, SHAP could be used for the explanation of ML models trained
on different types of data including textual, visual, and tabular data. The
literature reports the effectiveness of the method in several application
domains, such as healthcare, defense, agriculture, etc [2, 29, 30].

##  IV Experiments and Results

###  IV-A Dataset

For the evaluation of proposed solutions, we used a dataset composed of the
engine degradation simulation (C-MAPSS) data, which is collected in a
simulated engine degradation environment under different combinations of
operational conditions and modes [31]. The dataset is provided by the
Prognostics Center of Excellence (PCoE) where the data is based on time series
ranging from the working state to the failure state of the components. The
dataset provides a 24 features vector containing 21 sensor readings and 3
operational settings. The sensor readings are mostly related to temperature,
pressure, the fan speed of an engine, fuel, etc. The description of all of
these sensor readings is provided in Table IV.

The dataset is widely used for fault detection and prognostics (i.e.,
predicting the time at which the machine components will no longer work). In
this paper, we are interested in predicting the Remaining Useful Life (RUL) of
machine components, which is a continuous target/value. We treat the problem
as a regression task. The dataset is composed of more than 20,000 data
samples, which are divided into training and test sets. The training set is
composed of 16504 samples while the test set contains 4127 samples.

Table IV: Description of the sensor readings provided in the dataset.

Sensor No | Readings | Sensor No | Readings
---|---|---|---
1 | Total temperature at fan inlet | 2 | Total temperature at LPC outlet
3 | Total temperature at HPC outlet | 4 | Total temperature at LPT outlet
5 | Pressure at fan inlet | 6 | Total pressure in bypass-duct
7 | Total pressure at HPC outlet | 8 | Physical fan speed
9 | Physical core speed | 10 | Engine pressure ratio
11 | Engine pressure ratio | 12 | Ratio of fuel flow to Ps30
13 | Corrected fan speed | 14 | Corrected core speed
15 | Bypass Ratio | 16 | Burner fuel-air ratio
17 | Bleed Enthalpy | 18 | Demanded fan speed
19 | Demanded corrected fan speed | 20 | HPT coolant bleed
21 | LPT coolant bleed | - | -

###  IV-B Experimental Results

In this section, we provide the experimental results of the proposed work.
Firstly, we report the results of all the algorithms employed in this work
followed by some samples of the explanations produced by LIME and SHAP.

####  IV-B1 Prediction Results

Table V provides the experimental results of all the algorithms employed in
this work including the classical and NNs-based solutions in terms of Mean
Squared Error (MSE) and Mean Absolute Error (MAE) on a test set containing a
total of 4127 samples. We note that there is a trade-off between performance
and explainability. The classical ML algorithms are more explainable compared
to NNs, however, their performance is generally on the lower side. Similar
trends have been also observed in this work. As can be seen, the MSE and MAE
values for the NNs-based solution are significantly lower compared to the
classical algorithms, such as RF and SVMs.

As far as the explanations of the NNs-based solutions is concerned, this could
be overcome with the use of model agnostic methods of explainability. Next, we
analyze the explanations provided by two model agnostic explainable AI
methods.

Table V: Experimental results in terms of Mean Squared Error (MSE) and Mean Absolute Error (MAE) on a test set containing 4127 samples. Method | MSE | MAE
---|---|---
RF | 1767.06 | 29.84
ElasticNetGLM | 2043.03 | 34.60
Gradient Boosting | 1768.45 | 29.92
SVMs | 2043.03 | 34.60
MLP | 1742.08 | 28.46

####  IV-B2 Model Explanations via LIME and SHAP

Figure 2 and Figure 3 provides explanations generated by LIME and SHAP method,
respectively. In this sample, the actual value of RUL of the machine component
is 151 cycles while the predicted value is 148.35 cycles.

In Figure 2, the features (i.e., sensor values) on the right side in red color
are the ones that contribute to increasing the prediction values while the
ones in blue color are the features that have a negative effect or decrease
the predicted value. For example, the values of the sensors 11, 7, 2, etc. on
the right side indicate that the machine component is in a good condition and
its RUL is supposed to be high. The sensor value on the left side in blue
color, for example, $sensor-1=<518.67$ indicates that the conditions of the
component are not good and it tries to reduce the predicted RUL value.

Figure 2: Sample explanations provided by LIME. Here the actual value of RUL
of the component is 151 while the predicted value is 148.3.

Figure 3 provides the explanation of the same sample. Similar to LIME, SHAP
explanations are also composed of several values including:

  * •

The predicted value (i.e., 148.35)

  * •

The base value, which is the average of the model output over the training
dataset.

  * •

The feature values that contributed to the prediction. The red values are the
feature values that increased the predicted value while the values in blue
color (i.e., $sensor-14=8.151$ contributed to the reduction of the predicted
RUL value. In other words, the sensor values in red color mean the condition
of the component is good while the blue ones indicate something is wrong with
the component.

  * •

The size/length of the arrow shows the impact of the feature on the
prediction. For example, in the given sample, sensor-11, sensor-12, and
sensor-14 have a higher impact on the prediction.

Figure 3: Sample explanations provided by SHAP. Here the actual value of RUL
of the component is 151 while the predicted value is 148.3.

##  V Conclusions and Future Work

In this work, we proposed an explainable regression framework for the
prediction of the RUL of machine components. We evaluated several ML
techniques including classical ML and NNs approaches for the prediction. For
the explanation of the models, we employed two different models agnostic
explainable AI methods. The explanation provided by these methods is very
insightful that could help the stakeholders in making correct decisions in
such critical applications. The explanation of the models could also help the
developers to rectify the limitations of their proposed solutions. In the
future, we aim to further extend the scope of the work by tackling more
relevant tasks of predictive maintenance.

## References

  * [1] S. K. Jagatheesaperumal, M. Rahouti, K. Ahmad, A. Al-Fuqaha, and M. Guizani, “The duo of artificial intelligence and big data for industry 4.0: Applications, techniques, challenges, and future research directions,” _IEEE Internet of Things Journal_ , 2021.
  * [2] K. Ahmad, M. Maabreh, M. Ghaly, K. Khan, J. Qadir, and A. Al-Fuqaha, “Developing future human-centered smart cities: Critical analysis of smart city security, data management, and ethical challenges,” _Computer Science Review_ , vol. 43, p. 100452, 2022.
  * [3] Y. S. Afridi, K. Ahmad, and L. Hassan, “Artificial intelligence based prognostic maintenance of renewable energy systems: A review of techniques, challenges, and future research directions,” _International Journal of Energy Research_ , 2021.
  * [4] S. Selcuk, “Predictive maintenance, its implementation and latest trends,” _Proceedings of the Institution of Mechanical Engineers, Part B: Journal of Engineering Manufacture_ , vol. 231, no. 9, pp. 1670–1679, 2017.
  * [5] T. Zonta, C. A. da Costa, R. da Rosa Righi, M. J. de Lima, E. S. da Trindade, and G. P. Li, “Predictive maintenance in the industry 4.0: A systematic literature review,” _Computers & Industrial Engineering_, vol. 150, p. 106889, 2020.
  * [6] M. V. Shcherbakov, A. V. Glotov, and S. V. Cheremisinov, “Proactive and predictive maintenance of cyber-physical systems,” in _Cyber-Physical Systems: Advances in Design & Modelling_. Springer, 2020, pp. 263–278.
  * [7] J. Yan, Y. Meng, L. Lu, and L. Li, “Industrial big data in an industry 4.0 environment: Challenges, schemes, and applications for predictive maintenance,” _IEEE Access_ , vol. 5, pp. 23 484–23 491, 2017.
  * [8] A. Kusiak and W. Li, “The prediction and diagnosis of wind turbine faults,” _Renewable energy_ , vol. 36, no. 1, pp. 16–23, 2011.
  * [9] S. A. Zaki, H. Zhu, and J. Yao, “Fault detection and diagnosis of photovoltaic system using fuzzy logic control,” in _E3S Web of Conferences_ , vol. 107\. EDP Sciences, 2019, p. 02001.
  * [10] H. O. Omoregbee and P. S. Heyns, “Fault detection in roller bearing operating at low speed and varying loads using bayesian robust new hidden markov model,” _Journal of Mechanical Science and Technology_ , vol. 32, no. 9, pp. 4025–4036, 2018.
  * [11] W. Shin, J. Han, and W. Rhee, “Ai-assistance for predictive maintenance of renewable energy systems,” _Energy_ , vol. 221, p. 119775, 2021.
  * [12] M. Ulmer, E. Jarlskog, G. Pizza, J. Manninen, and L. Goren Huber, “Early fault detection based on wind turbine scada data using convolutional neural networks,” in _5th European Conference of the Prognostics and Health Management Society, Virtual Conference, 27-31 July 2020_ , vol. 5, no. 1. PHM Society, 2020.
  * [13] F. Zou, H. Zhang, S. Sang, X. Li, W. He, and X. Liu, “Bearing fault diagnosis based on combined multi-scale weighted entropy morphological filtering and bi-lstm,” _Applied Intelligence_ , vol. 51, no. 10, pp. 6647–6664, 2021\.
  * [14] L. Xiang, P. Wang, X. Yang, A. Hu, and H. Su, “Fault detection of wind turbine based on scada data analysis using cnn and lstm with attention mechanism,” _Measurement_ , vol. 175, p. 109094, 2021.
  * [15] I. Khan, K. Ahmad, N. Gul, T. Khan, N. Ahmad, and A. Al-Fuqaha, “Explainable event recognition,” _arXiv preprint arXiv:2110.00755_ , 2021.
  * [16] A. B. Arrieta, N. Díaz-Rodríguez, J. Del Ser, A. Bennetot, S. Tabik, A. Barbado, S. García, S. Gil-López, D. Molina, R. Benjamins _et al._ , “Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai,” _Information fusion_ , vol. 58, pp. 82–115, 2020.
  * [17] I. Kamwa, S. Samantaray, and G. Joós, “On the accuracy versus transparency trade-off of data-mining models for fast-response pmu-based catastrophe predictors,” _IEEE Transactions on Smart Grid_ , vol. 3, no. 1, pp. 152–161, 2011.
  * [18] A. Bennetot, I. Donadello, A. E. Qadi, M. Dragoni, T. Frossard, B. Wagner, A. Saranti, S. Tulli, M. Trocan, R. Chatila _et al._ , “A practical tutorial on explainable ai techniques,” _arXiv preprint arXiv:2111.14260_ , 2021.
  * [19] M. T. Ribeiro, S. Singh, and C. Guestrin, “” why should i trust you?” explaining the predictions of any classifier,” in _Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining_ , 2016, pp. 1135–1144.
  * [20] S. M. Lundberg and S.-I. Lee, “A unified approach to interpreting model predictions,” _Advances in neural information processing systems_ , vol. 30, 2017.
  * [21] R. R. Selvaraju, M. Cogswell, A. Das, R. Vedantam, D. Parikh, and D. Batra, “Grad-cam: Visual explanations from deep networks via gradient-based localization,” in _Proceedings of the IEEE international conference on computer vision_ , 2017, pp. 618–626.
  * [22] R. K. Mothilal, A. Sharma, and C. Tan, “Explaining machine learning classifiers through diverse counterfactual explanations,” in _Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency_ , 2020, pp. 607–617.
  * [23] A. Adadi and M. Berrada, “Explainable ai for healthcare: from black box to interpretable models,” in _Embedded Systems and Artificial Intelligence_. Springer, 2020, pp. 327–337.
  * [24] E. Tjoa and C. Guan, “A survey on explainable artificial intelligence (xai): Toward medical xai,” _IEEE transactions on neural networks and learning systems_ , vol. 32, no. 11, pp. 4793–4813, 2020.
  * [25] L. Vigano and D. Magazzeni, “Explainable security,” in _2020 IEEE European Symposium on Security and Privacy Workshops (EuroS &PW)_. IEEE, 2020, pp. 293–300.
  * [26] K. Gade, S. C. Geyik, K. Kenthapadi, V. Mithal, and A. Taly, “Explainable ai in industry,” in _Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining_, 2019, pp. 3203–3204.
  * [27] B. Shukla, I.-S. Fan, and I. Jennions, “Opportunities for explainable artificial intelligence in aerospace predictive maintenance,” in _PHM Society European Conference_ , vol. 5, no. 1, 2020, pp. 11–11.
  * [28] T. Hastie, R. Tibshirani, J. H. Friedman, and J. H. Friedman, _The elements of statistical learning: data mining, inference, and prediction_. Springer, 2009, vol. 2.
  * [29] Y. Zhou, W. Wu, H. Wang, X. Zhang, C. Yang, and H. Liu, “Identification of soil texture classes under vegetation cover based on sentinel-2 data with svm and shap techniques,” _IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing_ , 2022.
  * [30] R. ElShawi, Y. Sherif, M. Al-Mallah, and S. Sakr, “Interpretability in healthcare: A comparative study of local machine learning interpretability techniques,” _Computational Intelligence_ , vol. 37, no. 4, pp. 1633–1650, 2021.
  * [31] A. Saxena, K. Goebel, D. Simon, and N. Eklund, “Damage propagation modeling for aircraft engine run-to-failure simulation,” in _2008 international conference on prognostics and health management_. IEEE, 2008, pp. 1–9.


# Do Autonomous Agents Benefit from Hearing?

###### Abstract

Mapping states to actions in deep reinforcement learning is mainly based on
visual information. The commonly used approach for dealing with visual
information is to extract pixels from images and use them as state
representation for reinforcement learning agent. But, any vision only agent is
handicapped by not being able to sense audible cues. Using hearing, animals
are able to sense targets that are outside of their visual range. In this
work, we propose the use of audio as complementary information to visual only
in state representation. We assess the impact of such multi-modal setup in
reach-the-goal tasks in ViZDoom environment. Results show that the agent
improves its behaviour when visual information is accompanied with audio
features.

Index Terms: pitch, pixels, raw audio, reinforcement learning, speech features

##  1 Introduction

The recent advances in deep learning makes it possible to extract high-level
features from raw sensory data that lead to breakthroughs in computer vision
[1] and speech recognition [2]. Recently, Convolutional Neural Networks (CNNs)
have shown a remarkable progress in visual recognition tasks [3] and audio
tagging [4]. They have also successfully been applied for feature extraction
and classification tasks in the audio domain [5, 6].

The design and training of an agent that interacts with an environment and
solves a given problem has been studied for years using different methods
including reinforcement learning [7]. Reinforcement learning studies problems
and algorithms that learn policies so that the decisions maximize long term
reward from the environment [7, 8]. Reinforcement learning has been
successfully used in different tasks such as object recognition [9] and
solving physics-based control problems [10]. The recent advances in deep
reinforcement learning have allowed autonomous agents to perform well on
different games using only visual information [11]. It has also been applied
successfully in different applications including Atari [12], robotic arms [13]
and self-driving vehicles [14]. However, only visual information is used as
state representation in all of these mentioned and other related works [12].

Learning using only visual information may not always be easy for the learning
agent. For example, it is difficult for the agent to reach the target using
only visual information in scenarios where there are many rooms and there is
no direct line of sight between the agent and the target. Without recurrent
neural networks or other architectures that allow memorizing the past, the
agent can’t recall which rooms it has already searched.111While the task of
this work could be solved with recurrent neural networks, this work focuses on
the benefits from using audio information rather than solving the task.. We
see example of searching task in Fig. 1 where an agent with only visual
information is not able to systematically search the goal. Thus, the use of
audio features could provide valuable information for such problems. Hence, in
this work, we propose the use of audio as complementary input information to
visuals. The work assesses the impact of audio features using different
environment setups. The visual and audio information are used together to find
the best action for a given state.

This work was motivated by the natural tendency of guiding with spoken
language when assisting someone to try to find an object. This lead us to a
question ”Why hearing is not used for the autonomous agents? One way to give
agent a hearing would be utilizing an automatic speech recognition component.
However, using an automatic speech recognition component may not be ideal as
it would not necessarily detect audio signals other than speech, e.g.
whistling. Hence, we study about training the agent with direct access to raw
audio.

Figure 1:  The agent (i.e., green) searching for the goal (i.e., red). By
providing agent with sound from the goal, agent was able to reach the goal
more reliably when trained with reinforcement learning.

The following sets of experiments have been carried out on VizDoom environment
[15]: using only visuals, and using visuals together with audio features. We
also experiment the impact of using audio information occasionally and all
time. Simulation results on VizDoom environment manifest that a Deep Q-Network
trained using visual and audio information provides better average reward
compared to using only visual.

##  2 Deep Q-Network

Deep Q-networks (DQN) were proposed by [12], making it possible to use
Q-learning [16] in complex environments like video games. DQN uses deep
learning to estimate the value function rather than a discrete table, and uses
two techniques to make learning stable: replay memory and target networks. DQN
has been successfully applied to play a set of Atari games [12], and Doom
[15]. Because of these reasons and its simplicity, DQN is used in our
experiments.

Reinforcement learning [7] methods aim to solve a task in an environment by
gathering experiences from the environment and learning from them. The
environment is modelled as a Markov Decision Process which consists of a set
of possible states $s\in S$, a set of possible actions $a\in A$, a reward
function $R(s_{t},a_{t},t_{t+1})\in\mathbb{R}$ and transition distribution
between the states $P(s_{t+1}|s_{t},a_{t})\in[0,1]$. Values at different time
steps are distinguished with $t\in\mathbb{N}$. In each state, a policy
$\pi:s\rightarrow a$ selects an action $a_{t}$, after which the environment
will move to next state $s_{t+1}\sim P(s_{t+1}|s_{t},a_{t})$. The policy
(agent) then receives a reward $r_{t}=R(s_{t},a_{t},s_{t+1})$ based on the
experience. For compactness, we focus on episodic games, where the game ends
once agent reaches a terminal state.

The goal of reinforcement learning is to learn a policy that maximizes the
return from any time step $t\in\mathbb{N}$ till end of the episode
$T\in\mathbb{N}$

$G_{t}=\sum^{T}_{k=0}\gamma^{k}r_{t+k},$ (1)

where discount factor $\gamma\in(0,1]$ is used to stabilize the learning
and/or weight the importance of future states vs. nearby states.

Q-learning [16] solves this by learning state-action value function $Q:s\times
a\rightarrow G$, which represents the expected return from taking an action
$a_{t}$ in state $s_{t}$. If we knew the true state-action values, selecting
action with highest value $a_{t}=\arg\max_{a}Q(s_{t},a)$ equals to optimal
policy [16]. Q-learning learns this function by creating a table of all
possible state-action pairs, and updating the value on each experience with

$\displaystyle Q(s_{t},a_{t})$ $\displaystyle=$ (2) $\displaystyle
Q(s_{t},a_{t})+\alpha\left(r_{t}+\gamma\max_{a}Q(s_{t+1},a)-Q(s_{t},a_{t})\right)$

where learning rate $\alpha\in\mathbb{R}$ controls rate of learning. DQN
extends this by using a deep neural network rather than a table to estimate
the $Q$-function. This allows the use of Q-learning with high-dimensional
states and actions (e.g. video games, where states are pixel of an image). The
update rule of DQN (Eq. 2) is modified for training neural networks. To
stabilize learning, DQN uses a replay memory and target networks [12].

To encourage trying out different solutions to the task, we use
$\epsilon$-greedy exploration strategy, where the DQN selects random action
with probability $\epsilon$ and optimal action with probability $1-\epsilon$.
This value $\epsilon$ starts high at the beginning of the training, and slowly
anneals towards zero over training to encourage using more optimal actions
instead of always taking random actions (exploitation).

This work uses DQN agent as the basis for all experiments because of its
flexibility to different types of data (i.e image pixels and audio features).

##  3 Multimodal Reinforcement Learning Agent

This work proposes the use of audio information along with the visual
information. For the experiments, we use two different type of features based
on the audio information: Pitch of the audio and raw samples. As it is
displayed in Fig. 2, the autonomous agent is trained using two sources of
input (i.e., pixels and audio features).

###  3.1 Pitch

Pitch is the variations in fundamental frequency which serves as an important
acoustic cue for tone, lexical stress, and intonation [17]. The pitch values
are extracted from the audio clip and fed into the CNN architecture.

###  3.2 Raw Audio

Raw audio features are the samples of the signal itself, normalized
appropriately for the neural network for processing. Using samples directly
removes need for feature extraction, but may be too complex data for learning
algorithm to use appropriately.

###  3.3 Proposed Reinforcement Learning System Architecture

Our proposed system uses both visual and audio features to train the agent.
Both input sources have their own convolutional layers. As it is displayed in
Fig. 2, the learned high-level features are combined in the final layer of the
network by concatenating the outputs of the two separate CNN heads.

Figure 2:  Proposed reinforcement learning architecture.

In the experiments of this work, we use raw audio as audio for the agent.
Specifically, we encode information about the environment (distance to the
goal) into the pitch of the played speech sample. Then, the sample is provided
to the agent along with the image. Note that the pitch of the audio sample is
updated every time step.

When using pitch features, we extract pitch for the sample with windowing of
size $30$ms and step size of $10$ms, resulting into a vector of length $114$
elements for the neural network. Since distance to the goal is encoded in the
overall pitch of the speech sample, these features could be easily digested
for useful information for the agent (higher pitch equals closer to target).
These features work as a sanity check that providing information on distance
to goal is beneficial to the agent.

For raw audio, we select $100$ speech samples for the network. We limit the
number of samples to $100$ to reduce computational costs. Unlike with pitch
features, agent now has to learn to process the frequency of signal provided
to obtain the encoded information.

##  4 Experimental Setup

We use VizDoom [15] as the environment in this work because of its
flexibility, easiness to use, and efficient 3D platform. It allows developing
artificial intelligence bots that play DOOM [18] using visual information. The
experiments have been carried out on machine with Intel Xeon processor, Nvidia
RTX 2080ti, CUDA 10.0, Python 3.6, TensorFlow 1.12 and Ubuntu LTS 18.04.

###  4.1 Scenario

VizDoom allows different scenarios including different maps and sets the
available buttons for the agent. Hence, we created our own scenario map (see
Fig. 3) to assess the performance of the proposed work. In the scenario map,
the agent’s task is to navigate in the rooms and reach to the target goal. The
actions of the agent are: turn left, turn right, move forward and move
backward.

The episode ends when agent reaches the goal or on timeout. The agent gets a
reward of $1$ if it reaches the goal, otherwise the agent receives $-1$ reward
on every game tick. We want to emphasize that this environment could be solved
with recurrent neural networks like the one used in [19]. However, the goal of
this work is not to solve the task, rather it is to study if audio information
can help an agent to complete the task.

Figure 3: Scenario map and image of agent’s perspective. The goal, a red
pilar, can be seen hiding behind the corner on the image on right.

###  4.2 Training

The training process starts with initialization parameters such as start
$\epsilon$, end $\epsilon$, discount factor and learning rate. The learning
process is run for multiple training epochs where each epoch contains many
learning steps. During the learning process, the value of $\epsilon$ is
linearly decreased. Initially, the agent makes completely random actions
(i.e., exploration). Afterwards, the actions are selected based on policy
(i.e., exploitation). The last training model is used to evaluate the
performance of the system.

###  4.3 Hyperparameters

The convolutional neural network model is trained using RMSProp algorithm and
minibatches of size 64. The most recent $10,000$ timesteps have been stored in
the replay memory. While the discount factor is set to $0.99$, the learning
rate is fixed to $0.00025$. The baseline and proposed systems are trained for
$600,000$ training steps. In the experiments where goal only spawns in the big
room , the timeout was set to $1000$. In the experiment with all five rooms,
timeout was set to $2100$.

Since the VizDoom environment provides a relatively high resolution image
(640x480), the resolution is downsampled to $40\times 40$ at each frame step.
This reduces the computational time of training the neural network models. We
have used the three channels of the image. For the audio features, we feed
vectors of size $1\times 114$ and $1\times 100$ for the pitch and raw audio
samples, respectively.

The same CNN architectures are used both for the image and audio features.
While the first layer is a convolutional layer with 8 filters of size $3\times
3$ and a stride of $2$, the second layer is a convolutional layer with $16$
filters of size $3\times 3$ and stride of $2$. The next fully connected layer
transforms the input to $128$ units which are then transformed by another
fully connected layer to an output size equal to the $4$ number of actions in
each game.

An $\epsilon$ greedy policy is applied both during training and testing.
During the training of DQN, $\epsilon$ starts initially with $1.0$ and
gradually anneals to $0.1$ after a number of steps. During the initial
training, the possible actions are uniformly selected but as the training
progresses, the optimal action is selected more frequently. This allows
maximum exploration at the beginning which eventually switch to exploitation.
During testing, the value of $\epsilon$ is fixed to $0.05$.

We use frame-skip of $10$ frames, i.e. the agent is given a chance to select
action only every 10th game frame (corresponds to $285$ms of in-game time, as
runs at $35$ frames-per-second). The previously chosen action is repeated over
all frames between these agent steps.

###  4.4 Performance Metric

To assess the performance of reinforcement learning, it is common to plot the
time versus reward and visualize how the reward increases/decreases during
training [19].

In this work, we have used the average reward per episode to evaluate the
performance of the baseline (no audio) and proposed work. Since the average
rewards of individual runs vary considerably, it is common to launch multiple
runs and plot the time versus average of these runs. Hence, the reported
results in the following subsection are run and averaged over five experiment
runs, i.e., each point in the learning curve figures is averaged over five
episodes.

##  5 Experimental Results

The experiments are carried out using different types of setups: the target is
spawned randomly in the big room and the target is spawned randomly in any of
the five rooms (see Fig.3). The agent always starts in the center of the big
room facing to random direction.

For the experiment that spawns the target randomly in the big room, we have
observed that using visual information both with pitch and raw audio provides
better average reward per episode compared to using only the visual. While the
use of only visual yields average reward of $-150$ over the last $100$
episodes, the augmentation of audio features to the visual yields average
reward of $-100$ over the last same $100$ episodes.

We have also carried out other experiments where the target is placed randomly
in any of the five rooms. From the learning curves of Fig. 4, we can see that
the agent does not reach the target most of the time using only visuals. But,
the use of audio features together with the visual one makes the agent to
reach the goal most of the time. The figure exhibits that while using visual
and audio features provides $-750$ average reward during the last episodes,
the use of only visual gives $-1500$ average reward in same episodes. Fig. 4
shows that the average learning curves of using visual together with audio
features provide better mean rewards compared to using only the visual. We
have also assessed the individual learning curves of these rewards. The
individual learning curves also display that the addition of audio features to
the visual ones yield better learning curve compared to the one based only on
visuals.

Table 1 shows the test results of $100$ episodes using visual and visual with
audios. The test experiment is carried out using the last training model. The
table compares the mean reward and number of steps required to reach the
target using visual and audio features for the two environment setups (i.e.,
target randomly spawned in big room and target randomly spawned in any of the
five rooms).

From the table, we can observe that although the use of only visual enables
the agent to reach the target almost all the time for the experiment that
spawns the target in the big room, the addition of complementary audio
features to the visual one provides lower number of steps to reach the target
compared to using visual.

But, when we compare the average success rate and required number of steps for
the experiment that spawns the target in any of the five rooms, we observe a
significant improvement both in success rate and required number of steps. The
use of only visual provides average success rate of $43$%. But, the
augmentation of visual with raw audio, and visual with pitch provides average
success rates of $87$% and $86$%, respectively. Similarly, the average
required number of steps to reach the target using only visual information is
$1420$. But, the addition of complementary raw audio and pitch to the visual
reduces the number of steps to $751$ and $614$, respectively. From the results
of the table, we can see that it is difficult for the agent to make
intelligent moves to reach to the target using only visual information in
difficult scenarios.

Figure 4:  Mean reward of using only pixels, pixels with raw audio and pixels with pitch. The target is spawned randomly in any of the five rooms for each episode. The mean reward is computed every $2000$ steps.  Table 1: Average success rate and average number of steps of the test experiment. The test experiment is carried out on $100$ game episodes using the last training model. | Target | Target | Target
---|---|---|---
| randomly | fixed in | randomly
| in big room | big room | in all rooms
| success/ | success/ | success/
| steps | steps | steps
Visual | 99/132 | 99/164 | 38/1502
Visual + Raw | 99/98 | 99/138 | 91/588
Visual + Pitch | 99/95 | 99/98 | 93/514

In the results of Fig. 4, the audio features are always used together with the
visual one to get the best action. We have also carried out other experiments
where the visual is used all the time but the audio features are occasionally
used. Instead of receiving audio at every time step, the agent gets audio in
addition to image with certain probability. We test with probability values of
[20%, 50%] and compare these results. Figure 5 displays the results of these
experiments. We see that the use of visual and occasional use of audio
features gives better average reward compared to using only the visuals. But,
its reward is not as good as the one that uses audio information all the time.
After conducting the experiments, we noticed that the goal spawned inside the
wall (i.e., it is unreachable) $2-3\%$ of the time. However, this affects to
all of the experiment and the comparison is possible.

Figure 5:  Mean reward of using only pixels, pixels with raw audio and pixels
with pitch. The audio features are occasionally used with 20% and 50%
probability chances. The mean reward is computed every $2000$ steps.

The results of Fig. 4 and Fig. 5 show that providing audio features to the
visual all the time provides better average reward than providing audio
features occasionally. While the provision of audio features to the visual all
the time gives $-800$ reward during the last episodes, the occasional use of
audio features yields average reward of $-1000$ during the same episodes.

The experimental results of the training and test demonstrate the use of audio
features for reinforcement learning task. The reported results show that audio
features are useful for the reinforcement learning agent in scenarios where
visual information may not be found. The experimental results also demonstrate
that the addition of complementary audio information to the visual one enables
the agent to reach the goal faster than using only the visuals.

##  6 Conclusions

In this work, we have proposed the use of audio features for reinforcement
learning and evaluate its performance in ViZDoom environment. First of all,
the simulation results show that the use of visual with audio provides better
average reward and lower number of steps compared to using only visual
information. Secondly, the results demonstrate the usefulness of audio
features for the autonomous agent in scenarios where the agent may not get
visual information. The results of our work manifest the usefulness of audios
for reinforcement learning (i.e., agents do benefit from hearing).

The future work could focus on confirming these results in different
environments and tasks such as other video games or high-fidelity audio
simulations. This could be done with or without visual information, and
comparing it against visual information.

##  7 Acknowledgements

This research was partially funded by the Academy of Finland (grant #313970)
and Finnish Scientific Advisory Board for Defence (MATINE) project
#2500M-0106. We gratefully acknowledge the support of NVIDIA Corporation with
the donation of the Titan Xp GPU used for this research.

## References

  * [1] B. Sallans and G. E. Hinton, “Reinforcement learning with factored states and actions,” _Journal of Machine Learning Research_ , vol. 5, no. Aug, pp. 1063–1088, 2004.
  * [2] A. Graves, A.-r. Mohamed, and G. Hinton, “Speech recognition with deep recurrent neural networks,” in _2013 IEEE international conference on acoustics, speech and signal processing_. IEEE, 2013, pp. 6645–6649.
  * [3] O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein _et al._ , “Imagenet large scale visual recognition challenge,” _International journal of computer vision_ , vol. 115, no. 3, pp. 211–252, 2015.
  * [4] M. Lederle and B. Wilhelm, “Combining high-level features of raw audio waves and mel-spectrograms for audio tagging,” _arXiv preprint arXiv:1811.10708_ , 2018.
  * [5] O. Abdel-Hamid, A.-r. Mohamed, H. Jiang, L. Deng, G. Penn, and D. Yu, “Convolutional neural networks for speech recognition,” _IEEE/ACM Transactions on audio, speech, and language processing_ , vol. 22, no. 10, pp. 1533–1545, 2014.
  * [6] K. J. Piczak, “Environmental sound classification with convolutional neural networks,” in _2015 IEEE 25th International Workshop on Machine Learning for Signal Processing (MLSP)_. IEEE, 2015, pp. 1–6.
  * [7] R. S. Sutton, A. G. Barto _et al._ , _Introduction to reinforcement learning_. MIT press Cambridge, 1998, vol. 135.
  * [8] L. P. Kaelbling, M. L. Littman, and A. W. Moore, “Reinforcement learning: A survey,” _Journal of artificial intelligence research_ , vol. 4, pp. 237–285, 1996.
  * [9] J. Ba, V. Mnih, and K. Kavukcuoglu, “Multiple object recognition with visual attention,” _arXiv preprint arXiv:1412.7755_ , 2014.
  * [10] N. Heess, G. Wayne, D. Silver, T. Lillicrap, T. Erez, and Y. Tassa, “Learning continuous control policies by stochastic value gradients,” in _Advances in Neural Information Processing Systems_ , 2015, pp. 2944–2952.
  * [11] O. Vinyals, T. Ewalds, S. Bartunov, P. Georgiev, A. S. Vezhnevets, M. Yeo, A. Makhzani, H. Küttler, J. Agapiou, J. Schrittwieser _et al._ , “Starcraft ii: A new challenge for reinforcement learning,” _arXiv preprint arXiv:1708.04782_ , 2017.
  * [12] V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski _et al._ , “Human-level control through deep reinforcement learning,” _Nature_ , vol. 518, no. 7540, p. 529, 2015.
  * [13] S. Levine, C. Finn, T. Darrell, and P. Abbeel, “End-to-end training of deep visuomotor policies,” _The Journal of Machine Learning Research_ , vol. 17, no. 1, pp. 1334–1373, 2016.
  * [14] S. Shalev-Shwartz, S. Shammah, and A. Shashua, “Safe, multi-agent, reinforcement learning for autonomous driving,” _arXiv preprint arXiv:1610.03295_ , 2016.
  * [15] M. Kempka, M. Wydmuch, G. Runc, J. Toczek, and W. Jaśkowski, “ViZDoom: A Doom-based AI research platform for visual reinforcement learning,” in _IEEE Conference on Computational Intelligence and Games_. Santorini, Greece: IEEE, Sep 2016, pp. 341–348, the best paper award. [Online]. Available: http://arxiv.org/abs/1605.02097
  * [16] C. J. Watkins and P. Dayan, “Q-learning,” _Machine learning_ , vol. 8, no. 3-4, pp. 279–292, 1992.
  * [17] D. Ververidis and C. Kotropoulos, “Emotional speech recognition: Resources, features, and methods,” _Speech communication_ , vol. 48, no. 9, pp. 1162–1181, 2006.
  * [18] id Software, “Doom,” 1993.
  * [19] V. Mnih, A. P. Badia, M. Mirza, A. Graves, T. Lillicrap, T. Harley, D. Silver, and K. Kavukcuoglu, “Asynchronous methods for deep reinforcement learning,” in _International conference on machine learning_ , 2016, pp. 1928–1937.