# Encoding Cardiopulmonary Exercise Testing Time Series as Images for
Classification using Convolutional Neural Network

Yash Sharma1, Nick Coronato2 and Donald E. Brown2 1Department of Pediatrics,
University of Virginia<EMAIL_ADDRESS>of Engineering Systems
and Environment, University of Virginia

###### Abstract

Exercise testing has been available for more than a half-century and is a
remarkably versatile tool for diagnostic and prognostic information of
patients for a range of diseases, especially cardiovascular and pulmonary.
With rapid advancements in technology, wearables, and learning algorithm in
the last decade, its scope has evolved. Specifically, Cardiopulmonary exercise
testing (CPX) is one of the most commonly used laboratory tests for objective
evaluation of exercise capacity and performance levels in patients. CPX
provides a non-invasive, integrative assessment of the pulmonary,
cardiovascular, and skeletal muscle systems involving the measurement of gas
exchanges. However, its assessment is challenging, requiring the individual to
process multiple time series data points, leading to simplification to peak
values and slopes. But this simplification can discard the valuable trend
information present in these time series. In this work, we encode the time
series as images using the Gramian Angular Field and Markov Transition Field
and use it with a convolutional neural network and attention pooling approach
for the classification of heart failure and metabolic syndrome patients. Using
GradCAMs, we highlight the discriminative features identified by the model.

Clinical relevance — The proposed framework can process multivariate exercise
testing time-series data and accurately predict cardiovascular diseases.
Interpretable Grad-CAMs can be obtained to explain the prediction.

##  I INTRODUCTION

Exercise intolerance is a major clinical feature from the early stages and a
source of symptoms for referral to a physician. In the last decade,
Cardiopulmonary Exercise Testing (CPX) has emerged as an important tool for
non-invasive monitoring of cardiopulmonary vitals of patients. It provides an
objective, reliable, and reproducible assessment of cardiorespiratory fitness
and, as such, an effective instrument for use by clinical practitioners to
inform strategies to improve the health outcomes of their patients. CPX
enables the measurement of physiological response to physical exercise through
an array of pulmonary, cardiovascular, and metabolic measurements built around
breath-by-breath gas exchange analysis. In cardiology, CPX was introduced in
early 1980s for the classification of patients with Heart failure with reduced
ejection fraction [1]. As physical work is an essential activity of daily
living, CPX is “disease agnostic” and, over the years, has been adopted as an
assessment tool for a wide range of conditions such as heart failure,
hypertension, Gaucher’s disease, and cystic fibrosis, among others [2, 3, 4,
5]. However, the very integrative and complex physiology revealed by CPX
renders analysis challenging–its interpretation requires the individual to
process a bewildering array of thousands of data points, including oxygen
uptake, carbon dioxide output, ventilation, heart rate, all of which are
changing dynamically with time as the exercise performed starts, stops, and
alters in intensity.

In 2010, in its scientific statement, AHA highlighted that CPX provides a wide
array of unique and clinically useful incremental information that has been
poorly understood and underutilized by practicing physicians [6]. Interpreting
multiple time series variables obtained in CPX is a non-trivial time-consuming
exercise. The interpretation requires extensive knowledge and a detailed
understanding of all variables, tables, and flow charts. However, recent
growth in machine learning (ML) research has propelled the applications of
CPX. Inbar et al. [7] used SVM for identifying chronic heart failure patients
using CPX, Yang et al. [8] utilized ML for studying the efficacy of aerobic
exercise intervention on young hypertensive patients, Diller et al. [9] used
deep learning on CPX variables along with other clinical, demographic, ECG,
and laboratory parameters for guiding therapy in adult congenital heart
disease. Moreover, due to strenuous testing involved in CPX, active research
is being conducted to develop ML-based approaches for early estimation of
patient response to exercise [10] and for estimating the VO2 dynamics from the
heart rate and inputs from the treadmill ergometer, cycle, and accelerometer,
with focus on incorporating these algorithms into smart devices [11, 12, 13,
14, 15, 16].

Interpretation of CPX is a multivariate time series problem involving
simultaneous assessment of generated heart rate, ventilation, gas exchange
(oxygen uptake), and carbon dioxide output. The manual evaluation and
traditional analytics of these time series are simplified to peak values,
summary indices, and slopes. VO2 peak is considered the gold standard
assessment of cardiorespiratory fitness. This process of simplification and
distilling signals out of long time series is known as feature engineering in
ML. However, this process is constrained by the expertise of users and already
discovered metrics in the literature, which can lead to the discarding of
valuable time-series information. In recent years, Deep learning (DL) has
emerged as the new flexible learning framework in which automatic extraction
of relevant features happens based on the learning objective. Myers et al.
[17] demonstrated that ANN performed better than conventional survival
analysis on CPX data for estimating cardiovascular mortality risk.

Among these multiple DL frameworks, due to parameter efficiency and spatial
invariance property, CNN has gained adoption for many problems, including time
series [18]. Further, Wang et al. [19] and Yang et al. [20] demonstrated that
encoding time series as images using Gramian Angular Field (GAF) and Markov
Transition Fields (MTF) for CNNs leads to superior performance and in the
identification of patterns not found in the one-dimensional sequence methods.
The image encoding step in combination with small CNN leads to a parameter
efficient framework, reducing the requirement of a big dataset required for
training parameter-heavy DL models. In this paper, we used GAF/MTF for
encoding CPX variables and used the attention pooling approach for aggregating
the multivariate time series representation. Further, we demonstrated on the
open Wafer data [21] that the attention pooling approach leads to superior
performance than the concatenation approach [22].

Figure 1: Proposed Architecture: Each of the univariate time series is encoded
as an image using GAF/MTF, passed through CNN model for generating
representation, aggregated using attention pooling approach to a single
representation, and passed through a linear layer for classification loss.

##  II Methods

###  II-A Gramian Angular Field

GAF encodes a time series into images by polar coordinates based matrix.
First, a Piecewise aggregate approximation is performed to reduce the length
of time series for aligning them to same length then values are normalized
between -1 and 1. Rescaled values are encoded as angular cosine and time stamp
as radius for encoding the values into polar coordinates. Encoded values warp
among different angular points on the spanning circles with an increase in
time. The encoding is bijective, and as opposed to cartesian coordinates,
preserves absolute temporal relations. These rescaled time series can be used
in 2 different forms: Summation Field (GASF) and Difference Field (GADF).
These formulations preserve the temporal dependency since time increases as
the position moves from top-left to bottom-right and embeds the temporal
correlation within different time intervals in the image.

###  II-B Markov Transition Field

Markov Transition field uses Markov transition probabilities to preserve the
information in the time domain. A $Q\times Q$ Markov transition matrix is
created by dividing the data into Q quantile bins. The quantile bins that
contain the data at time stamps $i$ and $j$ are $q_{i}$ and $q_{j}$. $M_{ij}$
in MTF denotes the transition probability of $q_{i}\rightarrow q_{j}$. The MTF
encodes the multi-span transition probabilities of the time series. The MTF
size is reduced by averaging the pixels in each non-overlapping $m\times m$
patch. However, as the probabilities of moving elements form the transformed
matrix, the MTF method cannot revert to the raw time-series data like GAF, and
it is not as symmetrical as the GAF method.

Figure 2: Illustration of encoding methods: a) Markov Transition Field b)
Gramian Angular Field

###  II-C Attention Pooling

We used the weighted-average aggregation for aggregating the univariate time
series representation to multivariate time series representation. This pooling
approach uses a two-layered neural network to compute weights for each time
series and use them for aggregating representation [22].

###  II-D Set-Up

The model was implemented with PyTorch and trained on a single RTX2080 GPU.
The framework was trained end-to-end with Adam optimizer with a batch size of
1 and for 100 epochs. Based on the validation accuracy, the best-performing
model was picked. A learning rate of 0.0023 was used in Wafer experiments and
3e-4 in CPX experiments.

##  III Results

###  III-A Data Description

We demonstrated the approach on open-sourced Wafer data for comparison to
previous approaches and reported our results on CPX data. The CPX dataset
consists of 30 patients diagnosed with either heart failure or metabolic
syndrome (15 patients each). For each of the patients, CPX data contain
breath-by-breath readings of the following variables: Metabolic equivalent of
task (1 MET = 3.5ml/kg/min); HR (beats/min); Absolute VO2 (L/min); Relative
VO2 adjusted to body mass (ml/kg/min); VCO2 (L/min); Respiratory exchange
ratio; VE (L/min); VE/VCO2; VE/VO2; respiratory rate (breaths/min); expiratory
tidal volume (L); and inspiratory tidal volume (L). Proper ethical review was
obtained for using this data in the study. The Wafer dataset was collected
from six vacuum chamber sensors that monitored the manufacture of
semiconductor microelectronics and have two classes normal and abnormal [21].
For experimentation, five-fold cross-validation was used, and results averaged
from 20 runs were reported.

###  III-B Wafer Data

We compared our approach to other proposed approaches for Wafer data [21, 20].
For consistency, we use the same encoder architecture used in [20], originally
proposed in [23] (Figure 3). We demonstrated that attention pooling leads to
lower error rates than the concatenation approach. Moreover, we highlight that
CNN-based methods on image encoded time series perform competitively to the
other approaches. Based on our observations, we hypothesize that for
multivariate time series problems, a small CNN model on GAF/MTF based encoding
of time series with attention pooling is capable of strong baselines in a
limited data scenario.

TABLE I: Comparison of average classification error rates (%) of different methods on the Wafer dataset and performance of our proposed approach on CPX dataset. Approach | Error (%)
---|---
| Wafer | CPX
DTW [24] | 2.01 | -
DDTW [24] | 9.21 | -
DDDTW [24] | 1.92 | -
STKG-SVM-K3 [25] | 1.23 | -
STKG-NB-K5 [25] | 3.69 | -
STKG-IF-PSVM-DT+M [25] | 0.84 | -
STKG-IF-NB-SVM+M [25] | 2.23 | -
normDTW [26] | 3.85 | -
combDTW [26] | 2.01 | -
LSTM-FCN [27] | 1.00 | -
MLSTM-FCN [27] | 1.00 | -
ALSTM-FCN [27] | 1.00 | -
MALSTM-FCN [27] | 1.00 | -
MALSTM-FCN [27] | 1.00 | -
concat-MTF-RGB [20] | 0.40 | -
concat-GASF-RGB [20] | 0.57 | -
concat-GADF-RGB [20] | 0.44 | -
concat-MTF (ours) | 0.63 | 23.40
concat-GASF (ours) | 1.18 | 12.34
concat-GADF (ours) | 0.46 | 8.67
Attn-MTF (ours) | 0.41 | 15.00
Attn-GASF (ours) | 0.55 | 16.17
Attn-GADF (ours) | 0.15 | 7.67
Figure 3: CNN Architecture: a) Proposed in [23] and used for Wafer Data
Experiment b) Used in CPX data experiments

###  III-C CPX Data

In the CPX data, for high-quality GradCAMs, we made minor changes to the CNN
architecture used in Wafer experiments, removed the max-pooling layer and
replaced it with single adaptive average pooling, and increased the number of
channels in both the layers (Figure 3). Among the three time-series encoding
approaches - GASF, GADF, and MTF, GADF was performing best. We attributed the
high performance to the temporal changes that GADF captured during the CPX
exercise. In the GADF approach, HR, RER, and V’CO2 were assigned the highest
attention among all the time series. We analyzed these frames using GradCAMs
to probe the patterns CNN identified for differentiating diseases. In
metabolic syndrome patients, we observed that CNNs focused on the increase in
the slope of the time series. For heart failure patients, we observed that the
model was sensitive to alterations happening in quick succession and to the
drop in time series during exercise (Figure 4). Since all the images were
encoded independently using GAF/MTF methods, the model was exposed to temporal
trends instead of absolute values, which identified trends unique to heart
failure and metabolic syndrome. The patterns identified in the GradCAM plots
were deemed relevant in our medical review by a panel of cardiologists and
pulmonologists.

Figure 4: GradCAM examples of the top attended patches (HR, RER, V’CO2). The
part highlighted as green was considered relevant by the models.
Interpretation of charts is discussed in Section III-C

##  IV Conclusions

In this paper, we demonstrated the strong performance of small CNNs on CPX
data for differentiating heart failure and metabolic syndrome patients. With
the advancement in the internet of things and wearables, a large amount of
fitness data is getting collected, which can be used with deep learning
modeling to diagnose diseases. Similar approaches like ours can aid experts in
identifying markers relevant for the early diagnosis of health conditions. Our
approach is able to accurately classify heart failure and metabolic syndrome
patients using temporal trends instead of simplified values such as peak and
slope. Further, we highlighted that the attention pooling approach could be
used with GAF/MTF approaches to aggregate univariate time series
representation. Interpretation based on this approach has vast scope in the
field of exercise testing.

## ACKNOWLEDGMENT

This work was supported by the National Center for Advancing Translational
Science of the National Institutes of Health Award UL1TR003015/ KL2TR003016.

We would like to thank Dr. Arthur L. Weltman and Exercise Physiology Core
Laboratory for providing the data and helping us with queries.

## References

  * [1] K. T. Weber, G. T. Kinasewitz, J. S. Janicki, and A. P. Fishman, “Oxygen utilization and ventilation during exercise in patients with chronic cardiac failure.” _Circulation_ , vol. 65, no. 6, pp. 1213–1223, 1982.
  * [2] M. Guazzi, F. Bandera, C. Ozemek, D. Systrom, and R. Arena, “Cardiopulmonary exercise testing: what is its value?” _Journal of the American College of Cardiology_ , vol. 70, no. 13, pp. 1618–1636, 2017.
  * [3] M. Guazzi, R. Raimondo, M. Vicenzi, R. Arena, C. Proserpio, S. Sarzi Braga, and R. Pedretti, “Exercise oscillatory ventilation may predict sudden cardiac death in heart failure patients,” _Journal of the American College of Cardiology_ , vol. 50, no. 4, pp. 299–308, 2007.
  * [4] M. Bjelobrk, M. Lakocevic, S. Damjanovic, M. Petakov, M. Petrovic, Z. Bosnic, R. Arena, and D. Popovic, “Cardiopulmonary assessment of patients diagnosed with gaucher’s disease type i,” _Molecular Genetics & Genomic Medicine_, vol. 9, no. 8, p. e1757, 2021.
  * [5] C. A. Williams, K. C. Wedgwood, H. Mohammadi, K. Prouse, O. W. Tomlinson, and K. Tsaneva-Atanasova, “Cardiopulmonary responses to maximal aerobic exercise in patients with cystic fibrosis,” _PloS one_ , vol. 14, no. 2, p. e0211219, 2019.
  * [6] G. J. Balady, R. Arena, K. Sietsema, J. Myers, L. Coke, G. F. Fletcher, D. Forman, B. Franklin, M. Guazzi, M. Gulati _et al._ , “Clinician’s guide to cardiopulmonary exercise testing in adults: a scientific statement from the american heart association,” _Circulation_ , vol. 122, no. 2, pp. 191–225, 2010.
  * [7] O. Inbar, O. Inbar, R. Reuveny, M. J. Segel, H. Greenspan, and M. Scheinowitz, “A machine learning approach to the interpretation of cardiopulmonary exercise tests: Development and validation,” _Pulmonary Medicine_ , vol. 2021, 2021.
  * [8] G. Yang, X. Leng, F. Huang, M. V. Kasukurthi, Y. Huang, D. Li, J. Lin, S. Tan, G. Lu, R. Benton _et al._ , “Use cpet data to predict the intervention effect of aerobic exercise on young hypertensive patients,” in _2019 IEEE International Conference on Bioinformatics and Biomedicine (BIBM)_. IEEE, 2019, pp. 1699–1702.
  * [9] G.-P. Diller, A. Kempny, S. V. Babu-Narayan, M. Henrichs, M. Brida, A. Uebing, A. E. Lammers, H. Baumgartner, W. Li, S. J. Wort _et al._ , “Machine learning algorithms estimating prognosis and guiding therapy in adult congenital heart disease: data from a single tertiary centre including 10 019 patients,” _European heart journal_ , vol. 40, no. 13, pp. 1069–1077, 2019\.
  * [10] E. Baralis, T. Cerquitelli, S. Chiusano, A. Giordano, A. Mezzani, D. Susta, and X. Xiao, “Predicting cardiopulmonary response to incremental exercise test,” in _2015 IEEE 28th International Symposium on Computer-Based Medical Systems_. IEEE, 2015, pp. 135–140.
  * [11] M. M. H. Shandhi, S. Hersek, J. Fan, E. Sander, T. De Marco, J. A. Heller, M. Etemadi, L. Klein, and O. T. Inan, “Wearable patch-based estimation of oxygen uptake and assessment of clinical status during cardiopulmonary exercise testing in patients with heart failure,” _Journal of Cardiac Failure_ , vol. 26, no. 11, pp. 948–958, 2020.
  * [12] T. Beltrame, R. Amelard, R. Villar, M. J. Shafiee, A. Wong, and R. L. Hughson, “Estimating oxygen uptake and energy expenditure during treadmill walking by neural network analysis of easy-to-obtain inputs,” _Journal of Applied Physiology_ , vol. 121, no. 5, pp. 1226–1233, 2016.
  * [13] M. Mazzoleni, “A multi-disciplinary systems approach for modeling and predicting physiological responses and biomechanical movement patterns,” Ph.D. dissertation, Duke University, 2017.
  * [14] M. F. Akay, E. Çetin, İ. Yarım, Ö. Bozkurt, and M. M. Özçiloğlu, “development of novel maximal oxygen uptake prediction models for turkish college students using machine learning and exercise data,” in _2017 9th International Conference on Computational Intelligence and Communication Networks (CICN)_. IEEE, 2017, pp. 186–189.
  * [15] A. De Brabandere, T. Op De Beéck, K. H. Schütte, W. Meert, B. Vanwanseele, and J. Davis, “Data fusion of body-worn accelerometers and heart rate to predict vo2max during submaximal running,” _PloS one_ , vol. 13, no. 6, p. e0199509, 2018.
  * [16] A. Borror, M. Mazzoleni, J. Coppock, B. C. Jensen, W. A. Wood, B. Mann, and C. L. Battaglini, “Predicting oxygen uptake responses during cycling at varied intensities using an artificial neural network,” _Biomedical Human Kinetics_ , vol. 11, no. 1, pp. 60–68, 2019.
  * [17] J. Myers, C. R. De Souza, A. Borghi-Silva, M. Guazzi, P. Chase, D. Bensimhon, M. A. Peberdy, E. Ashley, E. West, L. P. Cahalin _et al._ , “A neural network approach to predicting outcomes in heart failure using cardiopulmonary exercise testing,” _International journal of cardiology_ , vol. 171, no. 2, pp. 265–269, 2014.
  * [18] G. Ciaburro and G. Iannace, “Machine learning-based algorithms to knowledge extraction from time series data: A review,” _Data_ , vol. 6, no. 6, p. 55, 2021.
  * [19] Z. Wang and T. Oates, “Encoding time series as images for visual inspection and classification using tiled convolutional neural networks,” in _Workshops at the twenty-ninth AAAI conference on artificial intelligence_ , 2015.
  * [20] C.-L. Yang, Z.-X. Chen, and C.-Y. Yang, “Sensor classification using convolutional neural network by encoding multivariate time series as two-dimensional colored images,” _Sensors_ , vol. 20, no. 1, p. 168, 2020\.
  * [21] R. T. Olszewski, _Generalized feature extraction for structural pattern recognition in time-series data_. Carnegie Mellon University, 2001.
  * [22] M. Ilse, J. Tomczak, and M. Welling, “Attention-based deep multiple instance learning,” in _International conference on machine learning_. PMLR, 2018, pp. 2127–2136.
  * [23] R. B. Palm, “Prediction as a candidate for learning deep hierarchical models of data,” _Technical University of Denmark_ , vol. 5, 2012.
  * [24] T. Górecki and M. Łuczak, “Multivariate time series classification with parametric derivative dynamic time warping,” _Expert Systems with Applications_ , vol. 42, no. 5, pp. 2305–2312, 2015.
  * [25] O. J. Prieto, C. J. Alonso-González, and J. J. Rodríguez, “Stacking for multivariate time series classification,” _Pattern Analysis and Applications_ , vol. 18, no. 2, pp. 297–312, 2015.
  * [26] M. Łuczak, “Combining raw and normalized data in multivariate time series classification with dynamic time warping,” _Journal of Intelligent & Fuzzy Systems_, vol. 34, no. 1, pp. 373–380, 2018.
  * [27] F. Karim, S. Majumdar, H. Darabi, and S. Harford, “Multivariate lstm-fcns for time series classification,” _Neural Networks_ , vol. 116, pp. 237–245, 2019\.