As more use cases are attracted, the platform is broadly validated, usability will be optimised, and collaboration with European users and projects can be established. This will put all components to a test, including the iRODS systems of our EUDATbased data infrastructure, which takes care of transparent data transfers within workflows and serves the users for managing their data via the LEXIS Portal. As a result, the LEXIS platform will be optimised to reliably and efficiently execute an entire spectrum of workloads, including visualisation and GPU- or even FPGAaccelerated tasks. Orchestrated LEXIS workflows will thus efficiently combine different computing paradigms (HPC, HTC, Cloud) and analysis methods (classical modelling, AI, HPDA).

We are looking forward to extend the LEXIS federation to more computing and data sites, and software necessary to join the platform will be conveniently packaged. With this open approach, we will continue to push towards a convergence of industrial and academic data science in Europe and towards a convergence of the HPC, Cloud and Big Data ecosystems on the strongest European compute systems. Enlarging the federated LEXIS platform and making it sustainable, besides work on novel functionalities, is the key focus of the project in its second half.

**Acknowledgments** This work and all contributing authors are funded/co-funded by the EU's Horizon 2020 research and innovation programme (2014–2020) under grant agreement no. 825532 (Project LEXIS – 'Large-scale EXecution for Industry and Society'). The work at IT4I is supported by The Ministry of Education, Youth and Sports from the Large Infrastructures for Research, Experimental Development and Innovations project 'e-Infrastructure CZ – LM2018140'. Likewise, we gratefully acknowledge the use of the computing facilities at LRZ and IT4I, as mentioned in Sect. 2.2. We thank various colleagues from EUDAT for very kind help and good collaboration.

#### **References**

**Open Access** This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence and indicate if changes were made.

The images or other third party material in this chapter are included in the chapter's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

# **Part II Processes and Applications**

# **The DeepHealth Toolkit: A Key European Free and Open-Source Software for Deep Learning and Computer Vision Ready to Exploit Heterogeneous HPC and Cloud Architectures**

**Marco Aldinucci, David Atienza, Federico Bolelli, Mónica Caballero, Iacopo Colonnelli, José Flich, Jon A. Gómez, David González, Costantino Grana, Marco Grangetto, Simone Leo, Pedro López, Dana Oniga, Roberto Paredes, Luca Pireddu, Eduardo Quiñones, Tatiana Silva, Enzo Tartaglione, and Marina Zapater**

**Abstract** At the present time, we are immersed in the convergence between Big Data, High-Performance Computing and Artificial Intelligence. Technological progress in these three areas has accelerated in recent years, forcing different players like software companies and stakeholders to move quickly. The European Union is dedicating a lot of resources to maintain its relevant position in this

F. Bolelli · C. Grana Università degli studi di Modena e Reggio Emilia, Modena, Italy

M. Caballero NTT Data Spain, Barcelona, Spain

J. Flich · J. A. Gómez (-) · P. López · R. Paredes Universitat Politècnica de València, Valencia, Spain e-mail<EMAIL_ADDRESS>
D. González · T. Silva TREE Technology, Asturias, Spain

D. Oniga Softwae Imagination and Vision, Bucures*,*ti, Romania

E. Quiñones Barcelona Supercomputing Center, Barcelona, Spain

M. Aldinucci · I. Colonnelli · M. Grangetto · E. Tartaglione Università degli studi di Torino, Torino, Italy

D. Atienza · M. Zapater Ecole Polytechnique Fédérale de Lausanne, Lausanne, Switzerland

S. Leo · L. Pireddu Center for Advanced Studies, Research and Development in Sardinia, Sardinia, Italy

scenario, funding projects to implement large-scale pilot testbeds that combine the latest advances in Artificial Intelligence, High-Performance Computing, Cloud and Big Data technologies. The DeepHealth project is an example focused on the health sector whose main outcome is the DeepHealth toolkit, a European unified framework that offers deep learning and computer vision capabilities, completely adapted to exploit underlying heterogeneous High-Performance Computing, Big Data and cloud architectures, and ready to be integrated into any software platform to facilitate the development and deployment of new applications for specific problems in any sector. This toolkit is intended to be one of the European contributions to the field of AI. This chapter introduces the toolkit with its main components and complementary tools, providing a clear view to facilitate and encourage its adoption and wide use by the European community of developers of AI-based solutions and data scientists working in the healthcare sector and others.

**Keywords** Hybrid big data HPC architectures · High performance data analytics · Hardware-specific capabilities for big data GPUs FPGAs · Performance for large-scale processing

# **1 Context: The European AI and HPC Landscape and the DeepHealth Project**

The rapid progress of different technologies is taking place within a virtuous circle that emerged thanks to the synergies between such technologies and has brought us three important advances in recent years, namely, the increase in storage capacity at a reduced price, the increase in data transmission speed and the increase in computing power provided by High-Performance Computing (HPC) and hardware accelerators. These three advances, in combination with the availability of largeenough volumes of data, have considerably boosted the growth and development of Artificial Intelligence (AI) in recent years. Mainly, thanks to the fact that the techniques of Machine Learning (ML), able to learn from data, have reached a good level of maturity and are improving the best results obtained by expert systems at the core of knowledge-based solutions in most application domains. Machine Learning is one of the most important areas of AI, which in turn includes Deep Learning (DL). As such, descriptive/predictive/prescriptive models based on AI/ML/DL techniques are becoming key components of applications and systems deployed in real scenarios for solving problems in a wide variety of sectors (e.g., manufacturing, agriculture and food, Earth sciences, retail, fintech and smart cities, among others). Nevertheless, its use in the health sector is still far from being widely spread (see [1]).

In this scenario, the European Union (EU) is fostering strategic actions to position the EU as a big worldwide player in AI, HPC and Big Data, capable of creating and deploying solutions based on cutting-edge technologies. The DeepHealth project whose title is "Deep-Learning and HPC to Boost Biomedical Applications for Health" [2], funded by the EC under the topic ICT-11-2018-2019 "HPC and Big Data enabled Large-scale Test-beds and Applications", is one of the innovation actions supported by the EU to boost AI and HPC leadership and promote large-scale pilots. DeepHealth is a 3-year project, kicked off in January 2019 and scheduled to conclude in December 2021. DeepHealth aims to foster the use of technology in the Healthcare sector by reducing the current gap between the availability of mature-enough AI-based medical imaging solutions and their deployment in real scenarios. The main goal of the DeepHealth project is to put HPC power at the service of biomedical applications that require the analysis of large and complex biomedical datasets and apply DL and Computer Vision (CV) techniques to support new and more efficient ways of diagnosis, monitoring and treatment of diseases.

Following this aim, one of the main outcomes of DeepHealth addressing industry needs is the DeepHealth toolkit, a free and open-source software designed to be a European unified framework to offer DL and CV capabilities completely adapted to exploit underlying heterogeneous HPC, Big Data and cloud architectures. The DeepHealth toolkit is aimed at computer and data scientists as well as to developers of AI-based solutions working in any sector. It is a piece of software ready to be integrated into any software platform, designed to facilitate the development and deployment of new applications for specific problems. Within the framework of the DeepHealth project, the toolkit is being developed, tested and validated by using it to implement descriptive/predictive models for 14 healthcare use cases; nevertheless, its usefulness goes beyond the health sector, being applicable, as said, to any application domain or industrial sector. Thanks to all its features, which will be detailed throughout this chapter, the DeepHealth toolkit is technology made in EU that contributes to the development of AI in Europe.

This chapter is aligned with the technical priorities of Data Processing Architectures and Data Analytics of the European Big Data Value Strategic Research and Innovation Agenda [3]. It addresses the vertical concern Engineering and DevOps of the BDV Technical Reference Model, and the horizontal concerns Data Analytics and Data Processing Architectures focusing on the Cloud and HPC. And this chapter also relates to the *Systems, Methodologies, Hardware and Tools* cross-sectorial technology enablers of the AI, Data and Robotics Strategic Research, Innovation and Deployment Agenda [4].

This chapter also introduces the toolkit, its functionalities and its enabling capabilities with the objective to bring it closer to potential users in both industry and academia. To do so, the authors present the toolkit, its components, the adaptations that allow exploiting HPC and cloud computing infrastructures thanks to complementary HPC frameworks, and describe practical aspects to guide on its use and how to effectively integrate it for the development of AI-based applications.

#### **2 A General Overview of the DeepHealth Toolkit**

The DeepHealth toolkit is a general-purpose deep learning framework, including image processing and computer vision functionalities, enabled to exploit HPC and cloud infrastructures for running parallel/distributed training and inference processes. All the components of the toolkit are available as free and open-source software under the MIT license [5]. This framework enables data scientists to design and train predictive models based on deep neural networks, and developers to easily integrate the predictive models into existing software applications/platforms in order to quickly build and deploy AI-based solutions (e.g., support decision tools for diagnosis).

The toolkit is specifically designed to cope with big and constantly growing datasets (e.g., medical imaging datasets). Large-enough datasets enable the use of more complex neural networks and drive to improve both the accuracy and robustness of predictive models, but at the cost of dramatically increasing the demand of computing power. To do so, the DeepHealth toolkit incorporates, in a transparent manner, the most advanced parallel programming models to exploit the parallel performance capabilities of HPC and cloud infrastructures, featuring different acceleration technologies such as symmetric multi-processors (SMPs), graphic processing units (GPUs) and field-programmable gate arrays (FPGAs). It also integrates additional frameworks (i.e., COMPSs [6] and StreamFlow [7]) that allow to exploit specialized infrastructures, enabling parallelization mechanisms at different levels. Moreover, the toolkit provides functionalities to be used for both training and inference, addressing the complexity of the different available computational resources and target architectures at both the training and inference stages. Training is performed by AI experts, commonly in research-focused environments, using specialized HPC architectures equipped with FPGAs and GPUs; the goal is to maximize the number of samples processed per second keeping the overall accuracy. Inference is done with trained models in production environments (even using small devices in the edge), where the response time for predicting single samples is crucial.

The core of the toolkit consists of two libraries, namely the European Computer Vision Library (ECVL) and the European Distributed Deep Learning Library (EDDL), that are accompanied by the back end and the front end, two components to allow and facilitate the use of the libraries. The back end is a software-as-aservice module that offers a RESTful API to give access to all the functionalities of both libraries and provides independency from the programming language. The front end is a web-based graphical user interface, mainly oriented to be used by data scientists, for designing, training and testing deep neural networks. Both libraries are implemented in C++ and include a Python API to facilitate the development of client applications and integration with the wide array of Python-based data analysis libraries. Figure 1 depicts the components of the toolkit and highlights the two possible alternatives for developers of domain-specific applications to use the functionalities provided by both libraries: (1) the use of a RESTful API provided

by the back end (represented in Fig. 1 by the arrows labelled 1) or (2) the use of the specific APIs (C++ or Python) of each library (represented in Fig. 1 by the arrows labelled 2). This second option is not independent of the programming language, yet it provides more control and flexibility over the software at the cost of additional programming complexity. The former option requires less effort from platform developers and makes applications independent of specific versions of the libraries. Only versions including changes in the RESTful API will require updating and recompiling applications. It can also be observed from Fig. 1 that all the DL and CV functionalities can be thoroughly used via the front end or via a software application/platform, in this last case with or without the back end. Additionally, Fig. 1 shows the runtime of both libraries, which can be used to launch distributed learning processes to train models on HPC and cloud architectures. Both libraries are designed to run under the control of HPC-specific workflow managers such as COMPSs [6] and StreamFlow [7], presented in Sect. 6. Once the trained models are tested and validated, they are ready to be used in production environments to perform inference from new samples by using the software applications/platforms in which libraries are integrated.

The following describes the typical workflow of the usage of the toolkit by a development team who is requested to address a new use case in a real scenario, considering that the libraries of the DeepHealth toolkit are already integrated in the platform any company developed to deploy AI-based solutions in the health sector. First, (i) data scientists, members of the team, prepare the dataset by splitting it into three subsets, namely training, validation and testing subsets. Next, (ii) the team designs several artificial neural networks and (iii) launches the training processes on HPC and cloud architectures by means of the runtime of the toolkit adapted to HPC frameworks like the ones described in Sect. 6. (iv) The team evaluates the models using the validation subset, and goes back to step (ii) to redesign some models if necessary. Sometimes, the team should come back to step (i) to consider the dataset itself with the knowledge gained from previous iterations. (v) The model that gets the best accuracy using the testing subset is selected; then (vi) computer scientists, members of the same team, configure an instance of the application with the best model and deploy the solution in a production environment.

In itself, the DeepHealth toolkit provides the following features to AI-based application developers, data scientists and ML practitioners in general:

#### **3 The European Distributed Deep Learning Library**

EDDL is a general-purpose deep learning library initially developed to cover deep learning needs in healthcare use cases within the DeepHealth project. As part of the DeepHealth toolkit, EDDL is a free and open-source software available on a GitHub public repository [5]. Currently, it supports most widely used deep neural network topologies, including convolutional and sequence-to-sequence models, and is being used in different tasks like classification, semantic segmentation of images, image description, event prediction from time-series data, and machine translation. In order to be compatible with existing developments and other deep learning toolkits, the EDDL uses ONNX [8], the standard format for neural network interchange, to import and export neural networks, including both weights and topology.

EDDL provides hardware-agnostic tensor operations to facilitate the development of hardware-accelerated deep learning functionalities and the implementation of the necessary tensor operators, activation functions, regularization functions, optimization methods and all layer types (dense, convolutional and recurrent) to implement state-of-the-art neural network topologies. The EDDL exposes two APIs (C++ and Python) with functionalities belonging to two main groups: neural network manipulation (models, layers, regularization functions, initializers) and tensor operations (creation, serialization, I/O, mathematical transformations). The neural networks section provides both high-level tools, such as fitting and evaluating the whole model, and lower-level ones that allow developers to act on individual epochs and batches, providing finer control albeit with a slight efficiency loss in the case of using the Python API, since a larger part of the program needs to be written in Python to handle loops.

EDDL is implemented in C++, and the Python API, called PyEDDL [5], has been developed to enhance the value of EDDL to the scientific community. The availability of a Python library allows to integrate EDDL functionalities with widely used scientific programming tools such as NumPy/SciPy [9] and Pandas [10]. In particular, PyEDDL supports converting between EDDL tensors and NumPy arrays, which are key to enable interoperability with other Python scientific libraries. Moreover, since PyEDDL is based on a native extension module that wraps the C++ EDDL code, users can take advantage of the simplicity and speed of development of Python without sacrificing performance, using Python as a "glue" language that ties together computationally intensive native routines. PyEDDL allows Python access to the EDDL API and, as mentioned above, adds NumPy interoperability, allowing interaction with a wide array of data sources and tools. Like the rest of the DeepHealth Toolkit, PyEDDL is released as free and opensource software and its source code is available on GitHub [5].

In relation to hardware accelerators, EDDL is ready to run on single computers using either all or a subset of the available cores, all or a subset of the available GPU cards, and coordinating the computation flow on the FPGA cards connected to a single computer. The C++ and Python APIs of the EDDL both include a function to build neural networks that creates all the data structures according to the network topology and allocates all the necessary memory; one of the parameters of the build function is an object for describing the available hardware devices the EDDL will use to run the training and inference processes. EDDL defines the concept of Computing Service to describe hardware devices. Currently, three types of computing services are defined, namely CPU, GPU and FPGA. The number of CPU cores, GPU cards or FPGA cards to be used are indicated by the Computing Service.

Any neural network topology is internally represented by means of two directed and acyclic graphs (DAGs), one for the forward step and another one for the backward step. Each DAG defines the sequence of tensor operations to perform the computation corresponding to the entire network, so that the computations corresponding to a given layer will be performed when all its input dependencies according to the DAG have been satisfied, i.e., when the output of all the layers used as input to a given one are ready. Tensor operations are performed using the hardware devices specified by means of the Computing Service provided as a parameter to the build function. On manycore CPUs, tensor operations are performed by using the Eigen library [11] and parallelized using OpenMP [12]. When using GPU or FPGA cards, the forward and backward algorithms are designed to minimize the number of memory transfers between the CPU and GPU/FPGA cards. In the particular case of GPUs, EDDL has three modes of memory management to address the lack of memory when a given batch size does not fit in the memory of a GPU. The most efficient one tries to allocate the whole batch in the GPU memory to reduce memory transfers at the minimum, the intermediate and least-efficient modes allow to work with larger batch sizes at the cost of increasing the number of memory transfers to perform the forward and backward steps for a given batch of samples.

GPU support in EDDL is done by means of CUDA kernels developed as part of the EDDL code. As mentioned above, the use of different hardware accelerators is completely transparent to developers and programmers who use the EDDL; they only need to create the corresponding Computing Service to use all or a subset of the computational resources. Integrating NVIDIA cuDNN library in the EDDL as an alternative to CUDA kernels is in the work plan of the DeepHealth project.

Table 1 shows the performance in terms of the accuracy obtained with the test set and the time per epoch in seconds during training. The EDDL is compared with TensorFlow [13] and PyTorch [14], the two most popular DL toolkits. The Cifar10 dataset was used. It can be observed that EDDL performs similar to the other toolkits, but EDDL still needs to improve the performance on both CPUs and GPUs when using Batch Normalization and larger topologies like VGG16 and VGG19.

EDDL support for FPGA cards is quite similar to the support for GPU cards. The developer or data scientist using the EDDL simply indicates the target device to run training or inference processes by means of a Computing Service object. Although FPGAs can also be used for training, they are more appealing for inference processes, and, therefore, FPGA support has been optimized for the inference process. Depending on the trained model, FPGA cards can be directly used. This

**Table 1** Benchmark to compare EDDL with TensorFlow and PyTorch using Cifar10 with and without Batch Normalization

is the case when the models fit on the typically lower memory resources available on FPGA devices. If models do not fit, then two options can be used. The first one is to iteratively use FPGA cards to run the complete inference process on the model, performing operations on a step-by-step manner driven by the CPU. The FPGA support has been provided to allow this operational mode. However, quantization and compression strategies can be deployed once the model has been trained. In the DeepHealth project, the FPGA kernels mostly used on the Medical sector use cases are being optimized and adapted to quantized and compressed models. In order to deploy a model on FPGAs targeting low resource constraints and high energy efficiency, the EDDL incorporates a strategy to reduce the complexity of a deep neural network. Many techniques have been proposed recently to reduce such complexity [15–17]. These approaches include the so-called pruning techniques, whose aim is to detect and remove the irrelevant parameters from a model [18]. Removing parameters from a model has a huge impact on the deployment of the trained model on FPGA cards, since the overall size of the model reduces as well as the number of operations to generate the outcome decreases and, for instance, the power consumption. This is allowed by the typically high dimensionality of these models, where sparser and more efficient solutions can be found [19]. Towards this end, in order to deploy the model on FPGA cards targeting low resource constraints and high energy efficiency, the approach used in the EDDL is to include a structured sparsity step, where as many neurons as possible are removed from the model with a negligible performance loss.

Regarding distributed learning on HPC/cloud/HPC + cloud architectures, the EDDL includes specific functions to simplify the distribution of batches when training and inference processes are run by means of HPC frameworks like COMPSs or StreamFlow. Concretely, the COMPSs framework allows to accelerate the DL training operations by dividing the training data sets across a large set of computing nodes available on HPC and cloud infrastructures, and upon which partial training operations can then be performed. To do so, EDDL allows to distribute the weights of the network from the master node to worker nodes, and to report gradients from worker nodes to the master node, both synchronously and asynchronously. The EDDL serializes networks using ONNX to transfer weights and gradients between the master node and worker nodes. The serialization includes the network topology, the weights and the bias. To facilitate distributed learning, the serialization functions implemented in the EDDL allow to select whether to include weights or gradients.

EDDL and PyEDDL code is covered by an extensive test suite and complemented by numerous usage examples in Python and C++, including network training and evaluation with different models, ONNX serialization and NumPy compatibility. To facilitate their adoption, EDDL and PyEDDL also provide extensive documentation on installation, tensor and neural network manipulation, API usage and examples [5]. The "getting started" section contains simple examples; the most advanced ones show the use of topologies like VGG16/VGG19 [20] and U-Net [21]. Concerning installation, developers can choose between installing from source code, via conda [22], and via Homebrew for Mac OS X [23]. Additionally, pre-built Docker images with the DeepHealth toolkit components ready to be used are available on DockerHub [24] (see Sect. 6.1).

#### **4 The European Computer Vision Library**

ECVL is a general-purpose computer vision library developed to support healthcare use cases within the DeepHealth project, with the aim of facilitating the integration of existing state-of-the-art libraries such as OpenCV [25]. ECVL currently includes high-level computer vision functionalities implementing specialized and accelerated versions of algorithms commonly employed in conjunction with deep learning; functionalities that are useful for image processing tasks in any sector beyond health.

The design of ECVL is based on the concept of *Image*, which represents the core element of the entire library. It allows to store raw data, images, and videos in a multi-dimensional dense numerical single- or multi-channel tensor. Multiple types of scientific imaging data and data formats (e.g., jpeg, png, bpm, ppm, pgm, etc.) are natively supported by ECVL. Moreover, the library provides specific functionalities to handling medical data, such as DICOM, NIfTI and many proprietary Virtual Slides (VS) formats. In the case of VS, the *Image* object allows to choose the area and the resolution to be extracted from the file. The availability of a common software architecture provided by the ECVL Hardware Abstraction Layer (HAL) allows great flexibility for device differentiation (SMPs, GPUs, and FPGAs) while keeping the same user interface. This hardware-agnostic API ensures versatility, flexibility, and extensibility, simplifying the library usage and facilitating the development of distributed image analysis tasks.

The *Image* class has been designed for representing and manipulating different types of images with diverse channel configurations, providing both reading and writing functionalities for all the aforementioned data formats. Arithmetic operations between images and scalars are performed through the *Image* class. Obviously, all the classic operations for image manipulation such as rotation, resizing, mirroring and colour space change are available. Extremely optimized processing functions, like noising, blurring, contour finding [26], image skeletonization [27] and connected components labelling [28] are implemented as well. ECVL image-processing operations can be applied on-the-fly during deep neural networks training to implement data augmentation. Given the relevance of data augmentation, ECVL provides with a simple Domain-Specific Language (DSL) to facilitate the definition of transformations to be applied and their configuration parameters. A set of transformations can thus be defined for each split of a dataset (train, validation and test subsets). Augmentation can be either provided in compiled code or through the DSL and thus read from file at runtime. More details are available in [29].

Optional modules are supplied with the library and can be activated to enable additional functionalities, such as the cross-platform GUI based on wxWidgets [30], which provides simple exploration and visualization of images contained in ECVL *Image* objects, and a 3D volumes visualizer to observe different slices of a CT scan from different views.

In order to ensure an efficient and straightforward mechanism to perform distributed model training, ECVL defines the DeepHealth Dataset Format (DDF): a simple and flexible YAML-based syntax [31] that allows to describe a dataset. Regardless of the task being analysed, a DDF file provides all the information required to characterize the dataset and thus performing data loading, image preand post-processing and model training. A detailed description of such a format can be found in [29]. Moreover, a specific module to load and parse DDF-defined datasets is implemented and exposed by the library interface.

Like EDDL, ECVL is complemented by a Python API called PyECVL [5]. In addition to simplified programming, its main advantage is the ability to integrate with other scientific programming tools, which are abundant in the Python ecosystem. This interoperability is enabled by supporting the conversion between ECVL images and NumPy arrays. Like PyEDDL, PyECVL is based on a wrapper extension module that reroutes calls to the C++ code, allowing to reap the benefits of Python development without taking a big hit on performance. PyECVL exposes ECVL functionalities to Python, including Image objects, data and colour types, arithmetic operations, image processing, image I/O, augmentations, the DeepHealth dataset parser and the ECVL-EDDL interaction layer. As discussed earlier, its support for to/from array conversion allows to process data with NumPy as well as many other scientific tools based on it.

Regarding hardware accelerators, the ECVL supports the use of GPU and FPGA cards to run the computer vision algorithms needed in training and inference processes. The implementation for GPUs has been done using CUDA kernels, while for FPGA cards it is somewhat more complicated as FPGA cards are reconfigurable devices which allow the designer to fully customize their design and to adapt it to the algorithm they need to run. This enables, for specific application domains, more power- and energy-efficient solutions than, for instance, CPUs and GPUs. The DeepHealth project advocates for the use of FPGAs as accelerator devices for the inference process. In particular, the trained models ready for production can be launched to an FPGA card by using the FPGA support provided in both the ECVL and the EDDL libraries. The use of FPGA cards is totally transparent to data scientists who use the DeepHealth toolkit. Indeed, both libraries enable the developers who use them just to indicate which type of device the application should be using. For the specificities of the ECVL library, the use of FPGAs is appealing as most computer vision algorithms (e.g., image resize, mirror) deal with pixels rather than floating point values. FPGA devices excel at integer operations and offer massive parallelism possibilities within the device.

Like other software packages of the toolkit, ECVL and PyECVL are available as free and open-source software on a public GitHub repository, including documentation, comprehensive tests, and several usage examples of both the C++ and Python APIs [5]. Examples include data augmentation usage, handling of DeepHealth datasets, interaction with EDDL/PyEDDL, image processing and I/O. The documentation includes detailed instructions to install ECVL and PyECVL from different options as in the case of EDDL and PyEDDL. As mentioned in the EDDL section, a set of pre-built Docker images including the components of the DeepHealth toolkit are available in the Docker hub for the DeepHealth project [24].

#### **5 The Back End and the Front End**

The four components of the DeepHealth toolkit are the ECVL, the EDDL, the back end and the front end. Figure 1 shows how the back end and the front end are interconnected with the libraries. The back end is a software module where ECVL and EDDL are fully integrated, which offers a RESTful API to allow any software application or platform to access all the functionalities provided by both libraries without the need to use the C++ or Python API. Ready-to-use pre-built Docker images (see Sect. 6.1) are available, including the back end and all the other components of the toolkit, in such a way that the developers of applications/platforms do not have to worry about the installation and configuration of the DeepHealth toolkit, they only need to provision Docker containers and, obviously, programming, using their preferred programming language, the module for their application/platform that will interact with the RESTful API offered by the back end. This way, the back end enables *managed service* usage scenarios, where a potentially complex and powerful computing infrastructure (e.g., highperformance computing, cloud computing or even heterogeneous hardware) could be transparently used to run deep learning jobs without the users needing to directly interface with it.

The front end is a web-based graphical user interface that facilitates the use of all the functionalities of the libraries by interacting with the back end through the RESTful API. The front end is the component of the toolkit visible to any type of user, but it has been mainly designed for data scientists. Without going into implementation details, the main functionalities provided by the front end are: (1) creation/edition of user profiles; (2) creation/edition of projects; (3) dataset uploading; (4) dataset selection; (5) model creation/import/export/edition/selection; (6) definition of tasks (currently supported types are classification and segmentation); (7) definition of data augmentation transformations; (8) launching training/inference processes; (9) monitoring of training processes, including visualization of the evolution of different neural network related KPIs (e.g., accuracy and loss) with respect to both training and validation data subsets; and (10) model evaluation.

In the common usage of the front end, users have the option of loading from the back end any one of the available models in the set of pre-designed models, which can be already trained. Trained models can be used to perform transfer learning tasks or just to reuse the topology by resetting weights and bias before launching a new training process.

#### **6 Complements to Leverage HPC/Cloud Infrastructures**

EDDL, ECVL and the back end are designed to be deployed on HPC and cloud infrastructures to distribute the workload of training and inference processes by following the data parallelization programming paradigm. Both libraries include specific functions to enable distributed learning. The distribution of the workload on multiple worker nodes is not directly performed by ECVL and EDDL. Instead, the libraries are complemented with workflow managers like COMPSs [6] and StreamFlow [7], specially designed for HPC/cloud environments, that manage the workload distribution of training and inference processes in combination with resource managers like SLURM [32]. To leverage hybrid HPC + cloud architectures, pre-built Docker images with all the components of the DeepHealth toolkit are ready to be deployed in scalable environments orchestrated by Kubernetes [33].

The DeepHealth toolkit is being tested on multiple HPC, cloud and hybrid HPC + cloud infrastructures to validate its ability to exploit a wide variety of architectures. The infrastructures considered in the DeepHealth project are:

The Marenostrum supercomputer and the ODH platform are similar in terms of use; both are HPC infrastructures and both are ready to hold private clouds. The hybrid cloud facilitates vertical and horizontal scalability, providing good adaptability to different situations and uses, the possibility of deploying applications and work with data that can be shared between clouds, improving the performance of the workload. The private part of the hybrid cloud can be deployed on Marenostrum and ODH, as well as in the on-premise computer cluster of any SME. On the other hand, the MANGO cluster is an FPGA-specific computing infrastructure that is being used to evaluate some use cases of the DeepHealth project.

It is worth noting that these infrastructures offer a wide range of computing environments at different levels:

• High number of CPU computing nodes on Marenostrum, multi-GPUs nodes on the ODH platform and FPGAs in the MANGO cluster.

#### *6.1 Use of Docker Images Orchestrated by Kubernetes*

The hybrid cloud platform provided by TREE Technology is a computing environment that offers the possibility of combining public and private clouds, allowing the deployment of applications and work with data that can be shared between them. This solution was built using Kubernetes technology [33], a distributed container and microservice platform that orchestrates computing, networking and storage infrastructure to support user workloads.

Software containers demonstrated to provide a good way to bundle and deploy applications. However, as system complexity increases (e.g., complex multi-component software applications, multi-node clusters) running deployments become increasingly difficult. Kubernetes supports the automation of much of the work required to maintain and operate such complex services in a distributed environment. The objective of this hybrid environment is to dynamically operate in different Kubernetes clusters running on several public clouds and onpremise infrastructures. Different Kubernetes clusters can have different hardware configurations, that is, they can have different memory and CPU settings, with or without GPUs. Once the different clusters are deployed, both in public and private clouds, it is necessary to orchestrate all the resources. For this ecosystem to work properly and be able to be coupled in the global scheme, two stages must be taken into account:

The proposed hybrid cloud based on Kubernetes is a complex system, and its scalability is determined by several factors, like the number and type of nodes in a pool of nodes, the number of Pods available (Pods are the minimum deployable computer units that can be created and managed in Kubernetes), the number of services or back ends behind a service and how resources are allocated. Usually, in a public cloud, the concept of autoscaling is available, which refers to the possibility of scaling the resources of a cluster in a self-managed manner. In the DeepHealth project, the public cloud has been configured with this autoscaling option, while for the private cloud there is no scaling policy in relation to machines.

Concerning the automatic deployment of the DeepHealth toolkit in any cloud configuration, and regardless of the complexity level of the computing infrastructures that any development team of AI-based solutions may have on hand, a set of pre-built CUDA-enabled DeepHealth Docker images, including all the components of the toolkit, are ready to be used on GPU-enabled computing resources to accelerate compute-intensive operations. All the DeepHealth Docker images available in the DockerHub [24] are CUDA-enabled and provide prebuilt binaries of the libraries along with all their dependencies, such that these images can be used to create Docker-ready applications. In addition, a *toolkit* image flavour is also provided to support the developers of applications/platforms directly integrating the EDDL and the ECVL, who may prefer the C++ or Python API. These Docker images are built on the *devel* flavour of the NVIDIA/CUDA images, and add a full DeepHealth build configuration to provide a ready-to-use compilation environment for applications.

For simplified scalable deployments on cloud computing resources, a Kubernetes [33] deployment of the DeepHealth toolkit, with the web service configured, has been created and made available. The deployment automatically configures a server for the DeepHealth front end and all the back-end components (i.e., web service, worker, database, job queue, and static content server) in a flexible and scalable way. In fact, once a deployment is created, the available processing capacity can be dynamically scaled using some of the standard features of Kubernetes, such as configuring the required number of worker replicas to achieve the required throughput. The Kubernetes deployment of the DeepHealth toolkit is packaged as a Helm chart for easy deployment [35]. For simpler use cases that do not have particular scalability requirements, a Docker-compose deployment is also available. This configuration cannot distribute work over multiple nodes, but it can be trivially deployed on a single node and thus is well suited for small workloads and exploratory or development work.

#### *6.2 COMPSs*

COMPSs [6] offers a portable programming environment based on a task model, whose main objective is to facilitate the parallelization of sequential source code, written in Java or Python programming languages, to run in a distributed and heterogeneous computing environment. In COMPSs, the programmer is responsible for identifying the units of parallelism (named COMPSs tasks) and the synchronization data dependencies existing among them by annotating the sequential source code. The task-based programming model of COMPSs is then supported by its runtime system, which manages several aspects of the application execution and keeps the underlying infrastructure transparent to the programmer. This is a key feature to guarantee the portability of COMPSs applications across a wide range of computing platforms. This will allow the DeepHealth toolkit to be tested and validated within the DeepHealth project in the infrastructures enumerated above. Regarding cloud configurations, the COMPSs runtime is being adapted within the DeepHealth project to support the hybrid cloud infrastructure. COMPSs runtime interacts with the API developed by TREE Technology to deploy workers and distribute the workload on hybrid cloud architectures. The COMPSs runtime is organized as a master-worker structure:

One key aspect is that the master maintains the internal representation of a COMPSs application as a Directed Acyclic Graph (DAG) to express the parallelism. Each node corresponds to a COMPSs task and edges represent data dependencies (and so potential data transfers). Based on this DAG, the runtime can automatically detect data dependencies between COMPSs tasks: as soon as a task becomes ready (i.e., when all its data dependencies are resolved), the master is in charge of distributing it among the available workers, transferring the input parameters before starting the execution. When the COMPSs task is completed, the result is either transferred to the worker in which the destination COMPSs task executes (as indicated in the DAG), or transferred to the master if a barrier synchronization call is invoked. The parallelization of the EDDL training operation has been developed with the COMPSs tasking programming model. Due to the fine grain data dependency synchronization mechanisms supported by COMPSs, two parallel training paradigms are supported: *synchronous*, in which weights are collected and aggregated at the end of each epoch, and *asynchronous*, in which weights are increasingly aggregated as soon as a partial training is completed on the corresponding data set.

COMPSs is perfectly adapted to run in environments managed by SLURM [32], an open-source resource manager widely used in High-Performance Computing data centres to manage job queues and job allocation of incoming tasks to servers. The Ecole Polytechnique Fédérale de Lausanne (EPFL) has enhanced the core version of the SLURM resource manager with novel plugins that enable energy- and performance-aware task allocation for CPU- and memory-intensive tasks in order to increase the efficiency (in terms of performance per watt) of multiple tasks when running simultaneously on the same server and cluster. EPFL do so by proposing the use of graph-based techniques and reinforcement learning, which are low overhead and do not impact the execution time of applications. SLURM can interact with COMPSs in order to launch multiple instances of applications in a coordinated way in an HPC infrastructure, creating a separation of concerns between resource managers, while still working in a coordinated way.

#### *6.3 StreamFlow*

StreamFlow is a novel Workflow Management System (WMS) explicitly designed in the DeepHealth project, supporting AI pipeline design and execution in different execution environments, including hybrid HPC + cloud and multi-cloud infrastructures. The portability of AI pipelines on critical data across different infrastructures is crucial for the sustainability of DeepHealth foreground technologies. To address this issue, *OpenDeepHealth* embraces StreamFlow. The ability of StreamFlow to handle sequences of computational steps makes it possible to describe a complex application as a workflow and annotate each step with an execution plan potentially targeting different nodes, e.g., selecting GPU nodes when needed, spawning across multiple sites—e.g., allowing transparent access to OCCAM and HPC4AI clusters. The idea behind this approach is that the ability to deal with hybrid workflows (i.e., to coordinate tasks running on different execution environments) can be a crucial aspect for performance optimization when working with massive amounts of input data and different needs in computational steps. Accelerators like GPUs and, in turn, different infrastructures like HPC and clouds, can be used more efficiently by selecting the execution plan that best suits the specific computational needs of each ML application developed in the project.

The StreamFlow framework is a container-native WMS written in Python. It has been designed to explore the potential benefits deriving from waiving two common properties of existing WMSs that can prevent them from fully exploiting the potential of containerization technologies. Instead of forcing a one-to-one mapping between workflow steps and Docker containers, StreamFlow allows the execution of tasks in potentially complex, multi-container environments. This allows support for concurrent execution of multiple communicating tasks in a multi-agent ecosystem, e.g., a SPMD application implemented with MPI or a COMPSs-based distributed training. StreamFlow relaxes the requirement of a single shared data space among all the worker nodes, allowing to spread different steps of a single workflow on multiple, potentially federated architectures without forcing direct communication channels among them. Moreover, StreamFlow clearly separates the definition of the AI pipeline, described as a declarative workflow, from the description of the runtime environment in charge of executing it, enforcing a separation of concerns. This allows taking advantage of using the most efficient infrastructures for the specific purpose of complex AI pipelines without burdening the AI experts with the configuration and management complexity of such infrastructures. At a very high level, an AI pipeline can comprise a training step, which usually requires very high computational power and distributed programming techniques to handle huge datasets, and an inference step, in which a fully trained model should be directly reachable from one or more user applications. StreamFlow can orchestrate the execution of the AI pipeline, targeting the training step on HPC facilities, e.g., by using EDDL with COMPSs distributed runtime, and the inference step on the cloud cluster, e.g., by leveraging EDDL Docker containers deployed on a Kubernetes infrastructure.

#### **7 Conclusions**

The DeepHealth toolkit is presented here as a new and emerging software framework that provides European industry and research institutions with deep learning and computer vision functionalities. To cope with huge and constantly growing data sets, the toolkit has been designed to leverage hybrid and heterogeneous HPC + cloud architectures in which either all or some of the worker nodes are equipped with hardware accelerators (e.g., GPUs, FPGAs). The distributed execution of learning and inference processes is done by the runtime of the DeepHealth toolkit in a transparent manner to the common user, i.e., computer and data scientists who do not need a deep understanding of parallel programming, HPC, deep learning or cloud architectures.

The two libraries at the core of the toolkit can be easily integrated into existing software applications/platforms that European companies (SMEs and large industry) have developed to deploy AI-based solutions in any sector (e.g., decision support systems that clinicians can use to diagnose), and can be used to boost the development of new platforms and solutions. All the components of the toolkit are free and open-source software available on public repositories.

In order to foster the use of the DeepHealth toolkit, the authors have introduced the potential user to all the toolkit components and how to integrate the libraries in existing or new software applications. It is worth mentioning that, thanks to prebuilt Docker images including all components with all dependencies satisfied, data scientists only need to provision Docker containers according to their needs.

The toolkit constitutes a contribution from Europe in Artificial Intelligence and smart big data analytics. Besides all the features introduced in this chapter (free and open-source framework, easy to use, portable to different architectures, wide application scope), it contributes to reducing the bottlenecks in turning AI into an enabling technology for Science (e.g., provides a way to reduce the complexity of numerical methods used in scientific environments), bringing closer the separate worlds of AI and HPC. Furthermore, it is expected to boost the adoption of AI and HPC technologies by the industry. The toolkit paves the way towards the offering of AI coupled with HPC as a service, which could be a game changer aspect in order to reach a greater number of companies. On the one hand, it offers improvements for companies that only have temporary needs for high-performance computing resources, which will be able to improve their productivity by developing their own AI solutions, and on the other hand, it could unlock the development of novel applications that need to run computationally intensive processes regularly.

**Acknowledgments** This chapter describes work undertaken in the context of the DeepHealth project, "Deep-Learning and HPC to Boost Biomedical Applications for Health", which has received funding from the European Union's Horizon 2020 research and innovation programme under grant agreement No 825111.

#### **References**

**Open Access** This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

# **Applying AI to Manage Acute and Chronic Clinical Condition**

**Rachael Hagan, Charles J. Gillan, and Murali Shyamsundar**

**Abstract** Computer systems deployed in hospital environments, particularly physiological and biochemical real-time monitoring of patients in an Intensive Care Unit (ICU) environment, routinely collect a large volume of data that can hold very useful information. However, the vast majority are either not stored and lost forever or are stored in digital archives and seldom re-examined. In recent years, there has been extensive work carried out by researchers utilizing Machine Learning (ML) and Artificial Intelligence (AI) techniques on these data streams, to predict and prevent disease states. Such work aims to improve patient outcomes, to decrease mortality rates and decrease hospital stays, and, more generally, to decrease healthcare costs.

This chapter reviews the state of the art in that field and reports on our own current research, with practicing clinicians, on improving ventilator weaning protocols and lung protective ventilation, using ML and AI methodologies for decision support, including but not limited to Neural Networks and Decision Trees. The chapter considers both the clinical and Computer Science aspects of the field. In addition, we look to the future and report how physiological data holds clinically important information to aid in decision support in the wider hospital environment.

**Keywords** Healthcare · Predictive analytics · Artificial Intelligence · Clinical decision support

The chapter relates to the technical priorities Data Analytics of the European Big Data Value Strategic Research and Innovation Agenda [1]. It addresses the horizontal concern Data Analytics of the BDV Technical Reference Model and addresses the vertical concerns of Big Data Types and Semantics. The chapter further relates to the Reasoning and Decision Making cross-sectorial technology enablers of the AI, Data and Robotics Strategic Research, Innovation and Deployment Agenda [2].

R. Hagan (-) · C. J. Gillan · M. Shyamsundar

Queen's University Belfast, Belfast, Northern Ireland, UK e-mail<EMAIL_ADDRESS>
#### **1 Overview**

The chapter begins by discussing Intensive Care medicine and the types of machines and data that is being recorded continuously, and thus producing 'Big Data'. It proceeds to explain some of the challenges that can arise when working with such data, including measurement errors and bias. The subsequent section explains some of the common methodologies used to provide predictive analytics with examples given for both acute and chronic clinical conditions, and discuss our own work for the promotion of lung protective ventilation, to highlight the accuracies that can be achieved when pairing health 'Big Data' with common machine learning methodologies. The chapter concludes by discussing the future of this field and how we, as a society, can provide value to our healthcare systems by utilizing the routinely collected data at our disposal.

#### **2 Intensive Care Medicine and Physiological Data**

Intensive Care Units (ICU) offer expensive and labor-intensive treatments for the critically ill and are therefore a costly resource for the health sector around the world. They can also be referred to as Intensive Therapy Units (ITU) or Critical Care Units (CCU). UK estimates in 2007 highlighted that intensive care in the NHS costs £719 million per year [3]. Comparably, American studies reported in 2000 have shown that median costs per ICU stay can range between \$10,916 and \$70,501 depending on the length of stay [4]. A typical length of stay varies depending on the condition of the patient with studies showing the mean length of stay being 5.04 days [3], while the condition of the patient can change quickly and sometimes unpredictably.

Patients will normally be admitted to intensive care after a serious accident, a serious condition such as a stroke, an infection or for surgical recovery. Throughout their stay in ICU, these patients are monitored closely due to their critical condition, and on average require one nurse for every one or two patients. Many devices and tests may be used to ensure the correct level of care is provided. Patients will be placed on machines to monitor their condition, support organ function and allow for the detection of any improvements or deterioration. The functions of these machines can vary: from the monitoring of vital parameters such as heart rate via the patient monitor to the use of mechanical ventilators that provide respiratory function when a patient is not able to do so themselves.

The workload on the clinical staff in ICU is intense and so utilizing Big Data analytics will allow for healthcare providers to improve their efficiency through better management of resources, detection of decompensation and adverse events, and treatment optimization, among many benefits to both patient outcomes and hospital costs [5].

#### *2.1 Physiological Data Acquisition*

Health data recorded in ICU can be classified as 'Big Data' due to the volume, complexity, diversity and timeliness of the parameters [6], the aim being to turn these 'Big Data' records into a valuable asset in the healthcare industry.

As highlighted, patients requiring critical care are placed on numerous monitors and machines to help with and provide their care. Equipment can include, but is not limited to [7]:

These machines are all monitoring continuously (24×7) and thus representing one example of the emerging field of Big Data. It is important to wean patients off these machines as quickly as possible to avoid dependency and to lower the risk of infection. In addition to the organ support machines, the electronic health record includes laboratory data, imaging reports such as X-rays and CT scans, and daily review record.

#### **2.1.1 Time Series Data**

The human physiologic state is a time-dependent picture of the functions and mechanisms that define life and is amenable to mathematical modeling and data analysis. Physiology involves processes operating across a range of time scales resulting in different granularities. Parameters such as heart rate and brain activity are monitored on the millisecond level while others such as breathing have time windows over minutes and blood glucose regulation over hours. Analysis of instantaneous values in these circumstances is rarely of value. On the other hand, application of analytical techniques for time series offers the opportunity to investigate both the trends in individual physiological variables and the temporal correlation between them, thereby enabling the possibility to make predictions.

Features can also be extracted from these time series using packages such as the Python *tsfresh* software toolkit to use as input into Machine Learning models and to gain further insights into the relationship between the parameter and time [8]. By analyzing the time series we can make predictions on the future trajectory and alert care givers of possible issues in order to prevent complications.

Prior research has shown that these streams of data have very useful information buried in them, yet in most medical institutions today, the vast majority of the data collected is either dropped and lost forever, or is stored in digital archives and seldom reexamined [9].

In the past number of years there has been a rapid implementation of Electronic health records, or EHRs, around the world. EHRs are a common practice to record and store real-time, patient health data, enabling authorized users to track a patient from initial admission into the hospital, their deterioration/improvement, diagnoses, and all physiological parameters monitored and drugs given across all healthcare systems.

#### **2.1.2 Publicly Available Datasets**

As part of a global effort to improve healthcare, numerous institutions have put together publicly available data sets based on their EHRs for people to use in research; enabling visualization, analysis, and model development.

One of the best-known and commonly used publicly available databases is MIMIC, the Multiparameter Intelligent Monitoring in Intensive Care database. Produced from the critical care units of the Beth Deaconess Medical Centre at Harvard Medical School, Boston, this large, freely available database consists of de-identified health-related data associated with over 40,000 patients who were in critical care between 2001 and 2012 [10]. After being approved access, users are provided with vital signs, medications, laboratory measurements, observations, and charted notes. Furthermore, waveforms are available for use, along with patient demographics and diagnoses.

PhysioNet offers access to a large collection of health data and related opensource software, including the MIMIC databases [11]. All data recorded in these publically available databases are anonymous, ensuring no patient can be identified from their data.

#### **3 Artificial Intelligence in ICU**

The broad field of Data Science has emerged within the discipline of Computer Science over the past 10 years, approximately. Its roots arguably lie in the work of Fayyad et al. [12], which defined a pipelined process for the extraction of abstract models from data using an amalgam of techniques, including Statistics and Machine Learning. Artificial Intelligence (AI) is the ability of a computer system to perform tasks that aim to reach the level of intelligent thinking of the human brain through applying machine learning methodologies, further details are discussed in the methodologies section.

#### *3.1 Challenges*

As in majority of real-world applications, there comes a series of challenges that arise when applying Big Data-driven analytics to such sensitive and intense data. We must aim to build trust and quality between the decision support tools and the end users. It is always important to question if there is real clinical impact from carrying out such work and exploring AI methodologies for solving particular problems.

## **3.1.1 Data Integrity**

Before building any Big Data-driven analytic system, it is important to ensure data has been collected and preprocessed correctly, with any errors handled accurately. It is crucial to ensure data harmonization, the process of bringing together all data formats from the different machines and tests into one database. Without this step, the algorithms will produce unreliable results, which in turn could result in a critically dangerous implementation and a lack of trust in the system. Errors in the data can be due to a variety of reasons:

We must ensure to implement appropriate techniques such as imputation, smoothing and oversampling to prevent errors in our data and build trust with the user.

#### **3.1.2 Alert Fatigue**

When building decision support tools, it is crucial to ensure the integrity of alerts raised. Too many can result in alert fatigue, leading to alarms being switched off or ignored. Research has shown that there is on average 10.6 alarms per hour, too many for a single person to handle amidst their already busy work schedule [15].

In addition, 16% of health care professionals will switch off an alarm and 40.4% will ignore or silence the alarm [16].

This influx of alarms resulting in the monitoring system being turned off will lead to vital physiological problems being missed. We need to be confident that our systems are only producing true alerts and that false alerts generated are minimized.

It has further been highlighted that reducing the number of alerts repeated per patient will help reduce override rates and fatigue [17].

#### **3.1.3 Bringing AI Systems to Clinical Trials**

While we know these AI models, derived from Big Data-driven predictive analytics, can provide value to the healthcare industry, we must be able to test them in real time in order for them to be implemented and used throughout the world. This requires a clinical trial to be carried out using the system on patients in real time. While there exist the SPRIRT 2013 and CONSORT 2010 checklists for clinical trials, covering checklists for what you intended to do and what you actually did, respectively, neither of these include steps for AI implementation [18].

With the rise of AI, these guidelines have been extended in 2020 to include steps for reporting the quality of data, errors, clinical context, intended use, and any human interactions involved. These additional steps allow reviewers to better evaluate and compare the quality of research and thus systems created in the future [19, 20]. Other authors are further expanding these checklists to include reporting for diagnostic accuracies and prediction models [21, 22].

Researchers have analyzed the current work that has been published in this area [23]. With 93% of papers exploring model development to demonstrate potential for such systems, and only 5% validating these models with data from other centers. A further 1% of the work currently published reports on models that have actually been implemented and tested in real-time hospital systems and the final 1% of research have integrated their models in routine clinical practice which has proven to work. This summary highlights the large potential for the future of AI in healthcare systems, with huge opportunities for bringing forward accurate models to real-world clinical trials.

#### *3.2 AI Methodology*

Machine Learning methodologies are commonly used to enable machines to become 'intelligent' and can be classified as being supervised, predicting using a labeled dataset, i.e., a known output or target, or unsupervised, i.e., finding patterns or groupings in unlabeled data [24]. Common methodologies are utilized across the board for predictive and analytical purposes. This chapter focuses on commonly used supervised learning techniques; however, unsupervised methods can further be used to understand data. We can categorize supervised learning techniques into regression and classification models. Regression techniques aim to find relationships between one dependent variable and a series of other independent variables, e.g., a time series as previously discussed is common in physiological data. Classification techniques on the other hand attempt to label outcomes and draw conclusions from observed values, e.g., if patient has disease or not.

The question of whether the numerical models that are generated can actually be understood by humans has become a hot research topic. In the United States, DARPA has conducted significant investigative research in this field [25]. Other research teams have begun to define the concept of explainable AI with respect to several problem domains. They identify three classifications: opaque systems that offer no insight; interpretable systems where mathematical analysis of the algorithm is viable; and comprehensible systems that emit symbols enabling userdriven explanations of how a conclusion is reached [26].

We know from previous research that utilizing Big Data within the ICU can lead to many benefits to both patient and hospital. We can not only greatly improve the care given and thus patient outcomes, see Table 1, but also reduce hospital costs [5] and the stress levels of care givers [27]. McGregor and colleagues have demonstrated the viability of monitoring physiological parameters to detect sleep apnea in neo-natal ICU leading to the software architecture of the IBM InfoSphere product, which has now been extended into a Cloud environment (the Artemis project) making the functionality available at remote hospital sites [28, 29].

Patient deterioration can often be missed due to the multitude of and complicated relational links between physiological parameters. AI-driven multivariate analysis has the potential to ameliorate the work load of ICU staff. Multiple studies have shown AI to be comparable to routine clinical decision making, including ECG analysis, delirium detection, sedation, and identification of septic patients [28, 37– 39].

AI-driven predictive analytics within healthcare most commonly uses supervised learning approaches, due to which we aim to base algorithms and decisions on previous examples and train our models with accurate outcomes, in particular regression analysis is used for time series data.

Below we review the more widely adopted Machine Learning methodologies developed [31], and examples are highlighted in Table 1, where these have been used in previous research.

#### **3.2.1 Expert Systems**

In a rule-based expert system, also known as a knowledge-based clinical decision support system, knowledge is represented by a series of IF-THEN rules that are created by knowledge acquisition, which involves observing and interviewing human experts and finalizing rules in format 'IF X happens THEN do Y'. These rules cannot be changed, learnt from or adapted to different environments, meaning the human experts must manually monitor and modify the knowledge base through careful management of the rules. Expert systems allow us to view and understand each of the rules applied by the system.

The systems can take over some mundane decision tasks and discussions of health care professionals, saving vital time and money. An example expert system was created for Diabetes patients to provide decision support for insulin adjustment based on simple rules depending on the regimen that patients were placed on [32].

#### **3.2.2 Decision Trees**

A decision tree can be used to visually represent a series of decisions used to reach a certain conclusion, useful when exploring medical reasons behind decisions made. The tree starts at the root, asking a specific question to split the data by the given condition. The tree then splits into branches and edges at each node representing the input variables, continually splitting by conditions until the final decision is achieved at the leaf node or the output variable.

They can also be referred to as Classification and Regression Trees (CART) as they can solve both classification and regression problems. A classification tree will arrive at a binary condition, leaf node, i.e., patient survives or not, whereas the regression trees will predict a certain continuous value, i.e., the heart rate of a patient.

The tree will not only explore the conditions used to split the data at each decision but also the features used and which features are the most significant at splitting the data, added as top-level nodes. Researchers have utilized simple decision tree models for the classification of patients with diabetes, among other disease states. Features include age, BMI, and both systolic and diastolic blood pressure of the patient to arrive at a decision whether the patient has diabetes or is healthy. Figure 1 shows how the features are split and decisions are made, in this circumstance [33].

When building decision tree models it is important to monitor the maximum depth of the tree to avoid overfitting and lengthy training times.

#### **3.2.3 Ensemble Methods**

To achieve the greatest predictive performance when working with complex problems, ensembles can be created. The decision trees are combined in different ways

**Fig. 1** Decision tree for the classification of diabetes [20]

**Fig. 2** Decision trees and ensemble methods

depending on the methodology used. The methods can be categorized as bagging or boosting (Fig. 2).

Boosting methods build the trees in a sequential way; for each predicted value multiple models or decision trees are made using different features and combinations of features, then weights are given to these models based on their error so that the final prediction is the most accurate. AdaBoost is an example of this method where very short trees are produced and higher weights are given to more difficult decisions. GradientBoosting further combines boosting with gradient descent, allowing for the optimization of loss functions.

In contrast, for bagging methods, each model is given equal weights, they are combined in parallel and all of the predictions are averaged to get a final, most accurate decision, examples include Bagging and Extra Trees. Bagging can be extended to the RandomForest algorithm by randomly selecting the features used, and decision trees are built to have as many layers as possible.

#### **3.2.4 Neural Networks**

A neural network saves a lot of time when working with large amounts of data by combining variables, figuring out which are important and finding patterns that humans might not ever see. The neural network is represented as a set of interconnected nodes, connected by neurons.

They feed the weighted sum of the input values through an activation function, which takes the value and transforms it before returning an output. These activation functions in turn improve the way the neural network learns and allows for more flexibility to model complex relationships between the input and output (Fig. 3).

The neural network can be described as 'Deep Learning' when it has multiple hidden layers between the input and output layers. The neural network learns by

**Fig. 3** Neural network structure

figuring out what it got wrong and working backwards to determine what values and connections made the prediction incorrect.

Additionally, there are different types of neural networks, and the list continues to expand as researchers propose new types. There are feedforward neural networks, where all the nodes only feed into the next layer from initial inputs to the output. Recurrent neural networks (RNN) make it possible to feed the output of a node back into the model as an input the next time you run it. Nodes in one layer can be connected to each other and even themselves. Furthermore, they work well with sequential data as they remember previous outputs. The long short-term memory form of the RNN enables the model to store information over longer periods of time, ideal for modeling time series data. Additionally, there are convolutional neural networks (CNN) which look at windows of variables rather than one at a time. Convolution applies a filter, or transformation, to the windows to create features. When working with large databases, pooling is a technique that takes a huge number of variables to create a smaller number of features. The neural network will then use the features generated by convolution and pooling to give its output.

The key parameters to distinguish neural networks are the number of layers and the shape of the input and the output layers.

Researchers have utilized many different formats of neural networks for prediction problems. A medical decision support tool for patient extubation was developed using a multilayer perception artificial neural network, a class of feedforward neural networks. The input layer consisted of 8 input parameters, defining 17 perceptions, and 2 perceptions in the output layer for prediction output. Perceptions can be described as a classifying decision. They explored the change in performance based on the number of perceptions in the hidden layer: 19 producing highly accurate results [34]. Other studies have shown that RNNs produce accurate results for the prediction of kidney failure [35] and CNNs have shown promise in the prediction of atrial fibrillation in ECG analysis [36],

However, it is difficult to explain predictions from neural networks to healthcare professionals due to having to understand a particular weight as a discrete piece of knowledge, although work is being done in the area [37].

With common limitations reoccurring, such as generalization, dataset size, noisy and unbalanced data, we highlight the importance of continuing research and building of larger datasets, across multiple centers, and further the exploration of methodologies to smooth noisy data in order to advance the work of producing accurate systems that can be implemented in real-world healthcare settings.

 Machines such as X-rays and CTs can produce images of internal organs to provide a greater insight into patient state. In addition, ECG waveforms and brain waves can be analyzed for diagnoses. Thus, signal processing is an integral part of understanding physiological data and getting the full picture of patient state.

Furthermore, Machine Learning methodologies can be used for natural language processing to analyze lab notes and patient charts in order to summarize and detect common words and phrases used by care givers, in turn automating the process of flipping through hundreds of pages of records and saving vital time.

# **4 Use Case: Prediction of Tidal Volume to Promote Lung Protective Ventilation**

Around 44.5% of patients in ICU are placed on mechanical ventilation to support respiratory function at any given hour [52]. However, the delivery of high tidal volume values can often lead to lung injury. Tidal volume being the amount of air delivered to the lungs; it is common knowledge amongst critical care providers that tidal volume values should be no greater than 8 ml per kg of ideal body weight. In our recent work we explored regressors for ensemble methods and the long shortterm memory (LSTM) form of neural networks to predict tidal volume values to aid in lung protective ventilation [53].

Data acquisition took place at the Regional Intensive Care Unit (RICU), Royal Victoria Hospital, Belfast, over a 3-year period and the VILIAlert system was introduced [54]. The data streams were monitored against the thresholds for lung protective ventilation and if thresholds were breached continuously, an alert was raised. We then turned our attention to predicting these alerts with the aim of preventing violations and protecting the patient's lungs. The VILIAlert system ran for nearly 3 years, recording minutely tidal volume values for almost a thousand patients.

As discussed, noisy signals are common in ICU data. Time series often needs to be filtered to remove noise in the data and produce smooth signals. Methods such as moving average and exponential smoothing can be applied to the data to extract true signals, such as the work we carried out to extract true tidal volume trends [53]. Figure 4 shows how smoothing the time series, shown in blue, removes anomaly in the data, the large jumps, and extracts the true patient trend as shown in red. This work is related to the efforts of the international project known as the Virtual Physiological Human (http://www.vph-institute.org), which seeks to use individualized physiology-based computer simulations in all aspects of prevention, diagnosis, and treatment of diseases [55]. This computational approach has provided immediate insight into the COVID-19 pandemic [56].

We compare multiple regressor ensemble methods for initially predicting 15 min ahead. For each patient, we use the *tsfresh* toolkit to extract features to use as input into the regressor models in order to predict one time bin ahead and report the RMSE between the true observed values and the predicted values from our models. Table 2 reports RMSE calculated for 8 of the patients using each of the ensemble methods. In all models the maximum number of trees is set to 10. We can compare the depth of the bagging method trees: RandomForest being 32 ± 5, ExtraTrees being 37 ± 4, and Bagging being 33 ± 5. In contrast, the boosting methods: AdaBoost and GradientBoosting, set trees of depth four by default. As expected, increasing the number of trees decreases the RMSE.

It is important to take computational time into consideration when choosing algorithms to make predictions in real time. From our experiments we found AdaBoost to give the best trade-off between RMSE and computation time.

**Fig. 4** Time series of a patient's tidal volume profile. Raw minutely values in blue, 15-min averaged bins in red

**Table 2** Comparison of regressor ensemble models performance for the prediction of patient's tidal volume one time step ahead

One might expect that predicting further ahead in time would lead to larger RMSE values, however, the change in RMSE is small. We therefore explore using AdaBoost regression for the prediction of tidal volume values up to 1 h ahead, finding very little increase in RMSE values across patients.

As described in this chapter, a benefit of using ensemble models made from decision trees is that we can visualize the features and decisions used to make decisions. Figure 5 is one of the decision trees created when using the AdaBoost method.

The features were extracted by *tsfresh* as the most significant features for this problem. It is interesting here to discuss what these features can mean for our

problem domain. The Ricker Wavelet is used to describe properties of a viscoelastic homogeneous media and the Friedrich coefficient aims at describing the random movement of a particle in a fluid. We can hypothesize from this finding that the amount of fluid in the lungs would be an impacting factor in how a patient's tidal volume can change over time.

A comparison is then made applying long short-term memory neural networks to the same problem: predicting tidal volume 1 h ahead. Two models are created: ModelA has one hidden layer and ModelB has three hidden layers, with a 20% dropout layer between the second and third layers to avoid overfitting. In contrast to the regressor models that work with features extracted from the time series, the LSTM models use the time series values directly; requiring 70% of the time series to train the models. Each layer in our LSTM models has 50 nodes and both models use 20 input points to predict 4 ahead.

The RMSE values for our LSTM models are significantly greater than the AdaBoost method for predicting 1 h ahead, so we deem AdaBoost the better method of the two for this problem.

The VILIAlert system alerted when four consecutive bins were greater than the 8 ml/kg tidal volume threshold. These alert times were stored in the database. We can thus work out the accuracy of our models to predict these alerts, showcasing the possibility of preventing threshold breaches and preventing injury. Table 3 highlights the predictive accuracy of the AdaBoost model for the 8 patients. Total Alerts being the total number of alerts recorded by the VILIAlert system, TP being the true positives: the alerts that would have been predicted by the model, and FN being false negatives: the alerts that would not have been predicted. The accuracy is then calculated using:

$$\text{Accuracy} = \frac{\text{TP}}{\text{TP} + \text{FN}.}$$

For the 84 alerts that were generated for patient 1, 81 would have been predicted using our models and thus those threshold breaches could have been prevented. These results showcase how Machine Learning algorithms, when paired with big

**Table 3** Prediction accuracy of alerts using AdaBoost

data, can provide value in preventing lung injury during mechanical ventilation of intensive care patients.

#### *4.1 The ATTITUDE Study*

The ATTITUDE study operated by Queen's University, Belfast, and funded by The Health Foundation UK, aims to develop a clinical decision support tool to improve weaning protocols commonly used in clinical practice, and further understand the barriers in implementing evidence-based care from these tools tested in a proof-ofconcept study carried out at the Royal Victoria Hospital ICU, Belfast. Improving patient care, outcomes, and mortality by reducing the duration of weaning can lead to reduced hospital stays and costs, and this study aims to find out if the use of clinical decision support tools can improve the quality of critical care practices.

#### **5 Future of ML and AI in ICU**

The methodology discussed in this chapter can, and must be, explored with various and extensive types and volumes of data, to investigate more disease states and clinical conditions. This data is currently being recorded worldwide in what are known as Electronic Health Records and these hold valuable insights which must be utilized to improve healthcare going forward.

From the European Big Data Value Strategic Research and Innovation Agenda [1] we understand the importance of Big Data and utilizing it to benefit both the economy and society. By exploring the already available EHRs we can provide societal benefit by improving patient outcomes and saving lives, and further economic benefits of saving millions in hospital costs, through shorter lengths of stays and disease prediction, among others. Healthcare, being one of the most important and largest sectors, can greatly impact the agenda of a datadriven economy across Europe. Data-driven predictive analytics, built using the methodologies discussed in the chapter, can produce clinical decision support tools, allowing for advanced decision making or automation of procedures. The Machine Learning methodologies can further provide greater insights into patient states and inform healthcare professionals with new information or possible reasoning that would not have been caught by a human. These new insights result in further research questions that can be explored.

This chapter can be aligned with the Big Data Reference model in various ways. The data recorded in ICU is extensive and in various different formats, from structured data and time series to imaging and text inputs. Advanced visualization tools can be used to add value to data by presenting it in user-friendly ways. Data analytics and Machine Learning can be applied for prediction and reasoning of disease states. There exist further protocols and guidelines for the handling of patient information, ensuring efficient data protection and management.

The new AI, data and robotics partnership highlights the value opportunities that exist when transforming the healthcare sector by applying AI to produce value-based and patient-centric care in areas such as pandemic response, disease prevention, diagnosis decision support, and treatment [2].

High-performance computing is an integral part in deploying real-time predictive analytic models in intensive care. We must ensure our machines can process the data efficiently and quickly. Utilizing parallelism will provide speed up data processing and model predictions.

This chapter has explored the use of Big Data-driven analytics for acute and chronic clinical conditions to provide value to healthcare services. While there exists vast research carried out in certain disease states, such as Sepsis [30, 40–42], work is needed to provide greater in-depth analysis and insights into patients' complex physiologic state while in such critical conditions. Recent developments have led to a greater acceptance and excitement in this field, resulting in updated guidelines for testing AI models in real-life clinical trials to promote worldwide acceptance of the use of AI in healthcare.

**Acknowledgments** Rachael Hagan acknowledges funding for a PhD studentship from the Department for the Economy, Northern Ireland, and Dr. Murali Shyamsundar is an NIHR Clinical Scientist fellow.

#### **References**

**Open Access** This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

# **3D Human Big Data Exchange Between the Healthcare and Garment Sectors**

**Juan V. Durá Gil, Alfredo Remon, Iván Martínez Rodriguez, Tomas Pariente-Lobo, Sergio Salmeron-Majadas, Antonio Perrone, Calina Ciuhu-Pijlman, Dmitry Znamenskiy, Konstantin Karavaev, Javier Ordono Codina, Laura Boura, Luísa Silva, Josep Redon, Jose Real, and Pietro Cipresso**

**Abstract** 3D personal data is a type of data that contains useful information for product design, online sale services, medical research and patient follow-up.

Currently, hospitals store and grow massive collections of 3D data that are not accessible by researchers, professionals or companies. About 2.7 petabytes a year are stored in the EU26.

In parallel to the advances made in the healthcare sector, a new, low-cost 3D body-surface scanning technology has been developed for the goods consumer sector, namely, apparel, animation and art. It is estimated that currently one person is scanned every 15 min in the USA and Europe. And increasing.

The 3D data of the healthcare sector can be used by designers and manufacturers of the consumer goods sector. At the same time, although 3D body-surface scanners

J. V. Durá Gil (-) · A. Remon Instituto de Biomecánica de Valencia, Universitat Politècnica de València, Valencia, Spain e-mail<EMAIL_ADDRESS>
I. M. Rodriguez · T. Pariente-Lobo ATOS Research and Innovation, Madrid, Spain

S. Salmeron-Majadas Atos IT Solutions and Services, Irving, TX, USA

A. Perrone · C. Ciuhu-Pijlman · D. Znamenskiy Philips Research, Eindhoven, Netherlands

K. Karavaev · J. O. Codina ELSE Corp Srl, Milan, Italy

L. Boura · L. Silva P&R Têxteis S.A., Tamel (São Veríssimo), Portugal

J. Redon · J. Real INCLIVA Research Institute, University of Valencia, Valencia, Spain

CIBERObn ISCIII, Madrid, Madrid, Spain

P. Cipresso Applied Technology for Neuro-Psychology Lab, Istituto Auxologico Italiano, Milan, Italy have been developed primarily for the garment industry, 3D scanners' low cost, non-invasive character and ease of use make them appealing for widespread clinical applications and large-scale epidemiological surveys.

However, companies and professionals of the consumer goods sector cannot easily access the 3D data of the healthcare sector. And vice versa. Even exchanging information between data owners in the same sector is a big problem today. It is necessary to overcome problems related to data privacy and the processing of huge 3D datasets.

To break these silos and foster the exchange of data between the two sectors, the BodyPass project has developed: (1) processes to harmonize 3D databases; (2) tools able to aggregate 3D data from different huge datasets; (3) tools for exchanging data and to assure anonymization and data protection (based on blockchain technology and distributed query engines); (4) services and visualization tools adapted to the necessities of the healthcare sector and the garment sector.

These developments have been applied in practical cases by hospitals and companies of in the garment sector.

**Keywords** 3D body · Blockchain · Clothing · Obesity

#### **1 Introduction**

Three-dimensional (3D) anthropometry refers to the measurement of the human body shape. From 3D data it is possible to obtain more information than from traditional one-dimensional anthropometry, which only uses length, radius or perimeter. Anthropometry plays an important role in industrial design, clothing design and ergonomics where statistical data about the distribution of body dimensions in the population are used to optimize products. Also, changes in lifestyles and nutrition lead to changes in the distribution of body dimensions (e.g. the rise in obesity) and require regular updating of anthropometric data collections.

3D anthropometric data is a key value in scientific and economic sectors like healthcare, consumer goods and professional sports. In the past, the usage of such data was limited by the high price and low precision of 3D scanning technologies. The recent advances in scanning technologies have notoriously widened their usage opportunities. As a result, different actors from different sectors work on the generation of their own dataset of anthropometric data for their own purposes, without taking advantage of similar effort taken by other actors.

In this context, a system to share anthropometric data would allow the different data consumers access to larger datasets and also to reduce the data acquisition costs, thus augmenting their scientific or business opportunities. Besides, data holders would be able to extend the economic benefit reported by their data.

Developing an effective data sharing system is challenging. Success requires a system that gives support to interests from any participant, forming a symbiotic ecosystem. Participants in a data-sharing system can be categorized as data providers and data consumers. Regarding this categorization, the participants will have, at least, the following requirements:

The chapter relates to the technical priorities *data management* and *data protection* of the European Big Data Value Strategic Research and Innovation Agenda [1]. The BodyPass project focuses on 3D anthropometric data management and data protection. In this context BodyPass has generated tools for:

The complete BodyPass ecosystem has been tested in practical cases described in Sects. 5 and 6.

#### **2 The Process to Harmonize 3D Datasets**

Anthropometric data has key value in many economic and social areas, among others healthcare, apparel and furniture design. The relevance of anthropometric data motivated a number of research efforts to standardize and effectively measure a human body [2–5]. But measuring a human body presents important challenges that are difficult to meet: important features such as repeatability, extrapolation of metrics or accuracy [6–9]. Some problems rely on the fact that the human body is not a rigid body, but a soft, articulated and never fully static body. Some other problems rely on the measuring tools, methodologies or different interpretations of the metric. In this context, the standardization efforts taken up to now seem to be insufficient.

The BodyPass project aims to develop a system where different actors share anthropometric data. Data sources in BodyPass belong to different sectors, currently healthcare and consumer goods sectors. As a matter of fact, this means that the measuring methodologies and measuring tools employed may differ for every dataset in the global federated database of the project. In particular, the healthcare sector data is produced by a commercial solution from Philips and is, in essence, an automatic digital measuring tape. Data from the consumer goods sector is provided by IBV and is obtained from its proprietary automatic digital measuring tape [10, 11]. As a result, all measurements in the federated BodyPass database come from digital measuring tools, meaning that there is a high risk of incompatible measurements due to the use different measuring algorithms.

In this context, the project success has required:

#### *2.1 The Anthropometric Data Dictionary*

The anthropometric data dictionary developed in the BodyPass project is based on previous international standardization efforts from organizations such as ISO<sup>1</sup> or ISAK,<sup>2</sup> and it considers the project data consumers' requirements as well. Every metric is defined by the following fields:

	- ID: code that identifies the metric. It is defined as a string that partly includes the metric designation.
	- Designation: It is a self-descriptive name of the metric.
	- Definition: Unambiguous description of the metric.
	- Source: Reference to the company or organization that defined the metric. It usually refers to a standard definition.
	- Other std.: Other standard definitions that can be compatible with the metric as defined.
	- Units: Metrics units employed.

<sup>1</sup> International Organization for Standardization (http://iso.org).

<sup>2</sup> International Society for the Advancement of Kinanthropometry (https://www.isak.global/ Home/Index).

**Fig. 1** Details of the definition of a metric in the BodyPass dictionary

The BodyPass anthropometric data dictionary includes about 100 body metrics and it is available via the project's API and its web interface.3 The web interface includes resources to access all the BodyPass API services. Among them, the resource DataResourceCatalogue [12] permits to visit the list of metrics in the project's anthropometric data dictionary. BodyPass members can use this resource to append, edit or delete metrics to/from the dictionary. In addition, the resource DataProviderMetrics lists the metrics in the dictionary supported by a given data provider. These two resources allow data consumers to consult which metrics are available in the project and also to check which data provider supports a given metric.

To facilitate the dictionary consultation to data providers, IBV lists all the metrics available in their BodyPass database, about 90 metrics, on a webpage.<sup>4</sup> This webpage includes an index and is far more readable than the BodyPass services deployed to this end. For every metric, the web displays the content of all the mandatory fields and also the related images in the dictionary.

Figure 1 shows the description of a metric in the aforementioned webpage. This particular metric is defined in three standardization international publications, two from ISO and one from ASTM.<sup>5</sup> Its definition includes two images that facilitates its understanding.

<sup>3</sup> http://<IP_ADDRESS>:3001/explorer/

<sup>4</sup> http://personales.upv.es/alrego/body/BodyPass\_DigitalMeasuringTape.html

<sup>5</sup> American Society for Testing and Materials (http://astm.org).

#### *2.2 3D Templates*

The procedure to manage on-the-fly templates is the following: anytime a data query arrives to the BodyPass system, a template is created on-the-fly and distributed among the processing nodes. In the next query, a new template is created and used. The template creation will provide it a random parameterization and geometry, making it unique. All the subsequent outputs from the nodes must be compliant with the template's features (namely parameterization and pose). Thus, they all can be aggregated in simple operation, i.e. weighted average of average body surfaces created by different nodes (where the weight assigned to a particular average is related to the amount of individual data used to obtain it). In summary, on-thefly templates will be unique in terms of parameterization and geometry and will determine the parameterization, rigid alignment and pose of the query outputs, i.e., the 3D contents delivered to the client will match the template's features (Fig. 2).

#### **3 Anonymization and Privacy**

In this section, we will briefly address the solutions adopted to protect personal data. We have applied two solutions:

**Fig. 2** Examples of on-the-fly templates with random geometry and topology (A-Pose)

The General Data Protection Regulation (GDPR) in Europe classifies health data as sensible data, and it asks for special protection. For this reason, it was necessary to implement the architectural solution (on-premise data nodes) for processing medical images and to assure that only aggregated data is obtained from the hospitals.

#### *3.1 The Anonymization of Body Scans*

The data node of the Institute of Biomechanics (IBV) stores a database with more than 30,000 3D anonymous avatars and their related metrics. BodyPass provides tools to add new data to the database and to query and retrieve data from the database as well. This means that BodyPass's users are allowed to incorporate their own avatars to the database. To this end, the user just needs a 3D scan and basic data from the subject. During the registration of a new avatar in the database, the avatar is measured using the IBV's automatic digital measuring tape, which provides about 80 metrics from the avatar that are also stored in the database.

A basic requirement of the database is to assure the anonymization of the models. The data stored from every model includes:

From this data, the only one that can be used to identify the model is the avatar id. Given an avatar id, it would be possible to identify the model using marks on the skin (e.g. tattoos) or facial features. For this reason, avatars are stored without texture, so no skin features are stored. In this context, the face represents the only identifiable feature of the avatar.

A straightforward solution would be to remove or blur the face of the avatar, but this would end in a database full of non-human-faced avatars. This is why we considered a more sophisticated solution that incorporates a synthetic human face to every avatar.

This is performed in a three-step process:

This process presents two main challenges: The first one is to identify the vertices in the raw scan that belong to the model's face and the second is provide the avatar a human-like face. To identify the vertices which lay on the face, we use AI, in particular a convolutional neural network. The development of such a network required the exploitation of a large dataset generated in previous projects. The dataset was used to train the net and to improve its effectivity.

The computational kernel developed presents a high performance. Although the time-to-solution highly depends on the number of vertices of the input 3D object, a regular mesh of about 50,000 vertices can be processed in just few seconds. In addition, the algorithm presents a high degree of parallelism, allowing the use of massive parallel architectures like GPUs. This brings the possibility to further optimize the kernel if needed.

Once the vertices are identified, they are removed from the 3D object, obtaining a de-faced version of the raw scan.

Finally, the last step is again a challenging process that removes artifacts from the input data, including tasks as hole-filling and noise removal. This is performed via a template-fitting approach [13], which provides a realistic 3D closed avatar. This process is capable of replacing missing data, i.e. holes in the original mesh, with realistic data. In the case of the face, this means that the final avatar will have a face that will perfectly fit to the rest of the avatar and also present a human flavour (Fig. 3).

# *3.2 Architectural Solution Ensuring Data Security and Privacy at Hospitals*

The purpose of the software installed in hospitals is twofold: extract body measurements by processing CT images and serve queries for aggregated data coming

**Fig. 3** Detail of the head in a raw scan (left), identification of vertices in the face (middle), synthetic face (right)

**Fig. 4** Software architecture

from the Consortium users via Hyperledger. The software was created to meet the following privacy and security requirements: (1) no personal data should leave the hospital; clients outside the hospital can only receive aggregated data, resistant to de-anonymization efforts via differential privacy; (2) the server holding the personal data cannot be connected to the internet; (3) hospitals should be able to review all outgoing data; (4) no data is sent through the Hyperledger; Consortium users should be able to pick up their data directly at the hospital endpoint; (5) the derived individuals' data (e.g. measurements) is IP sensitive and therefore must be kept out of the hospital's reach, whilst also being unavailable to Philips due to privacy restrictions. Our software solution addressing the above requirements is shown in Fig. 4.

The solution requires the use of two virtual machines (VMs): 'Edge' and 'Internal'. The Edge VM has software that enables it to communicate with BodyPass members. It receives data queries from Hyperledger and makes the results available for download by users. The Internal machine is intended for the processing of raw and derived data. Due to security constraints, it operates without any connection to the internet. Every 20 s data is exchanged between Edge and Internal by means of secure bidirectional file transfers controlled by the hospital, where hospital can review all outgoing data. Both VMs use 'Barista'<sup>6</sup> infrastructure for storage and access to the data. Internally Barista consists of two modules. The first module is a cloud-based data repository (SDR) organized in different 'Studies' and 'Datasets', with access control capabilities. The second module is a web GUI (CUSI) for data manipulation. Datasets in the SDR are used both for the storage of derived data as well as acting as a sort of basic message passing interface for various data processing agents. This way the data processing agents are released from the implementation of their own endpoints. CUSI provides general (image) annotation and 3D visualization capabilities for the data stored in SDR.

#### **4 The Secure Exchange of 3D Data with Blockchain**

This chapter aims to describe in detail how the BodyPass approach solves the secure exchange of 3D data from a blockchain perspective. The main objective is to foster exchange, linking and re-use, as well as to integrate 3D data assets from the different business sectors. To cover this, BodyPass has adapted and created tools that allow a secure exchange of information between data owners, companies and subjects (patients and customers). In the BodyPass context, 3D personal data contains useful information for product design, online sale services, medical research and patient follow-up.

A conceptual view of the BodyPass solution is shown in Fig. 5, including the different stakeholders and main building blocks. The building blocks represent aggregated functionality, giving an idea of the flows of usage and retrieval tools.

One of the main drawbacks of the blockchain technology is the incompatibility with managing large amounts of data because of the size limitation in the transactions and the need to replicate the information in the different nodes for the consensus. To overcome this issue, we divided the BodyPass architecture into two different planes as shown in Fig. 6. The first plane depicts the data sharing itself (the Data Sharing Plane), while the second (the Data Management Plane) focuses on the management of the large data elements not so suitable to the blockchain-based plane. In addition, this modular architecture provides two key advantages: It allows scaling the number of data providers easily and facilitates additional control over the data made available by these data providers in the network and who accesses them.

<sup>6</sup> Barista is an integrated suite of tools, developed in Philips Research, enabling study-oriented data collection, AI algorithm creation and rapid implementation in user-facing workflows, see https:// barista.eu1.phsdp.com

**Fig. 5** High level abstraction of the BodyPass ecosystem

**Fig. 6** BodyPass conceptual architecture

There are three different profiles that interact in the BodyPass ecosystem considering their relationship with the data: (1) data providers, (2) data processors and (3) data consumers.

Figure 6 shows these three different profiles, including examples of users of each profile with members of BodyPass.7 BodyPass attempts to break data silos from several data providers from different sectors and foster exchange and reutilization of data assets that may help data consumers of the network to get external data. On the other hand, trust is a key aspect when devising a data sharing platform, especially if this platform is decentralized and should avoid the intervention of a central

<sup>7</sup> https://www.bodypass.eu/partners

authority or intermediary to certify that trust. Finally, there are several entities in the network that may generate interactions, transactions or dependencies between transactions that must also be shared in the repository. This set of characteristics led to the selection of blockchain technology [14] as the main driver of the BodyPass architecture.

The BodyPass ecosystem makes use of blockchain as a distributed ledger. Given the characteristics of the project and the business objectives, the approach selected is a permissioned blockchain, utilized by the members of BodyPass. Public blockchain networks are completely open to interact with the network and require self-governance, and private blockchain networks only allow the participation of selected entities. However, permissioned blockchain networks can adapt to hybrid scenarios like the one in BodyPass, not only letting the participants access the network once their identity has been verified but also assigning concrete permissions that can restrict which activities each participant is able to perform on the network.

Data providers usually have storage solutions for their existing assets. Therefore, the storage of the assets is not part of the scope of the blockchain in the BodyPass ecosystem. Considering these non-transactional data, which occasionally could be dynamically changed or too large, the BodyPass architecture has adopted the design pattern of off-chain storage [15], composed consequently of off-chain data (big chunks of data managed outside the blockchain network).

This approach carries some benefits, like saving bandwidth and storage capacity in the BodyPass ecosystem nodes and avoiding potential confidentiality issues derived from data being distributed out of a designated storage center. General Data Protection Regulation (GDPR) in Europe and companies doing business in Europe drive the need for new off-chain storage in blockchain applications. It is recommended to store sensitive information as off-chain data so that you can delete it if need be [16].

The *Data Sharing Plane* shown in Fig. 6 represents the implementation of the blockchain and provides the features to be a flexible trust model. It is built upon a modular architecture, configurable to choose the most suitable consensus mechanism or certification authority. There is a *Membership Service Provider* to deal with identity management and authentication. Inside the *Data Sharing Plane*, members can participate as if they were private groups by means of channels. Each member can be included in more than one channel, each of them with their own policies. All transactions are stored in the distributed ledger and, therefore, audit efficiency and quality are improved.

Due to the limitation regarding the handling of big data volumes of blockchain networks, this Data Sharing Plane is where the metadata and the permissions of the network components are managed. This has been implemented in BodyPass using Hyperledger Fabric [17] following the logic specified in the chaincode (i.e. a set of rules that govern the blockchain network) developed through the Hyperledger tool Composer [18]. Hyperledger Fabric is a permissioned blockchain network.

Considering that a building block is an asset or software piece from an architectural point of view, the BodyPass functional building blocks have interfaces to access the functionality that they provide. The green boxes shown in Fig. 7 represent

the implementation of these interfaces. This means both planes represented in Fig. 8 have their own API (BODYPASS REST API for the interactions with the Data Sharing Plane and a Data Plane REST API called from the Data Sharing Plane to perform the operations in the Data Management Plane).

The *Data Management Plane* exposes a REST API that will only be consumed by the *Data Sharing Plane*, and therefore the BodyPass actors do not have direct access to the Data Management plane, thus reinforcing the sense of security. Any data-related query will be executed via this API (after it has been 'authorized' and initiated in the Data Sharing Plane). The functionality of the whole ecosystem is exposed also as a REST API, in such a way the interactions with network assets, participants and transactions are available through standard HTTP operations, following the REST architecture (Representational State Transfer). This way, each HTTP request contains all the information necessary to execute it, which allows neither client nor server to remember any previous status. The interface is uniform, only specific actions (POST, GET, PUT and DELETE) are applied on the resources. As benefits, the protocol increases the scalability of the project and allows the internal components to evolve independently.

The *Data Management Plane*, as shown in Fig. 7, is the component that manages access to all off-chain data sources and orchestrates queries to be executed over the distributed storage. The Data Management Plane provides the security and required constraints for the blockchain members. Hash values are stored in order to verify the data when objects are accessed subsequently.

The Data Management Plane contains all the information supplied by the different data providers, which may be accessed for certain users of the BodyPass system who have previously reached an agreement with the data owners.

A detailed view of the Data Management Plane is shown in Fig. 8.

To access the Data Sharing plane REST API, it will be necessary to login in the system, since it is protected with an OAuth Keycloak server. The ATOS Data Hub

**Fig. 8** Data storage plane architecture

REST API will consult the different catalogues supplied by the data providers and will obtain the necessary access information so that the user can retrieve these data. This access information is returned to the third-party application that will make the necessary call to the data providers' APIs, obtaining as a response the required information.

For the access of the blockchain network to the different data catalogs supplied by the data providers through the data plane REST API, PrestoDB [19] will be used. PrestoDB allows to perform federated queries among multiple (relational and NoSQL) databases such as Cassandra, Redis, MongoDB or PostgreSQL with a reasonable performance.

Although it has a slightly higher response time than other query engines, such as Apache Impala, it has a much more complete SQL syntax and has a longer list of database connectors (widening the scope of technologies to be used by the data providers) (Fig. 9).

The BodyPass system will generate a query plan from every request made by the user in the blockchain network. Using this query plan, PrestoDB will be able to consult the necessary data catalogs in order to obtain the necessary access information to retrieve the final data of the supplier. Figure 10 depicts the different steps taken in the application architecture in order to execute a query, showing the interaction between the different data planes and the data providers.

The query planner is the component of the BodyPass system that will be responsible for translating the requirements of the end users in queries to those catalog databases that contain the desired information, considering possible combinations between different data sources. This will be invoked through the POST

**Fig. 9** Transaction data pipeline in a high-level representation of the BodyPass architecture

**Fig. 10** Internal organ visualization and measurement

method *getQueryPlan* of the data plane REST API and used by the POST method *runFederatedQuery*.

In the case of aggregate queries, these could be directly carried out by PrestoDB for simple metrics, but it would be necessary to coordinate between 3D image processing services in the case of more complex aggregates, such as the average chest 3D scans of a certain segment of the population.

In this way, both the blockchain network and the data providers can have an exhaustive control over what information has been accessed by which user, thus facilitating monetization of data if necessary.

Finally, to conclude, it is important to note that the data sharing approach designed and implemented for the BodyPass project is accessible through the public endpoint http://<IP_ADDRESS>:3001/explorer/#/, through which an end user could access the functionalities behind the data sharing solution described in this chapter. A Jason Web Token (JWT) will be required for accessing BodyPass endpoint in addition to a valid identity for accessing the private blockchain network that supports BodyPass. The identity will be provided by one of the existing Certification Authorities (CA) in the BodyPass blockchain private network.

# **5 The Application of BodyPass Results in Healthcare: Obesity**

BodyPass generates tools to access huge data sets extracting useful 3D data information for assessment of body shape and its relationship to the amount of fat and body distribution. BodyPass has adapted and created tools for the secure exchange of information of 3D body shape and CT scan fat distribution.

Overweight and obesity are conditions highly prevalent worldwide, and 60% of adults in Europe meet the criteria that define these conditions [20]. Worldwide, obesity is a growing health concern. Endocrinologists treat patients who are obese because of metabolic and hormonal problems and want to provide insights into metabolic and cardiovascular disease risk [21]. The classification of obesity using BMI (Body Mass Index) does not fully encompass the complex biology of excess adiposity [22]. Obesity is closely related to metabolic risk factors and is associated with significant cardiovascular morbidity and mortality. Obese patients with metabolic abnormalities have insulin resistance, atherogenic dyslipidemia, lowgrade inflammation and hypertension with a high risk to develop type 2 diabetes, atherosclerosis and cardiovascular diseases. However, not all obese subjects have these cardiometabolic abnormalities, and it is crucial to know which patients are at risk and which are not, since prognosis and therapeutic approach are different in those named Obese Healthy and Obese Unhealthy [23, 24].

The different metabolic statuses are related to fat distribution. In the Obese Healthy, fat accumulates in the subcutaneous tissue of the abdomen, around the hip and in the legs, while in the Obese Unhealthy accumulation is mainly in the mesenterium, liver, mediastinum, muscle and epicardial.

Assessment in the clinic has been the object of multiple research studies trying to properly identify measurements that predict fat distribution. Two kinds of approaches have been used, a combination of anthropometric parameters and different kinds of scans [25].

The most used among the anthropometric measurement were BMI, which indicates the presence or not of overweight and obesity but not fat distribution, abdominal circumference, waist-to-hip ratio and body shape index calculated with [WC (cm) <sup>×</sup> BMI0.66 <sup>×</sup> height (m)0.3]. All these parameters have the challenge of low accuracy and variability when measured in the clinic [26].

The scan methods obtain images from CT scan (Computed Tomography), MRI (Magnetic Resonance Imaging) and MRS (Magnetic Resonance Spectroscopy). The challenges of these methods is irradiation of patients, cost of the equipment, maintenance, cost of operators and time consuming. Using the BodyPass ecosystem it is possible to develop an easy, fast, accurate and inexpensive method to assess fat distribution.

Two hospitals in Italy and Spain developed a pilot with patients in order to test BodyPass, the first one includes the recording and integration of 3D Body Surface shape with data of internal fat amount and disposition in subcutaneous and visceral territories in a research environment and the second collect and transfer 3D Body Surface from a clinical environment. A summary or the pilot process is described below.

*Patients recruitment* was based on inclusion and exclusion criteria. This should be done in the hospital when subjects come to take CT.

*Inclusion criteria*: Age 19 or above, both sexes, thorax-abdominal CT and signed consent form. *Exclusion criteria:* Limitations for stand up and no signed consent form.

*Data collection*. In the Radiology Department a researcher recorded in a research protocol several data including:

*Data integration:* Integration of demographic and biochemical parameters with the results of the 3D Body Scan and CT images obtained the following data:

The Data Node consists of a number of software components, which, besides the extraction of measurements, also processes data queries for average measurements and body shapes, resisting data de-anonymization and providing graphical tools for exploration of the data. The software components extend the BARISTA platform which is an integrated suite of tools, developed in Philips Research, enabling study-oriented data collection, AI algorithm creation and rapid implementation in user-facing workflows; see https://barista.eu1.phsdp.com. The two components of Barista, CUSI and SDR, have different functions. SDR is a web-based repository organized in the studies and datasets with a separate access control. SDR is used to store data within the workflow of processing individual raw data and within the data query workflow. Section 5.1 describes in more detail the workflow of processing individual raw data, extracting measurements, such as *organ fat* from data and the anthropometric data. Section 5.2 describes the workflow for processing data queries coming from the BodyPass system. CUSI is a graphical web interface that allows data manipulation, data visualization and annotation tools. Within the Data Node CUSI is used as a GUI to control the workflows, for visualization of the avatars and for annotation of the DICOM images. During the BodyPass project doctors used CUSI to define new volumes inside the parametric body for fat measurements and to get the visualization of the computed avatars.

This pilot has demonstrated that BodyPass ecosystem is a promising tool for developing new, less-invasive methods to measure fat than current ones.

Below, we describe the process for:

#### *5.1 CT Image Processing*

The primary aim of DICOM processing is to extract internal and external body measurements required by healthcare and consumer goods pilots, see Table 1.

**Table 1** Body measurements

aThe small size of kidney and psoas organs put high demand on the accuracy of the localization of the organs for fat and fat fraction counting, which requires further feasibility and accuracy evaluations

bSince raw DICOM scans are limited to thorax-abdomen scans, the measurement model of the subcutaneous volume is also limited to the thorax-abdomen area

**Table 2** Different surfaces and volumes in the avatar

**Fig. 11** Introduction of new fat measurements using BARISTA CUSI

The software relies on proprietary Philips algorithms. Since these algorithms have never been published, we can only disclose a brief outline on the underlying computations.

We start with pre-processing of the raw DICOM scans including automatic segmentation of some key 3D surfaces of the human body's anatomy. At the beginning of the BodyPass project, we had a choice whether to extract fat measurements directly from the CT scans, which would require to find and segment each organ of interest, or to register a 3D body avatar and use it as organ atlas. We have chosen the second approach because it allows to define and add new fat measurement locations after the avatar registration. Thus we register the volumetric 3D body avatar to the DICOM scan volume. The volumetric 3D body avatar is a graph in Euclidean space consisting, in our case, of 356,106 vertices, where neighbouring vertices are connected by 185,259 triangles and 1,937,558 tetrahedrons. Table 2 shows an example of the allocation of the avatar elements to different organs of interest.

Barista CUSI provides a collection of data annotation tools, and, amongst others, a tool for manual segmentation of slices on a reference CT scan, which can be used to define new volumes for fat measurements, see Fig. 11.

The registration process consists of two stages: at the first stage an approximate avatar is computed from the *Age, Gender, Weight* and *Height* parameters of the patient, using statistical shape models trained on about 30,000 3D scans. In the second stage the avatar is consistently refined so that the boundaries of organs on the avatar become aligned to the boundaries of the organs segmented on the DICOM image. In the initial version of the registration algorithm only three organs are automatically segmented—skin, subcutaneous volume and lungs—which limits the accuracy of the registration in smaller organs of the visceral cavity like kidney and psoas. There is ongoing work to improve the registration accuracy by adding more automatically segmented organs. The registered avatar is then used as a sort of 'atlas' to compute the local fat volumes according to the parametrically defined sampling positions. This results in the creation of 'internal' measurements. Please, note that the DICOM images are acquired in the laying pose with hands lifted up, while, according to D3.1, it is required that the measurements are collected in the standing A-pose. Therefore, an experimental algorithm was implemented that, for every registered volumetric avatar, computes another avatar in the standing pose. The external measurements are defined parametrically, as close as possible to ISO 8559-1:2017 according to the anthropometric data dictionary previously defined. This assures data harmonization with the consumer goods sector.

#### *5.2 Data Query Processing*