Building a healthcare data pipeline on AWS with IBM Cloud Pak for Data
AWS Architecture BlogBuilding a healthcare data pipeline on AWS with IBM Cloud Pak for Databy Eduardo Monich Fronza, Rishit G. Barochia, and Sunil Gajula | on14 DEC 2022| inAmazon Simple Notification Service (SNS),Amazon Simple Storage Service (S3),Amazon VPC,Architecture,AWS CloudTrail,Kinesis Data Firehose|Permalink|ShareHealthcare data is being generated at an increased rate with the proliferation of connected medical devices and clinical systems. Some examples of these data are time-sensitive patient information, including results of laboratory tests, pathology reports, X-rays, digital imaging, and medical devices to monitor a patient’s vital signs, such as blood pressure, heart rate, and temperature.These different types of data can be difficult to work with, but when combined they can be used to build data pipelines and machine learning (ML) models to address various challenges in the healthcare industry, like the prediction of patient outcome, readmission rate, or disease progression.In this post, we demonstrate how to bring data from different sources, likeSnowflakeand connected health devices, to form a healthcare data lake on Amazon Web Services (AWS). We also explore how to use this data withIBM Watsonto build, train, and deploy ML models. You can learn how to integrate model endpoints with clinical health applications to generate predictions for patient health conditions.Solution overviewThe main parts of the architecture we discuss are (Figure 1):Using patient data to improve health outcomesHealthcare data lake formation to store patient health informationAnalyzing clinical data to improve medical researchGaining operational insights from healthcare provider dataProviding data governance to maintain the data privacyBuilding, training, and deploying an ML modelIntegration with the healthcare systemFigure 1. Data pipeline for the healthcare industry using IBM CP4D on AWSIBM Cloud Pak for Data (CP4D)is deployed onRed Hat OpenShift Service on AWS (ROSA). It provides the componentsIBM DataStage,IBM Watson Knowledge Catalogue,IBM Watson Studio,IBM Watson Machine Learning, plus a wide variety of connections with data sources available in a public cloud or on-premises.Connected health devices, on the edge, use sensors and wireless connectivity to gather patient health data, such as biometrics, and send it to the AWS Cloud throughAmazon Kinesis Data Firehose.AWS Lambdatransforms the data that is persisted toAmazon Simple Storage Service (Amazon S3), making that information available to healthcare providers.Amazon Simple Notification Service (Amazon SNS)is used to send notifications whenever there is an issue with the real-time data ingestion from the connected health devices. In case of failures, messages are sent via Amazon SNS topics for rectifying and reprocessing of failure messages.DataStage performs ETL operations and move patient historical information from Snowflake into Amazon S3. This data, combined with the data from the connected health devices, form a healthcare data lake, which is used in IBM CP4D to build and train ML models.The pipeline described in architecture uses Watson Knowledge Catalogue, which provides data governance framework and artifacts to enrich our data assets. It protects sensitive patient information from unauthorized access, like individually identifiable information, medical history, test results, or insurance information.Data protection rules define how to control access to data, mask sensitive values, or filter rows from data assets. The rules are automatically evaluated and enforced each time a user attempts to access a data asset in any governed catalog of the platform.After this, the datasets are published to Watson Studio projects, where they are used to train ML models. You can develop models usingJupyter Notebook,IBM AutoAI(low-code), orIBM SPSSmodeler (no-code).For the purpose of this use case, we usedlogistic regression algorithmfor classifying and predicting the probability of an event, such as disease risk management to assist doctors in making critical medical decisions. You can also build ML models using algorithms likeClassification,Random Forest, andK-Nearest Neighbor. These are widely used to predict disease risk.Once the models are trained, they are exposed as endpoints with Watson Machine Learning and integrated with the healthcare application to generate predictions by analyzing patient symptoms.The healthcare applications are a type of clinical software that offer crucial physiological insights and predict the effects of illnesses and possible treatments. It provides built-in dashboards that display patient information together with the patient’s overall metrics for outcomes and treatments. This can help healthcare practitioners gain insights into patient conditions. It also can help medical institutions prioritize patients with more risk factors and curate clinical and behavioral health plans.Finally, we are usingIBM Security QRadar XDR SIEMto collect, process, and aggregateAmazon Virtual Private Cloud (Amazon VPC)flow logs,AWS CloudTraillogs, and IBM CP4D logs. QRadar XDR uses this information to manage security by providing real-time monitoring, alerts, and responses to threats.Healthcare data lakeA healthcare data lake can help health organizations turn data into insights. It is centralized, curated, and securely stores data on Amazon S3. It also enables you to break down data silos and combine different types of analytics to gain insights. We are using the DataStage, Kinesis Data Firehose, and Amazon S3 services to build the healthcare data lake.Data governanceWatson Knowledge Catalogue provides an ML catalogue for data discovery, cataloging, quality, and governance. We define policies in Watson Knowledge Catalogue to enable data privacy and overall access to and utilization of this data. This includes sensitive data and personal information that needs to be handled through data protection, quality, and automation rules. To learn more about IBM data governance, please refer toRunning a data quality analysis (Watson Knowledge Catalogue).Build, train, and deploy the ML modelWatson Studio empowers data scientists, developers, and analysts to build, run, and manage AI models on IBM CP4D.In this solution, we are building models using Watson Studio by:Promoting the governed data from Watson Knowledge Catalogue to Watson Studio for insightsUsing ETL features, such as built-in search, automatic metadata propagation, and simultaneous highlighting, to process and transform large amounts of dataTraining the model, including model technique selection and application, hyperparameter setting and adjustment, validation, ensemble model development and testing; algorithm selection; and model optimizationEvaluating the model based on metric evaluation, confusion matrix calculations, KPIs, model performance metrics, model quality measurements for accuracy and precisionDeploying the model on Watson Machine Learning using online deployments, which create an endpoint to generate a score or prediction in real timeIntegrating the endpoint with applications like health applications, as demonstrated in Figure 1ConclusionIn this blog, we demonstrated how to use patient data to improve health outcomes by creating a healthcare data lake and analyzing clinical data. This can help patients and healthcare practitioners make better, faster decisions and prioritize cases. We also discussed how to build an ML model using IBM Watson and integrate it with healthcare applications for health analysis.Additional resourcesIBM on AWS Partner PageDeploying IBM Cloud Pak for Data on Red Hat OpenShift Service on AWSExporting Cloud Pak for Data audit records to QRadarIBM Cloud Pak for Data on the AWS MarketplaceIBM Cloud Pak for Security on the AWS MarketplaceEduardo Monich FronzaEduardo Monich Fronza is a Partner Solutions Architect at AWS. His experience includes Cloud, solutions architecture, application platforms, containers, workload modernization, and hybrid solutions. In his current role, Eduardo helps AWS partners and customers in their cloud adoption journey.Rishit G. BarochiaRishit G. Barochia is a Cloud Software Architect at IBM. His experience includes technical architect, cloud, micro-services architectures, and hybrid solutions. He is working on IBM-AWS strategic partnership. In Rishit's role, he helps customer to design, plan, and architect IBM software solutions on the AWS Cloud.Sunil GajulaSunil Gajula is a Solution Architect in IBM-AWS strategic partnership team at IBM. His experience includes cloud, solution architect, and micro-services architectures. Sunil is passionate about helping clients to succeed in their analytics journey. In his current role, he works with, IBM partners and clients to adopt and develop AI, ML, and analytics solutions.
Healthcare data is being generated at an increased rate with the proliferation of connected medical devices and clinical systems. Some examples of these data are time-sensitive patient information, including results of laboratory tests, pathology reports, X-rays, digital imaging, and medical devices to monitor a patient’s vital signs, such as blood pressure, heart rate, and temperature. […]
Eduardo Monich Fronza
Amazon Simple Notification Service (SNS) Amazon Simple Storage Service (S3) Amazon VPC Architecture AWS CloudTrail Kinesis Data Firehose