Mainframe data integration: Using mainframe data to build cloud native services with AWS
AWS Architecture BlogMainframe data integration: Using mainframe data to build cloud native services with AWSby Malathi Pinnamaneni and Jeff Fritz | on16 MAR 2022| inAmazon Athena,Amazon DynamoDB,Amazon Managed Streaming for Apache Kafka (Amazon MSK),Amazon MQ,Architecture,AWS Glue,AWS Transfer Family,Kinesis Data Streams|Permalink|ShareForInternational Women’s Dayand Women’s History Month, we’re featuring more than a week’s worth of posts that highlight female builders and leaders. We’re showcasing women in the industry who are building, creating, and, above all, inspiring, empowering, and encouraging everyone—especially women and girls—in tech.Many companies in the financial services and insurance industries rely on mainframes for their most business-critical applications and data. But mainframe workloads typically lack agility. This is one reason that organizations struggle to innovate, iterate, and pivot quickly to develop new applications or release new capabilities. Unlocking the value of mainframe data is one option in your modernization journey.In this blog post, we will discuss some typical data integration patterns. Your goal might be developing new functions using mainframe data or creating new channels. Or, you might want to augment your mainframe capabilities with analytics, artificial intelligence (AI), and machine learning (ML). Your migration may require a hybrid architecture, and you need guidance on how to integrate the data.The mainframe data integration patterns in this post use software services that facilitate data replication toAmazon Web Services (AWS):File-based data synchronizationChange data captureEvent-sourced replicationOnce data is liberated from the mainframe, you can develop new agile applications for deeper insights using analytics and ML. Or you can extend the mainframe application capabilities by creating a microservices- or voice-based mobile application. For example, if a bank could access their historical mainframe data to analyze customer behavior, they could develop a new solution based on profiles for loan recommendations.If you are considering mainframe migration and modernization at scale, we recommend a tool-based industrial approach usingAWS Mainframe Modernizationservice. This should reduce risks and accelerate modernization business benefits.Solution overview: Mainframe data integrationFigure 1. Mainframe data integration conceptual flowMainframe integration: Architecture reference patternsFile-based batch integrationIntegration scenarios often require replicating files to AWS, or synchronizing between on-premises and AWS. Use cases include:Analyzing current and historical data to enhance business analyticsProviding data for further processing on downstream or upstream dependent systems. This is necessary for exchanging data between applications running on the mainframe, and applications running on AWSFigure 2. File-based batch ingestion pattern for interactive data analyticsFile-based batch integration – Batch ingestion for interactive data analytics (Figure 2)Data ingestion.In this example, we show how data can be ingested toAmazon S3usingAWS Transfer Family ServicesorAWS DataSync. Mainframe data is typically encoded in extended binary-coded decimal interchange code (EBCDIC) format.Prescriptive guidanceexists to convert EBCDIC to ASCII format.Data transformation.Before moving data to AWS data stores, transformation of the data may be necessary to use it for analytics.AWS analytics serviceslikeAWS GlueandAWS Lambdacan be used to transform the data. For large volume processing, use Apache Spark onAWS Elastic Map Reduce (Amazon EMR), or a custom Spring Boot application running onAmazon EC2to perform these transformations. This process can be orchestrated usingAWS Step FunctionsorAWS Data Pipeline.Data store.Data is transformed into a consumable format that can be stored in Amazon S3.Data consumption.You can useAWS analytics serviceslikeAmazon Athenafor interactive ad-hoc query access,Amazon QuickSightfor analytics, andAmazon Redshiftfor complex reporting and aggregations.Figure 3. File upload to operational data stores for further processingFile-based batch integration – File upload to operational data stores for further processing (Figure 3)Using AWS File Transfer Services, upload CSV files to Amazon S3.Once the files are uploaded, S3 event notification can invoke an AWS Lambda function to load toAmazon Aurora. For low latency data access requirements, you can use ascalable serverless importpattern with AWS Lambda andAmazon SQSto load intoAmazon DynamoDB.Once the data is in data stores, it can be consumed for further processing.Transactional replication-based integration (Figure 4)Several integration scenarios require continuous near-real-time replication of relational data to keep a copy of the data in the cloud. Change Data Capture (CDC) for near-real-time transactional replication works by capturing change log activity to drive changes in the target dataset. Use cases include:Command Query Responsibility Segregation (CQRS) architectures that use AWS to service all read-only and retrieve functionsOn-premises systems with tightly coupled applications that require a phased modernizationReal-time operational analyticsFigure 4. Transactional replication (CDC) patternCDC tools in theAWS Marketplacecan be used to manage real-time data movement between the mainframe and AWS.You can use a fan-out pattern to read once from the mainframe to reduce processing requirements and replicate data to multiple data stores based on your requirements:For low latency requirements, replicate toAmazon Kinesis Data Streamsand use AWS Lambda to store in Amazon DynamoDB.For critical business functionality with complex logic, use Amazon Aurora orAmazon Relational Database Service (RDS)as targets.To build data lake or use as an intermediary for ETL processing, customers can replicate to S3 as target.Once the data is in AWS, customers can build agile microservices for read-only functions.Message-oriented middleware (event sourcing) integration (Figure 5)With message-oriented middleware (MOM) systems like IBM MQ on mainframe, several scenarios require integrating with cloud-based streaming and messaging services. These act as a buffer to keep your data in sync. Use cases include:Consume data from AWS data stores to enable new communication channels. Examples of new channels can be mobile or voice-based applications and can be innovations based on MLMigrate the producer (senders) and consumer (receivers) applications communicating with on-premises MOM platforms to AWS with an end goal to retire on-premises MOM platformFigure 5. Event-sourcing integration patternMainframe transactions from IBM MQ can be read using a connector or a bridge solution. They can then be published toAmazon MQqueues orAmazon Managed Streaming for Apache Kakfa (MSK)topics.Once the data is published to the queue or topic, consumers encoded in AWS Lambda functions or Amazon compute services can process, map, transform, or filter the messages. They can store the data in Amazon RDS,Amazon ElastiCache, S3, or DynamoDB.Now that the data resides in AWS, you can build new cloud-native applications and do the following:Push notifications.Use Amazon RDS or S3 event triggers to callAmazon Simple Notification Services (SNS)to push notifications to mobile devices.Build innovative services.Invoke ML services such asAmazon SageMakerdirectly from these data stores. Build voice interfaces usingAmazon API Gateway,Amazon Lex, or Amazon Alexa skills.Enable new functions.Build new applications with business logic residing in microservices hosted by AWS Lambda or in containers withinAmazon Elastic Container Service (ECS).ConclusionMainframe data integration using AWS services enables you to reduce cost, create modern architectures, and integrate your mainframe and cloud-native technologies. You’ll be able to inform your business decisions with improved analytics, and create new opportunities for innovation and the development of modern applications.More posts for Women’s History Month!Celebrate International Women’s Day all week with the Architecture BlogDeploying service-mesh-based architectures using AWS App Mesh and Amazon ECSfromKesha Williams, anAWS Heroand award-winning software engineer.A collection of several blog posts written and co-authored by womenCurated contentfrom theLet’s Architect!team and alive Twitter chatWomen at AWS – Diverse backgrounds make great solutions architectsExtend SQL Server DR using log shipping for SQL Server FCI with Amazon FSx for Windows configurationBuilding your brand as a Solutions ArchitectMessage to the next generation of women disruptors in technologyMigrating petabytes of data from on-premises file systems to Amazon FSx for LustreOther ways to participateShare Your 2022 #SheCoded StoryInternational Women’s Day: Improve Equality for Women in TechAWS She Builds CloudUp programTAGS:International Women's DayMalathi PinnamaneniMalathi Pinnamaneni is a Senior Migration & Modernization Solutions Architect at AWS where she works with large enterprise customers modernize their IT landscape. Her focus is on developing migration and modernization architecture reference patterns that accelerates customer’s journey to AWS. Malathi lives in the San Francisco Bay Area, and in her free time enjoys spending time with family and friends.Jeff FritzJeff Fritz is a Senior Modernization Solutions Architect focused on identifying and developing migration methods and patterns that accelerate the customer's journey to AWS. He has over 30 years of experience in designing, building, and operating complex information systems. He has extensive experience in database platforms and technologies.
For International Women’s Day and Women’s History Month, we’re featuring more than a week’s worth of posts that highlight female builders and leaders. We’re showcasing women in the industry who are building, creating, and, above all, inspiring, empowering, and encouraging everyone—especially women and girls—in tech. Many companies in the financial services and insurance industries rely […]
Malathi Pinnamaneni
Amazon Athena Amazon DynamoDB Amazon Managed Streaming for Apache Kafka (Amazon MSK) Amazon MQ Architecture AWS Glue AWS Transfer Family Kinesis Data Streams