Hybrid Cloud Architectures Using Self-hosted Apache Kafka and AWS Glue
AWS Architecture BlogHybrid Cloud Architectures Using Self-hosted Apache Kafka and AWS Glueby Brandon Rubadou, Ravi Menon, and John O'Donnell | on10 SEP 2021| inAdvanced (300),Amazon Simple Storage Service (S3),Architecture,AWS Glue|Permalink|ShareUsing analytics to gain insights from a variety of datasets is key to successful transformation. There are many options to consider to realize the full value and potential of our data in a hybrid cloud infrastructure. Common practice is to route data produced from on-premises to a central repository or data lake. Here it can be consumed by multiple applications.You can use anApache Kafkacluster for data movement from on-premises to the data lake, usingAmazon Simple Storage Service (Amazon S3). But you must either replicate the topics onto a cloud cluster, or develop a custom connector to read and copy the topics to Amazon S3. This presents a challenge for many customers.This blog presents another option; an architecture solution leveragingAWS Glue.Kafka and ETL processingApache Kafka is an open-source distributed event streaming platform used by thousands of companies for high-performance data pipelines, streaming analytics, data integration, and mission-critical applications. You can use Kafka clusters as a system to move data between systems.Producerstypically publish data (or push) to a Kafka topic, where an application can consume it.Consumersare usually custom applications that feed data into respective target applications. These targets can be a data warehouse, anAmazon OpenSearch Servicecluster, or others.AWS Glue offers the ability to create jobs that will extract, transform, and load (ETL) data. This allows you to consume from many sources, such as from Apache Kafka,Amazon Kinesis Data Streams, orAmazon Managed Streaming for Apache Kafka (Amazon MSK). The jobs cleanse and transform the data, and then load the results into Amazon S3 data lakes or JDBC data stores.Hybrid solution and architecture designIn most cases, the first step in building a responsive and manageable architecture is to review the data itself. For example, if we are processing insurance policy data from a financial organization, our data may contain fields that identify customer data. These can be account ID, an insurance claim identifier, and the dollar amount of the specific claim. Glue provides the ability to change any of these field types into the expected data lake schema type for processing.Figure 1. Data flow – Source to data lake targetNext, AWS Glue must be configured to connect to the on-premises Kafka server (see Figure 1). Private and secure connectivity to the on-premises environment can be established viaAWS Direct Connector a VPN solution. Traffic from theAmazon Virtual Private Cloud (Amazon VPC)is allowed to access the cluster directly. You can do this by creating a three-step streaming ETL job:Create a Glue connection to the on-premises Kafka sourceCreate a Data Catalog tableCreate an ETL job, which saves to an S3 data lakeConfiguring AWS GlueCreate a connection.Using AWS Glue, create a secure SSL connection in the Data Catalog using the predefined Kafka connection type. Enter the hostname of the on-premises cluster and use the custom-managed certificate option for additional security. If you are in a development environment, you are required to generate a self-signed SSL certificate. Use your Kafka SSL endpoint to connect to Glue. (AWS Glue also supports client authentication for Apache Kafka streams.)Specify a security group.To allow AWS Glue to communicate between its components, specify a security group with a self-referencing inbound rule for all TCP ports. By creating this rule, you can restrict the source to the same security group in the Amazon VPC. Ensure you check the default security group for your VPC, as it could have a preconfigured self-referencing inbound rule for ALL traffic.Create the Data Catalog.Glue can auto-create the data schema. Since it’s a simple flat file, use the schema detection function of Glue. Set up the Kafka topic and refer to the connection.Define the job properties.Create theAWS Identity and Access Management (IAM)role to allow Glue to connect to S3 data. Select an S3 bucket and format. In this case, we use CSV and enable schema detection.The Glue job can be scheduled, initiated manually, or by using an event driven architecture. Note that Glue does not yet support the “test connection” option within the console. Make sure you set the “Job Timeout” and enter a duration in minutes because the default value is blank.When the job runs, it pulls the latest topics from the source on-premises Kafka cluster. Glue supports checkpoints to ensure that all source data is processed. By default, AWS Glue processes and writes out data in 100-second windows. This allows data to be processed efficiently and permits aggregations to be performed on data arriving later. You can modify this window size to increase timeliness or aggregation accuracy. AWS Glue streaming jobs use checkpoints rather than job bookmarks to track the data that has been read.AWS Gluebills hourly for streaming ETL jobs only while they are running.Now that the connection is complete and the job is created, we can format the source data needed for the data lake. AWS Glue offers a set of built-intransformsthat you can use to process your data using your ETL script. The transformed data is then placed in S3, where it can be leveraged as part of a larger data lake environment.Many additional steps can be taken to render even more value from the information. For example, one team may choose to use a business intelligence tool likeAmazon QuickSightto visualize and embed the data into an internal dashboard. Another team may want to use event driven architectures to notify financial analysts and initiate downstream actions when specific types of data are discovered. There are endless opportunities that should be determined by the business needs.SummaryIn this blog post, we have given an overview of an architecture that provides hybrid cloud data integration and analytics capability. Once the data is transformed and hosted in the S3 data lake, we can provide secure, reliable access to gain valuable insights. This solution allows for a variety of different producers and consumers, with the ability to handle increasing volumes of data.AWS Glue along with Apache Kafka will ensure that your on-premises workloads are tightly integrated with your larger data lake solution.If you have questions, post your thoughts in the comments section.For further reading:AWS Glue supports reading from self-managed Apache KafkaKafka Documentation – Configuring Kafka BrokersAdding Streaming ETL Jobs in AWS GlueKafka Security Tutorial – Creating SSL keys and certificatesKafka Encryption and Authentication with SSLTAGS:ETL,KafkaBrandon RubadouBrandon Rubadou is a Senior Solutions Architect with Amazon Web Services. He primarily works with AWS customers and partners in the financial services industry to provide guidance on enterprise cloud migrations and well-architected security best practices. He leverages his background in data center storage and networking technologies to help his customers achieve successful modernization and business outcomes.Ravi MenonRavi is a Senior Solutions Architect at AWS based out of Boston, MA. He is passionate about helping customers achieve the best possible outcomes on their cloud journey. With a focus on Analytics, Ravi likes to learn and dive deep on new technologies. Outside of work, Ravi is an avid reader and a sports buff who watches almost anything and plays a few of them.John O'DonnellJohn O’Donnell is a Senior Solution Architect and is the Northeast Migration Ambassador as well as the HCLS Industry SA Region SME. John spent the previous three years in AWS Professional Services (ProServe), where he had the opportunity to guide the cloud journey at Vanguard (Malvern, PA).
Using analytics to gain insights from a variety of datasets is key to successful transformation. There are many options to consider to realize the full value and potential of our data in a hybrid cloud infrastructure. Common practice is to route data produced from on-premises to a central repository or data lake. Here it can […]
Brandon Rubadou
Advanced (300) Amazon Simple Storage Service (S3) Architecture AWS Glue