How to Accelerate Building a Lake House Architecture with AWS Glue
AWS Architecture BlogHow to Accelerate Building a Lake House Architecture with AWS Glueby Raghavarao Sodabathina and Harsha Tadiparthi | on24 AUG 2021| inAmazon Athena,Amazon EMR,Amazon Redshift,Amazon SageMaker,Analytics,Architecture,AWS Glue,AWS Glue DataBrew|Permalink|ShareCustomers are building databases, data warehouses, and data lake solutions in isolation from each other, each having its own separate data ingestion, storage, management, and governance layers. Often these disjointed efforts to build separate data stores end up creating data silos, data integration complexities, excessive data movement, and data consistency issues. These issues are preventing customers from getting deeper insights. To overcome these issues and easily move data around, aLake House approachon AWS was introduced.In this blog post, we illustrate theAWS Glueintegration components that you can use to accelerate building a Lake House architecture on AWS. We will also discuss how to derive persona-centric insights from your Lake House using AWS Glue.Components of the AWS Glue integration systemAWS Glue is a serverless data integration service that facilitates the discovery, preparation, and combination of data. It can be used for analytics, machine learning, and application development. AWS Glue provides all of the capabilities needed for data integration. So you can start analyzing your data and putting it to use in minutes, rather than months.The following diagram illustrates the various components of the AWS Glue integration system.Figure 1. AWS Glue integration componentsConnect – AWS Glue allows you to connect to various data sources anywhereGlue connector:AWS Glue provides built-in support for the most commonly used data stores. You can useAmazon Redshift,Amazon RDS,Amazon Aurora, Microsoft SQL Server, MySQL, MongoDB, or PostgreSQL using JDBC connections. AWS Glue also allows you to use custom JDBC drivers in your extract, transform, and load (ETL) jobs. For data stores that are not natively supported such as SaaS applications, you can useconnectors. You can also subscribe to several connectors offered in the AWS Marketplace.Glue crawlers:You can use acrawlerto populate the AWS Glue Data Catalog with tables. A crawler can crawl multiple data stores in a single pass. Upon completion, the crawler creates or updates one or more tables in your Data Catalog. Extract, transform, and load (ETL) jobs that you define in AWS Glue use these Data Catalog tables as sources and targets.Catalog – AWS Glue simplifies data discovery and governanceGlue Data Catalog:TheData Catalogserves as the central metadata catalog for the entire data landscape.Glue Schema Registry:TheAWS Glue Schema Registryallows you to centrally discover, control, and evolve data stream schemas. With AWS Glue Schema Registry, you can manage and enforce schemas on your data streaming applications.Data quality – AWS Glue helps you author and monitor data quality rulesGlue DataBrew:AWS Glue DataBrewallows data scientists and data analysts to clean and normalize data. You can use a visual interface, reducing the time it takes to prepare data by up to 80%. With Glue DataBrew, you can visualize, clean, and normalize data directly from your data lake, data warehouses, and databases.Curate data:You can use either Glue development endpoint orAWS Glue Studioto curate your data.AWS Glue development endpointis an environment that you can use to develop and test your AWS Glue scripts. You can choose eitherAmazon SageMakernotebook orApache Zeppelinnotebook as an environment.AWS Glue Studio is a new visual interface for AWS Glue that supports extract-transform-and-load (ETL) developers. You can author, run, and monitor AWS Glue ETL jobs. You can now use a visual interface to compose jobs that move and transform data, and run them on AWS Glue.AWS Data Exchangemakes it easy for AWS customers to securely exchange and use third-party data in AWS. This is for data providers who want to structure their data across multiple datasets or enrich their products with additional data. You can publish additional datasets to your products using the AWS Data Exchange.Deequis an open-source data quality library developed internally at Amazon, fordata quality. It provides multiple features such as automatic constraint suggestions and verification, metrics computation, and data profiling.Build a Lake House architecture faster, using AWS GlueFigure 2 illustrates how you can build a Lake House using AWS Glue components.Figure 2. Building Lake House architectures with AWS GlueThe architecture flow follows these general steps:Glue crawlers scan the data from various data sources and populate the Data Catalog for your Lake House.The Data Catalog serves as the central metadata catalog for the entire data landscape.Once data is cataloged, fine-grained access control is applied to the tables throughAWS Lake Formation.Curate your data with business and data quality rules by using Glue Studio, Glue development endpoints, or Glue DataBrew. Place transformed data in a curated Amazon S3 for purpose built analytics downstream.Facilitate data movement with AWS Glue to and from your data lake, databases, and data warehouse by using Glue connections. UseAWS Glue Elastic viewsto replicate the data across the Lake House.Derive persona-centric insights from your Lake House using AWS GlueMany organizations want to gather observations from increasingly larger volumes of acquired data. These insights help them make data-driven decisions with speed and agility. They must use a central data lake, a ring of purpose-built data services, and data warehouses based on persona or job function.Figure 3 illustrates the Lake House inside-out data movement with AWS Glue DataBrew,Amazon Athena, Amazon Redshift, andAmazon QuickSightto perform persona-centric data analytics.Figure 3. Lake House persona-centric data analytics using AWS GlueThis shows how Lake House components serve various personas in an organization:Data ingestion:Data is ingested toAmazon Simple Storage Service (S3)from different sources.Data processing:Data curators and data scientists use DataBrew to validate, clean, and enrich the data. Amazon Athena is also used to run improvised queries to analyze the data in the lake. The transformation is shared with data engineers to set up batch processing.Batch data processing:Data engineers or developers set up batch jobs in AWS Glue and AWS Glue DataBrew. Jobs can be initiated by an event, or can be scheduled to run periodically.Data analytics:Data/Business analysts can now analyze prepared dataset in Amazon Redshift or in Amazon S3 using Athena.Data visualizations:Business analysts can create visuals in QuickSight. Data curators can enrich data from multiple sources. Admins can enforce security and data governance. Developers can embed QuickSight dashboard in applications.ConclusionUsing a Lake House architecture will help you get persona-centric insights quickly from all of your data based on user role or job function. In this blog post, we describe several AWS Glue components and AWS purpose-built services that you can use to build Lake House architectures on AWS. We have also presented persona-centric Lake House analytics architecture using AWS Glue, to help you derive insights from your Lake House.Read more and get started on building Lake House Architectures on AWS.What is a Lake House approach?Harness the power of your data with AWS AnalyticsAWS Lake House reference architecture(scroll down in blog)Derive Insights from AWS Lake HousewhitepaperAWS Glue Developer GuideTAGS:data lake,data warehouseRaghavarao SodabathinaRaghavarao Sodabathina is a Principal Solutions Architect at AWS, focusing on Data Analytics, AI/ML, and Serverless Platform. He engages with customers to create innovative solutions that address customer business problems and to accelerate the adoption of AWS services. In his spare time, Raghavarao enjoys spending time with his family, reading books, and watching movies.Harsha TadiparthiHarsha Tadiparthi is a Specialist Sr. Solutions Architect, AWS Analytics .He enjoys solving complex customer problems in Databases and Analytics and delivering successful outcomes. Outside of work, he loves to spend time with his family, watch movies, and travel whenever possible.
Customers are building databases, data warehouses, and data lake solutions in isolation from each other, each having its own separate data ingestion, storage, management, and governance layers. Often these disjointed efforts to build separate data stores end up creating data silos, data integration complexities, excessive data movement, and data consistency issues. These issues are preventing […]
Raghavarao Sodabathina
Amazon Athena Amazon EMR Amazon Redshift Amazon SageMaker Analytics Architecture AWS Glue AWS Glue DataBrew