Architecting Persona-centric Data Platform with On-premises Data Sources
AWS Architecture BlogArchitecting Persona-centric Data Platform with On-premises Data Sourcesby Raghavarao Sodabathina, Jigna Gandhi, Jiwan Panjiker, and Rhiannon Liebowitz | on21 JUN 2021| inAmazon Athena,Amazon EMR,Amazon SageMaker,Architecture,AWS Glue|Permalink|ShareMany organizations are moving their data from silos and aggregating it in one location. Collecting this data in a data lake enables you to perform analytics and machine learning on that data. You can store your data in purpose-built data stores, like a data warehouse, to get quick results for complex queries on structured data.In this post, we show how to architect a persona-centric data platform with on-premises data sources by usingAWS purpose-built analytics servicesandApache NiFi. We will also discussLake House architecture on AWS, which is the next evolution from data warehouse and data lake-based solutions.Data movement servicesAWS provides a wide variety of services to bring data into a data lake:AWS Data Migration Service(AWS DMS) can connect to a variety of operational Relational Database Management System (RDBMS) and NoSQL databases. It ingests the data intoAmazon Simple Storage Service (S3)AWS Lake Formation providesblueprintsto ingest data from AWS native or on-premises database sources into Amazon S3Amazon Kinesisingests streaming data into Amazon S3AWS Identity and Access Management (IAM)andAWS Lake Formationcan offer you flexible coarse-grained and fine-grained access controls on your data storage and the analytics solution that you needYou may want to bring on-premises data into the AWS Cloud to take advantage of AWS purpose-built analytics services, derive insights, and make timely business decisions. Apache NiFi is an open source tool that enables you to move and process data using a graphical user interface.For this use case and solution architecture, we use Apache NiFi to ingest data into Amazon S3 and AWS purpose-built analytics services, based on user personas.Building persona-centric data platform on AWSWhen you are building a persona-centric data platform for analytics and machine learning, you must first identify your user personas. Who will be using your platform? Then choose the appropriate purpose-built analytics services. Envision a data platform analytics architecture as a stack of seven layers:User personas:Identify your user personas for data engineering, analytics, and machine learningData ingestion layer:Bring the data into your data platform and data lineage lifecycle view, while ingesting data into your storage layerStorage layer:Store your structured and unstructured dataCataloging layer:Store your business and technical metadata about datasets from the storage layerProcessing layer:Create data processing pipelinesConsumption layer:Enable your user personas for purpose-built analyticsSecurity and Governance:Protect your data across the layersReference architectureThe following diagram illustrates how to architect a persona-centric data platform with on-premises data sources by using AWS purpose-built analytics services and Apache NiFi.Figure 1. Example architecture for persona-centric data platform with on-premises data sourcesArchitecture flow:Identify user personas:You must first identify user personas to derive insights from your data platform. Let’s start with identifying your users:Enterprise data service users who would like to consume data from your data lake into their respective applications.Business users who would like to like create business intelligence dashboards by using your data lake datasets.IT users who would like to query data from your data lake by using traditional SQL queries.Data scientists who would like to run machine learning algorithms to derive recommendations.Enterprise data warehouse users who would like to run complex SQL queries on your data warehouse datasets.Data ingestion layer:Apache NiFi scans the on-premises data storesand ingest the data into your data lake (Amazon S3). Apache NiFi can also transform the data in transit. It supports both Extract, Transform, Load (ETL) and Extract, Load, Transform (ELT) data transformations. Apache NiFi also supports data lineage lifecycle while ingesting data into Amazon S3.Storage layer:For your data lake storage, we recommend using Amazon S3 to build a data lake. It has unmatched 11 nines of durability and 99.99% availability. You can also create raw, transformed, and enriched storage layers depending upon your use case.Cataloging layer:AWS Lake Formation provides the central catalog to store and manage metadata for all datasets hosted in the data lake byAWS Glue Data Catalog. AWS services such asAWS Glue,Amazon EMR, andAmazon Athenanatively integrate with Lake Formation. They automate discovering and registering dataset metadata into the Lake Formation catalog.Processing layer:Amazon EMR processes your raw data and places them into a new S3 bucket. UseAWS Glue DataBrewand AWS Glue to process the data as needed.Consumption layer or persona-centric analytics:Once data is transformed:AWS LambdaandAmazon API Gatewaywill allow you to develop data services for enterprise data service usersYou can develop user-friendly dashboards for your business users usingAmazon QuickSightUse Amazon Athena to query transformed data for your IT usersYour data scientists can utilize AWS Glue DataBrew to clean and normalize the data andAmazon SageMakerfor machine learning modelsYour enterprise data warehouse users can useAmazon Redshiftto derive business intelligenceSecurity and governance layer:AWS IAM provides users, groups, and role-level identity, in addition to the ability to configure coarse-grained access control for resources managed by AWS services in all layers. AWS Lake Formationprovides fine-grained access controlsandyou can grant/revoke permissionsat the database- or table- or column-level access.Lake House architecture on AWSThe vast majority of data lakes are built on Amazon S3. At the same time, customers are leveragingpurpose-built analytics storesthat are optimized for specific use cases. Customers want the freedom to move data between their centralized data lakes and the surrounding purpose-built analytics stores. And they want to get insights with speed and agility in a seamless, secure, and compliant manner. We call this modern approach to analytics theLake House architecture.Figure 2. Lake House architecture on AWSRefer to the whitepaperDerive Insights from AWS Lake housefor various design patterns to derive persona-centric analytics by using the AWS Lake House approach. Check out the blog postBuild a Lake House Architecture on AWSfor a Lake House reference architecture on AWS.ConclusionIn this post, we show you how to build a persona-centric data platform on AWS with a seven-layered approach. This usesApache NiFias a data ingestion tool and AWS purpose-built analytics services for persona-centric analytics and machine learning. We have also shown how to build persona-centric analytics by using the AWS Lake House approach.With the information in this post, you can now build your own data platform on AWS to gain faster and deeper insights from your data. AWS provides you the broadest and deepest portfolio of purpose-built analytics and machine learning services to support your business needs.Read more and get started on building a data platform on AWS:AWSpurpose-built analytics servicesAWS Lake HouseAmazon SageMakerand otherArtificial intelligence (AI) servicesfor data predictionTAGS:Apache Nifi,AWS analytics,AWS Lake houseRaghavarao SodabathinaRaghavarao Sodabathina is a Principal Solutions Architect at AWS, focusing on Data Analytics, AI/ML, and Serverless Platform. He engages with customers to create innovative solutions that address customer business problems and to accelerate the adoption of AWS services. In his spare time, Raghavarao enjoys spending time with his family, reading books, and watching movies.Jigna GandhiJigna Gandhi is a Solutions Architect at Amazon Web Services, based in the Greater New York City area. She works with AWS Digital Native Business (DNB) customers and specializes in analytics technologies. Outside work, she likes to go out with friends and family and practice martial arts for which she currently holds Brown Belt in Taekwondo.Jiwan PanjikerJiwan Panjiker is a Solutions Architect at Amazon Web Services, based in the Greater New York City area. He works with AWS enterprise customers, helping them in their cloud journey to solve complex business problems by making effective use of AWS services. Outside of work, he likes to spend time with his friends and family, going for long drives, and exploring local cuisine.Rhiannon LiebowitzRhiannon Liebowitz is a Senior Solutions Architect at AWS, based in Vancouver, Canada. He is focused on guiding small to medium business customers to achieve their goals on the AWS Cloud. When not in the office, he enjoys exploring the parks and trails of British Columbia with his family, 3D printing, and game development.
Many organizations are moving their data from silos and aggregating it in one location. Collecting this data in a data lake enables you to perform analytics and machine learning on that data. You can store your data in purpose-built data stores, like a data warehouse, to get quick results for complex queries on structured data. […]
Raghavarao Sodabathina
Amazon Athena Amazon EMR Amazon SageMaker Architecture AWS Glue