Let’s Architect! Modern data architectures
AWS Architecture BlogLet’s Architect! Modern data architecturesby Luca Mezzalira, Laura Hyatt, Vittorio Denti, and Zamira Jaupaj | on07 SEP 2022| inAnalytics,Architecture,AWS Glue,AWS Lake Formation|Permalink|ShareWith the rapid growth in data coming from data platforms and applications, and the continuous improvements in state-of-the-art machine learning algorithms, data are becoming key assets for companies.Modern data architectures include data mesh—a recent style that represents a paradigm shift, in which data is treated as a product and data architectures are designed around business domains. This type of approach supports the idea of distributed data, where each business domain focuses on the quality of the data it produces and exposes to the consumers.In this edition ofLet’s Architect!, we focus on data mesh and how it is designed on AWS, plus other approaches to adopt modern architectural patterns.Design a data mesh architecture using AWS Lake Formation and AWS GlueDomain Driven Design (DDD) is a software design approach where a solution is divided into domains aligned with business capabilities, software, and organizational boundaries. Unlike software architectures, most data architectures are often designed around technologies rather than business domains.In this blog, you can learn about data mesh, an architectural pattern that applies the principles of DDD to data architectures. Data are organized into domains and considered the product that each team owns and offers for consumption.A data mesh design organizes around data domains. Each domain owns multiple data products with their own data and technology stacksBuilding Data Mesh Architectures on AWSIn this video, discover how to use the data mesh approach in AWS. Specifically, how to implement certain design patterns for building a data mesh architecture with AWS services in the cloud.This is a pragmatic presentation to get a quick understanding of data mesh fundamentals, the benefits/challenges, and the AWS services that you can use to build it. This video provides additional context to theaforementioned blog postand includes several examples on the benefits of modern data architectures.This diagram demonstrates the pattern for sharing data catalogs between producer domains and consumer domainsBuild a modern data architecture on AWS with Amazon AppFlow, AWS Lake Formation, and Amazon RedshiftIn this blog, you can learn how to build a modern data strategy using AWS managed services to ingest data from sources like Salesforce. Also discussed is how to automatically create metadata catalogs and share data seamlessly between the data lake and data warehouse, plus creating alerts in the event of an orchestrated data workflow failure.The second part of the post explains how a data warehouse can be built by using an agile data modeling pattern, as well as how ELT jobs were quickly developed, orchestrated, and configured to perform automated data quality testing.A data platform architecture and the subcomponents used to build itAWS Lake Formation WorkshopWith a modern data architecture on AWS, architects and engineers can rapidly build scalable data lakes; use a broad and deep collection of purpose-built data services; and ensure compliance via unified data access, security, and governance. As data mesh is a modern architectural pattern, you can build it using a service likeAWS Lake Formation.Familiarize yourself with new technologies and services by not only learning how they work, but also to building prototypes and projects to gain hands-on experience. This workshop allows builders to become familiar with the features of AWS Lake Formation and its integrations with other AWS services.A data catalog is a key component in a data mesh architecture.AWS Gluecrawlers interact with data stores and other elements to populate the data catalogSee you next time!Thanks for joining our discussion on data mesh! See you in a couple of weeks when we talk more about architectures and the challenges that we face every day while working with distributed systems.Other posts in this seriesLet’s Architect! Architecting for the edgeLet’s Architect! Architecting for SustainabilityLet’s Architect! Architecting for Machine LearningLet’s Architect! Architecting for SecurityLet’s Architect! Tools for Cloud ArchitectsLet’s Architect! Architecting for BlockchainLet’s Architect! Architecting microservices with containersLet’s Architect! Using open-source technologies on AWSLet’s Architect! Serverless architecture on AWSLet’s Architect! Creating resilient architectureLet’s Architect! Architecting for governance and managementLet’s Architect! Architecting for front endLet’s Architect! Understanding the build versus buy dilemmaLet’s Architect! Architecting for DevOpsLet’s Architect! Designing Well-Architected systemsLet’s Architect! Architecting for big data workloadsLooking for more architecture content?AWS Architecture Centerprovides reference architecture diagrams, vetted architecture solutions, Well-Architected best practices, patterns, icons, and more!TAGS:Big Data,cloud,data architecture,data mesh,distributed systems,Let's Architect,ML,software architectureLuca MezzaliraLuca is Principal Solutions Architect based in London. He has authored several books and is an international speaker. He lent his expertise predominantly in the solution architecture field. Luca has gained accolades for revolutionizing the scalability of front-end architectures with micro-frontends, from increasing the efficiency of workflows, to delivering quality in products.Laura HyattLaura Hyatt is a Solutions Architect for AWS Public Sector and helps Education customers in the UK. Laura helps customers not only architect and develop scalable solutions but also think big on innovative solutions facing the education sector at present. Laura's specialty is IoT, and she is also the Alexa SME for Education across EMEA.Vittorio DentiVittorio Denti is a Machine Learning Engineer at Amazon based in London. After completing his M.Sc. in Computer Science and Engineering at Politecnico di Milano (Milan) and the KTH Royal Institute of Technology (Stockholm), he joined AWS. Vittorio has a background in distributed systems and machine learning. He's especially passionate about software engineering and the latest innovations in machine learning science.Zamira JaupajZamira is an Enterprise Solutions Architect based in the Netherlands. She is highly passionate IT professional with over 10 years of multi-national experience in designing and implementing critical and complex solutions with containers, serverless, and data analytics for small and enterprise companies.
With the rapid growth in data coming from data platforms and applications, and the continuous improvements in state-of-the-art machine learning algorithms, data are becoming key assets for companies. Modern data architectures include data mesh—a recent style that represents a paradigm shift, in which data is treated as a product and data architectures are designed around […]
Luca Mezzalira
Analytics Architecture AWS Glue AWS Lake Formation