Let’s Architect! Architecting a data mesh
AWS Architecture BlogLet’s Architect! Architecting a data meshby Luca Mezzalira, Federica Ciuffo, Laura Hyatt, Vittorio Denti, and Zamira Jaupaj | on08 MAR 2023| inArchitecture,AWS Well-Architected Framework,AWS Well-Architected Tool,Thought Leadership|Permalink|ShareData architectures were mainly designed around technologies rather than business domains in the past. This changed in 2019, when Zhamak Dehghani introduced thedata mesh. Data mesh is an application of the Domain-Driven-Design (DDD) principles to data architectures: Data is organized into data domains and the data is the product that the team owns and offers for consumption.A data mesh architecture unites the disparate data sources within an organization through centrally managed data-sharing and governance guidelines. Business functions can maintain control over how shared data is accessed because data mesh also solves advanced data security challenges through distributed, decentralized ownership.This edition ofLet’s Architect!introduces data mesh, highlights the foundational concepts of data architectures, and covers the patterns for designing a data mesh in theAWS cloudwith supporting resources.Data lakes, lake houses and data mesh: what, why, and how?Let’s explore a video introduction to data lakes, lake houses, and data mesh. This resource explains how to leverage those concepts to gain greater data insights across different business segments, with a special focus on best practices to build a well-architected, modern data architecture on AWS. It also gives an overview of the AWS cloud services that can be used to create such architectures and describes the fundamental pillars of designing them.Take me to this intro to data lakes, lake houses, and data mesh video!Data mesh is an architecture pattern where data are organized into domains and seen as products to expose for consumptionBuilding data mesh architectures on AWSKnowing what a data mesh architecture is, here is a step-by-step video from re:Invent 2022 on designing one. It covers a use case on howGoDaddyconsidered and implemented data mesh, in addition to:The fundamental pillars behind a well-architected data mesh in the cloudFinding an approach to build a data mesh architecture using native AWS servicesReasons for considering a data mesh architecture where data lakes provide limitations in some scenariosHow data mesh can be applied in practice to overcome themThe mental models to apply during the data mesh design processTake me to this re:Invent 2022 video!In the data mesh architecture the producers expose their data for consumption to the consumers. Access is regulated through a centralized governance layer.Amazon DataZone: Democratize data with governanceNow let’s explore data accessibility as it relates to data mesh architectures.Amazon DataZoneis a new AWS business data catalog allowing you to unlock data across organizational boundaries with built-in governance. This service provides a unified environment where everyone in an organization—from data producers to data consumers—can access, share, and consume data in a governed manner.Here is a video to learn how to apply AWS analytics services to discover, access, and share data across organizational boundaries within the context of a data mesh architecture.Take me to this re:Invent 2022 video!Amazon DataZone accelerates the adoption of the data mesh pattern by making it scalable to high number of producers and consumers.Build a data mesh on AWSFeeling inspired to build? Hands-on experience is a great way to learn and see how the theoretical concepts apply in practice.This workshop teaches you a data mesh architecture building approach on AWS. Many organizations are interested in implementing this architecture to:Move away from centralized data lakes to decentralized ownershipDeliver analytics solutions across business unitsLearn how a data mesh architecture can be implemented with AWS native services.Take me to this workshop!The diagrams shows how to separate the producers, consumers and governance components through a multi-account strategy.See you next time!Thanks for exploring architecture tools and resources with us!Next time we’ll talk about monitoring and observability.To find all the posts from this series, check out theLet’s Architect!page of theAWS Architecture Blog.TAGS:Architecture,Architecture Center,Big Data,data mesh,Let's ArchitectLuca MezzaliraLuca is Principal Solutions Architect based in London. He has authored several books and is an international speaker. He lent his expertise predominantly in the solution architecture field. Luca has gained accolades for revolutionizing the scalability of front-end architectures with micro-frontends, from increasing the efficiency of workflows, to delivering quality in products.Federica CiuffoFederica is a Sr. Containers Solutions Architect at AWS. She is passionate about networking and containers. Outside of the office, she enjoys reading, drawing, and spending time with her friends, preferably in restaurants trying out new dishes from different cuisines.Laura HyattLaura Hyatt is a Solutions Architect for AWS Public Sector and helps Education customers in the UK. Laura helps customers not only architect and develop scalable solutions but also think big on innovative solutions facing the education sector at present. Laura's specialty is IoT, and she is also the Alexa SME for Education across EMEA.Vittorio DentiVittorio Denti is a Machine Learning Engineer at Amazon based in London. After completing his M.Sc. in Computer Science and Engineering at Politecnico di Milano (Milan) and the KTH Royal Institute of Technology (Stockholm), he joined AWS. Vittorio has a background in distributed systems and machine learning. He's especially passionate about software engineering and the latest innovations in machine learning science.Zamira JaupajZamira is an Enterprise Solutions Architect based in the Netherlands. She is highly passionate IT professional with over 10 years of multi-national experience in designing and implementing critical and complex solutions with containers, serverless, and data analytics for small and enterprise companies.
Data architectures were mainly designed around technologies rather than business domains in the past. This changed in 2019, when Zhamak Dehghani introduced the data mesh. Data mesh is an application of the Domain-Driven-Design (DDD) principles to data architectures: Data is organized into data domains and the data is the product that the team owns and […]
Luca Mezzalira
Architecture AWS Well-Architected Framework AWS Well-Architected Tool Thought Leadership