Let’s Architect! Architecting for big data workloads
AWS Architecture BlogLet’s Architect! Architecting for big data workloadsby Luca Mezzalira, Laura Hyatt, Vittorio Denti, and Zamira Jaupaj | on10 AUG 2022| inArchitecture,AWS Big Data|Permalink|ShareBig data is often defined by 3 Vs: greatervariety,volumes, andvelocity. Because of the three Vs, big data poses data management challenges that cannot be solved with traditional databases. Not only that, but trying to overcome these issues can lead to scaling problems, bottlenecks, and spiraling costs.To help with this, you need to look at the whole data management pipeline. Don’t worry, AWS offer many tools and best practices to help you architect better for these challenges. In this post, we share insights into how to build and manage big data pipelines in your architecture.Everything You Need to Know About Big Data: From Architectural Principles to Best PracticesThere are so many tools, frameworks, and services for big data. It can be overwhelming to know where to start what best practices to apply.This video distills down good practice and good architecture and principles for big data systems into easy topics and guidance.Manos Samatas presenting the mental models for big data architecturesAWS workshops for big dataThis hands-on practice will show you what’s possible for big data services on AWS.If you are a builder, this AWS workshop catalog includes  several courses on big data and analytics. These resources provide new ideas and how to realize them in practice.AWS workshops can help you learn the cloud services and recommended architectural patternsSecurely share your data across AWS accounts using AWS Lake FormationIt’s very common to share data stored across organizations or business units, but sharing data often comes with security risks.This blog post explains how to share data across accounts viaAWS Lake Formationin a secure and controlled manner, so your data is never exposed to the wrong people.This diagram illustrates the architecture for cross-account accessHow Amazon leverages AWS to deliver analytics at enterprise scaleAmazon.comis a customer of AWS like any other customer. But, as you can imagine,Amazon.comhas very large and very complex datasets with tens of thousands of transactions at any one time.This video goes through howAmazon.comuses AWS technologies to run their business successfully, and how you can add the same architectures and principles for yours.Data warehouse architectures can be used to run queries on large amounts of data from different data sourcesSee you next time!Thanks for joining our discussion on big data architecture! See you in two weeks for more architecture best practices and guidance.Other posts in this seriesLet’s Architect! Architecting for SustainabilityLet’s Architect! Architecting for Machine LearningLet’s Architect! Architecting for SecurityLet’s Architect! Tools for Cloud ArchitectsLet’s Architect! Architecting for BlockchainLet’s Architect! Architecting microservices with containersLet’s Architect! Using open-source technologies on AWSLet’s Architect! Serverless architecture on AWSLet’s Architect! Creating resilient architectureLet’s Architect! Architecting for governance and managementLet’s Architect! Architecting for front endLet’s Architect! Understanding the build versus buy dilemmaLet’s Architect! Architecting for DevOpsLet’s Architect! Designing Well-Architected systemsLooking for more architecture content?AWS Architecture Centerprovides reference architecture diagrams, vetted architecture solutions, Well-Architected best practices, patterns, icons, and more!TAGS:Let's ArchitectLuca MezzaliraLuca is Principal Solutions Architect based in London. He has authored several books and is an international speaker. He lent his expertise predominantly in the solution architecture field. Luca has gained accolades for revolutionizing the scalability of front-end architectures with micro-frontends, from increasing the efficiency of workflows, to delivering quality in products.Laura HyattLaura Hyatt is a Solutions Architect for AWS Public Sector and helps Education customers in the UK. Laura helps customers not only architect and develop scalable solutions but also think big on innovative solutions facing the education sector at present. Laura's specialty is IoT, and she is also the Alexa SME for Education across EMEA.Vittorio DentiVittorio Denti is a Machine Learning Engineer at Amazon based in London. After completing his M.Sc. in Computer Science and Engineering at Politecnico di Milano (Milan) and the KTH Royal Institute of Technology (Stockholm), he joined AWS. Vittorio has a background in distributed systems and machine learning. He's especially passionate about software engineering and the latest innovations in machine learning science.Zamira JaupajZamira is an Enterprise Solutions Architect based in the Netherlands. She is highly passionate IT professional with over 10 years of multi-national experience in designing and implementing critical and complex solutions with containers, serverless, and data analytics for small and enterprise companies.
Big data is often defined by 3 Vs: greater variety, volumes, and velocity. Because of the three Vs, big data poses data management challenges that cannot be solved with traditional databases. Not only that, but trying to overcome these issues can lead to scaling problems, bottlenecks, and spiraling costs. To help with this, you need to […]
Luca Mezzalira
Architecture AWS Big Data