Let’s Architect! Architecting with custom chips and accelerators
AWS Architecture BlogLet’s Architect! Architecting with custom chips and acceleratorsby Luca Mezzalira, Laura Hyatt, Vittorio Denti, and Zamira Jaupaj | on21 SEP 2022| inAmazon EC2,Architecture,AWS Inferentia,AWS Lambda,Graviton|Permalink|ShareIt’s hard to imagine a world without computer chips. They are at the heart of the devices that we use to work and play every day. Currently, Amazon Web Services (AWS) is offering customers the next generation of computer chip, with lower cost, higher performance, and a reduced carbon footprint.This edition of Let’s Architect! focuses on custom computer chips, accelerators, and technologies developed by AWS, such asAWS Nitro System, custom-designed Arm-basedAWS Gravitonprocessors that support data-intensive workloads, as well asAWS Trainium, andAWS Inferentiachips optimized for machine learning training and inference.In this post, we discuss these new AWS technologies, their main characteristics, and how to take advantage of them in your architecture.Deliver high performance ML inference with AWS InferentiaAs Deep Learning models become increasingly large and complex, the training cost for these models increases, as well as the inference time for serving.With AWS Inferentia, machine learning practitioners can deploy complex neural-network models that are built and trained on popular frameworks, such as Tensorflow, PyTorch, and MXNet on AWS Inferentia-based Amazon EC2 Inf1 instances.This video introduces you to the main concepts of AWS Inferentia, a service designed to reduce both cost and latency for inference. To speed up inference, AWS Inferentia: selects and shares a model across multiple chips, places pieces inside the on-chip cache, then streams the data via pipeline for low-latency predictions.Presenters discuss through the structure of the chip, software considerations, as well as anecdotes from the Amazon Alexa team, who uses AWS Inferentia to serve predictions. If you want to learn more about high throughput coupled with low latency, exploreAchieve 12x higher throughput and lowest latency for PyTorch Natural Language Processing applications out-of-the-box on AWS Inferentiaon theAWS Machine Learning Blog.AWS Inferentia shares a model across different chips to speed up inferenceAWS Lambda Functions Powered by AWS Graviton2 Processor – Run Your Functions on Arm and Get Up to 34% Better Price PerformanceAWS Lambdais a serverless, event-driven compute service that enables code to run from virtually any type of application or backend service, without provisioning or managing servers. Lambda uses a high-availability compute infrastructure and performs all of the administration of the compute resources, including server- and operating-system maintenance, capacity-provisioning, and automatic scaling and logging.AWS Graviton processors are designed to deliver the best price and performance for cloud workloads. AWS Graviton3 processors are the latest in the AWS Graviton processor family and provide up to: 25% increased compute performance, two-times higher floating-point performance, and two-times faster cryptographic workload performance compared with AWS Graviton2 processors. This means you can migrate AWS Lambda functions to Graviton in minutes, plus get as much as 19% improved performance at approximately 20% lower cost (compared with x86).Comparison between x86 and Arm/Graviton2 results for the AWS Lambda function computing prime numbers (click to enlarge)Powering next-gen Amazon EC2: Deep dive on the Nitro SystemThe AWS Nitro System is a collection of building-block technologies that includes AWS-built hardware offload and security components. It is powering the next generation of Amazon EC2 instances, with a broadening selection of compute, storage, memory, and networking options.In this session, dive deep into the Nitro System, reviewing its design and architecture, exploring new innovations to the Nitro platform, and understanding how it allows for fasting innovation and increased security while reducing costs.Traditionally, hypervisors protect the physical hardware and bios; virtualize the CPU, storage, networking; and provide a rich set of management capabilities. With the AWS Nitro System, AWS breaks apart those functions and offloads them to dedicated hardware and software.AWS Nitro System separates functions and offloads them to dedicated hardware and software, in place of a traditional hypervisorHow Amazon migrated a large ecommerce platform to AWS GravitonIn this re:Invent 2021 session, we learn about the benefits Amazon’s ecommerce Datapath platform has realized with AWS Graviton.With a range of 25%-40% performance gains across 53,000 Amazon EC2 instances worldwide for Prime Day 2021, the Datapath team is lowering their internal costs with AWS Graviton’s improved price performance. Explore the software updates that were required to achieve this and the testing approach used to optimize and validate the deployments. Finally, learn about the Datapath team’s migration approach that was used for their production deployment.AWS Graviton2: core componentsSee you next time!Thanks for exploring custom computer chips, accelerators, and technologies developed by AWS. Join us in a couple of weeks when we talk more about architectures and the daily challenges faced while working with distributed systems.Other posts in this seriesLet’s Architect! Modern data architecturesLet’s Architect! Architecting for the edgeLet’s Architect! Architecting for SustainabilityLet’s Architect! Architecting for Machine LearningLet’s Architect! Architecting for SecurityLet’s Architect! Tools for Cloud ArchitectsLet’s Architect! Architecting for BlockchainLet’s Architect! Architecting microservices with containersLet’s Architect! Using open-source technologies on AWSLet’s Architect! Serverless architecture on AWSLet’s Architect! Creating resilient architectureLet’s Architect! Architecting for governance and managementLet’s Architect! Architecting for front endLet’s Architect! Understanding the build versus buy dilemmaLet’s Architect! Architecting for DevOpsLet’s Architect! Designing Well-Architected systemsLet’s Architect! Architecting for big data workloadsLooking for more architecture content?AWS Architecture Centerprovides reference architecture diagrams, vetted architecture solutions, Well-Architected best practices, patterns, icons, and more!TAGS:cloud,Compute,custom hardware,distributed systems,hardware,high performance computing,Let's Architect,machine learning,serverless,software architecture,virtualizationLuca MezzaliraLuca is Principal Solutions Architect based in London. He has authored several books and is an international speaker. He lent his expertise predominantly in the solution architecture field. Luca has gained accolades for revolutionizing the scalability of front-end architectures with micro-frontends, from increasing the efficiency of workflows, to delivering quality in products.Laura HyattLaura Hyatt is a Solutions Architect for AWS Public Sector and helps Education customers in the UK. Laura helps customers not only architect and develop scalable solutions but also think big on innovative solutions facing the education sector at present. Laura's specialty is IoT, and she is also the Alexa SME for Education across EMEA.Vittorio DentiVittorio Denti is a Machine Learning Engineer at Amazon based in London. After completing his M.Sc. in Computer Science and Engineering at Politecnico di Milano (Milan) and the KTH Royal Institute of Technology (Stockholm), he joined AWS. Vittorio has a background in distributed systems and machine learning. He's especially passionate about software engineering and the latest innovations in machine learning science.Zamira JaupajZamira is an Enterprise Solutions Architect based in the Netherlands. She is highly passionate IT professional with over 10 years of multi-national experience in designing and implementing critical and complex solutions with containers, serverless, and data analytics for small and enterprise companies.
It’s hard to imagine a world without computer chips. They are at the heart of the devices that we use to work and play every day. Currently, Amazon Web Services (AWS) is offering customers the next generation of computer chip, with lower cost, higher performance, and a reduced carbon footprint. This edition of Let’s Architect! […]
Luca Mezzalira
Amazon EC2 Architecture AWS Inferentia AWS Lambda Graviton