Let’s Architect! Open-source technologies on AWS
AWS Architecture BlogLet’s Architect! Open-source technologies on AWSby Vittorio Denti, Federica Ciuffo, Laura Hyatt, Luca Mezzalira, and Zamira Jaupaj | on21 JUN 2023| inAmazon Managed Streaming for Apache Kafka (Amazon MSK),Architecture,AWS Cloud Development Kit,AWS Lambda,Thought Leadership|Permalink|ShareWe brought you aLet’s Architect!blog postabout open-source on AWS that covered some technologies with development led by AWS/Amazon, as well as well-known solutions available on managed AWS services. Today, we’re following the same approach to share more insights about the process itself for developing open-source. That’s why the first topic we discuss in this post is a re:Invent talk from Heitor Lessa, Principal Solutions Architect at AWS, explaining some interesting approaches for developing and scaling successful open-source projects.This edition ofLet’s Architect!also touches on observability with Open Telemetry, Apache Kafka on AWS, and Infrastructure as Code with an hands-on workshop onAWS Cloud Development Kit(AWS CDK).Powertools for AWS Lambda: Lessons from the road to 10 million downloadsPowertools for AWS Lambdais an open-source library to help engineering teams implement serverless best practices. In two years, Powertools went from an initial prototype to a fast-growing project in the open-source world. Rapid growth along with support from a wide community led to challenges from balancing new features with operational excellence to triaging bug reports and RFCs and scaling and redesigning documentation.In this session, you can learn aboutPowertools for AWS Lambdato understand what it is and the problems it solves. Moreover, there are many valuable lessons to learn how to create and scale a successful open-source project. From managing the trade-off between releasing new features and achieving operational stability to measuring the impact of the project, there are many challenges in open-source projects that require careful thought.Take me to this video!Heitor Lessa describing one of the key lessons: development and releasing new features should be as important as the other activities (governance, operational excellence, and more).Observability the open-source wayThe recent blog postLet’s Architect!Monitoring production systems at scaletalks about the importance of monitoring. Setting up observability is critical to maintain application and infrastructure health, but instrumenting applications to collect monitoring signals such as metrics and logs can be challenging when using vendor-specific SDKs.This video introduces you toOpenTelemetry, an open-source observability framework. OpenTelemetry provides a flexible, single vendor-agnostic SDK based on open-source specifications that developers can use to instrument and collect signals from applications. This resource explains how it works in practice and how to monitor microservice-based applications with the OpenTelemetry SDK.Take me to this video!With AWS Distro for OpenTelemetry, you can collect data from your AWS resources.Best practices for right-sizing your Apache Kafka clusters to optimize performance and costApache Kafkais an open-source streaming data store that decouples applications producing streaming data (producers) into its data store from applications consuming streaming data (consumers) from its data store.Amazon Managed Streaming for Apache Kafka(Amazon MSK) allows you to use the open-source version of Apache Kafka with the service managing infrastructure and operations for you.This blog post explains how the underlying infrastructure configuration can affect Apache Kafka performance. You can learn strategies on how to size the clusters to meet the desired throughput, availability, and latency requirements. This resource helps you discover strategies to find the optimal sizing for your resources, and learn the mental models adopted to conduct the investigation and derive the conclusions.Take me to this blog!Comparisons of put latencies for three clusters with different broker sizesAWS Cloud Development Kit workshopAWS Cloud Development Kit(AWS CDK) is an open-source software development framework that allows you to provision cloud resources programmatically (Infrastructure as Code or IaC) by using familiar programming languages such as Python, Typescript, Javascript, Java, Go, and C#/.Net.CDK allows you to create reusable template and assets, test your infrastructure, make deployments repeatable, and make your cloud environment stable by removing manual (and error-prone) operations. This workshop introduces you to CDK, where you can learn how to provision an initial simple application as well as become familiar with more advanced concepts like CDK constructs.Take me to this workshop!This construct can be attached to any Lambda function that is used as an API Gateway backend. It counts how many requests were issued to each URL.See you next time!Thanks for joining our conversation! To find all the blogs from this series, you can check out theLet’s Architect! list of content on theAWS Architecture Blog.TAGS:devops engineer,Kafka,lambda,Let's Architect,Open Source,well architectedVittorio DentiVittorio Denti is a Machine Learning Engineer at Amazon based in London. After completing his M.Sc. in Computer Science and Engineering at Politecnico di Milano (Milan) and the KTH Royal Institute of Technology (Stockholm), he joined AWS. Vittorio has a background in distributed systems and machine learning. He's especially passionate about software engineering and the latest innovations in machine learning science.Federica CiuffoFederica is a Solutions Architect at Amazon Web Services. She is specialized in container services and is passionate about building infrastructure with code. Outside of the office, she enjoys reading, drawing, and spending time with her friends, preferably in restaurants trying out new dishes from different cuisines.Laura HyattLaura Hyatt is a Solutions Architect for AWS Public Sector and helps Education customers in the UK. Laura helps customers not only architect and develop scalable solutions but also think big on innovative solutions facing the education sector at present. Laura's specialty is IoT, and she is also the Alexa SME for Education across EMEA.Luca MezzaliraLuca is Principal Solutions Architect based in London. He has authored several books and is an international speaker. He lent his expertise predominantly in the solution architecture field. Luca has gained accolades for revolutionizing the scalability of front-end architectures with micro-frontends, from increasing the efficiency of workflows, to delivering quality in products.Zamira JaupajZamira is an Enterprise Solutions Architect based in the Netherlands. She is highly passionate IT professional with over 10 years of multi-national experience in designing and implementing critical and complex solutions with containers, serverless, and data analytics for small and enterprise companies.
We brought you a Let’s Architect! blog post about open-source on AWS that covered some technologies with development led by AWS/Amazon, as well as well-known solutions available on managed AWS services. Today, we’re following the same approach to share more insights about the process itself for developing open-source. That’s why the first topic we discuss […]
Vittorio Denti
Amazon Managed Streaming for Apache Kafka (Amazon MSK) Architecture AWS Cloud Development Kit AWS Lambda Thought Leadership