Best practices for implementing event-driven architectures in your organization
AWS Architecture BlogBest practices for implementing event-driven architectures in your organizationby Emanuele Levi | on24 JUL 2023| inAmazon EventBridge,Amazon Managed Streaming for Apache Kafka (Amazon MSK),Amazon Simple Notification Service (SNS),Amazon Simple Queue Service (SQS),Architecture,Kinesis Data Streams|Permalink|Comments|ShareEvent-driven architectures (EDA) are made up of components that detect business actions and changes in state, and encode this information in event notifications. Event-driven patterns are becoming more widespread in modern architectures because:they are the main invocation mechanism in serverless patterns.they are the preferred pattern for decoupling microservices, where asynchronous communications and event persistence are paramount.they are widely adopted as a loose-coupling mechanism between systems in different business domains, such as third-party or on-premises systems.Event-driven patterns have the advantage of enabling team independence through the decoupling and decentralization of responsibilities. This decentralization trend in turn, permits companies to move with unprecedented agility, enhancing feature development velocity.In this blog, we’ll explore the crucial components and architectural decisions you should consider when adopting event-driven patterns, and provide some guidance on organizational structures.Division of responsibilitiesThe communications flow in EDA (seeWhat is EDA?) is initiated by the occurrence of an event. Most production-grade event-driven implementations have three main components, as shown in Figure 1: producers, message brokers, and consumers.Figure 1. Three main components of an event-driven architectureProducers, message brokers, and consumers typically assume the following roles:ProducersProducersare responsible for publishing the events as they happen. They are the owners of the event schema (data structure) and semantics (meaning of the fields, such as the meaning of the value of an enum field). As this is the only contract (coupling) between producers and the downstream components of the system, the schema and its semantics are crucial in EDA. Producers are responsible for implementing a change management process, which involves both non-breaking and breaking changes. With introduction of breaking changes, consumers are able to negotiate the migration process with producers.Producers are “consumer agnostic”, as their boundary of responsibility ends when an event is published.Message brokersMessage brokersare responsible for the durability of the events, and will keep an event available for consumption until it is successfully processed. Message brokers ensure that producers are able to publish events for consumers to consume, and they regulate access and permissions to publish and consume messages.Message brokers are largely “events agnostic”, and do not generally access or interpret the event content. However, some systems provide a routing mechanism based on the event payload or metadata.ConsumersConsumersare responsible for consuming events, and own the semantics of theeffectof events. Consumers are usually bounded to one business context. This means the same event will have differenteffectsemantics for different consumers. Crucial architectural choices when implementing a consumer involve the handling of unsuccessful message deliveries or duplicate messages. Depending on the business interpretation of the event, when recovering from failure a consumer might permit duplicate events, such as with an idempotent consumer pattern.Crucially, consumers are “producer agnostic”, and their boundary of responsibility begins when an event is ready for consumption. This allows new consumers to onboard into the system without changing the producer contracts.Team independenceIn order to enforce the division of responsibilities, companies should organize their technical teams by ownership of producers, message brokers, and consumers. Although the ownership of producers and consumers is straightforward in an EDA implementation, the ownership of the message broker may not be. Different approaches can be taken to identify message broker ownership depending on your organizational structure.Decentralized ownershipFigure 2. Ownership of the message broker in a decentralized ownership organizational structureIn adecentralized ownershiporganizational structure (see Figure 2), the teams producing events are responsible for managing their own message brokers and the durability and availability of the events for consumers.The adoption oftopic fanoutpatterns based onAmazon Simple Queue Service(SQS) andAmazon Simple Notification Service(SNS) (see Figure 3), can help companies implement a decentralized ownership pattern. A bus-based pattern usingAmazon EventBridgecan also be similarly utilized (see Figure 4).Figure 3. Topic fanout pattern based on Amazon SQS and Amazon SNSFigure 4. Events bus pattern based on Amazon EventBridgeThe decentralized ownership approach has the advantage of promoting team independence, but it is not a fit for every organization. In order to be implemented effectively, a well-established DevOps culture is necessary. In this scenario, the producing teams are responsible for managing the message broker infrastructure and the non-functional requirements standards.Centralized ownershipFigure 5. Ownership of the message broker in a centralized ownership organizational structureIn acentralized ownershiporganizational structure, a central team (we’ll call it theplatform team) is responsible for the management of the message broker (see Figure 5). Having a specialized platform team offers the advantage of standardized implementation of non-functional requirements, such as reliability, availability, and security. One disadvantage is that the platform team is a single point of failure in both the development and deployment lifecycle. This could become a bottleneck and put team independence and operational efficiency at risk.Figure 6. Streaming pattern based on Amazon MSK and Kinesis Data StreamsOn top of the implementation patterns mentioned in the previous section, the presence of a dedicated team makes it easier to implement streaming patterns. In this case, a deeper understanding on how the data is partitioned and how the system scales is required. Streaming patterns can be implemented using services such asAmazon Managed Streaming for Apache Kafka(MSK) orAmazon Kinesis Data Streams(see Figure 6).Best practices for implementing event-driven architectures in your organizationThe centralized and decentralized ownership organizational structures enhance team independence or standardization of non-functional requirements respectively. However, they introduce possible limits to the growth of the engineering function in a company. Inspired by the two approaches, you can implement a set of best practices which are aimed at minimizing those limitations.Figure 7. Best practices for implementing event-driven architecturesIntroduce a cloud center of excellence (CCoE).A CCoE standardizes non-functional implementation across engineering teams. In order to promote a strong DevOps culture, the CCoE should not take the form of an external independent team, but rather be a collection of individual members representing the various engineering teams.Decentralize team ownership.Decentralize ownership and maintenance of the message broker to producing teams. This will maximize team independence and agility. It empowers the team to use the right tool for the right job, as long as they conform to the CCoE guidelines.Centralize logging standards and observability strategies.Although it is a best practice to decentralize team ownership of the components of an event-driven architecture, logging standards and observability strategies should be centralized and standardized across the engineering function. This centralization provides for end-to-end tracing of requests and events, which are powerful diagnosis tools in case of any failure.ConclusionIn this post, we have described the main architectural components of an event-driven architecture, and identified the ownership of the message broker as one of the most important architectural choices you can make. We have described a centralized and decentralized organizational approach, presenting the strengths of the two approaches, as well as the limits they impose on the growth of your engineering organization. We have provided some best practices you can implement in your organization to minimize these limitations.Further reading:To start your journey building event-driven architectures in AWS, explore the following:For bus-based and topic-based patterns, seeWhat is an Event-Driven Architecture?For streaming patterns, seeWhat is Apache Kafka?, andAmazon Kinesis.Emanuele LeviEmanuele is a Solutions Architect in the Enterprise Software and SaaS team, based in London. Emanuele helps UK customers on their journey to refactor monolithic applications into modern microservices SaaS architectures. Emanuele is mainly interested in event-driven patterns and designs, especially when applied to analytics and AI, where he has expertise in the fraud-detection industry.CommentsView Comments
Event-driven architectures (EDA) are made up of components that detect business actions and changes in state, and encode this information in event notifications. Event-driven patterns are becoming more widespread in modern architectures because: they are the main invocation mechanism in serverless patterns. they are the preferred pattern for decoupling microservices, where asynchronous communications and event […]
Emanuele Levi
Amazon EventBridge Amazon Managed Streaming for Apache Kafka (Amazon MSK) Amazon Simple Notification Service (SNS) Amazon Simple Queue Service (SQS) Architecture Kinesis Data Streams