Vertical Integration Strategy Powered by Amazon EventBridge
AWS Architecture BlogVertical Integration Strategy Powered by Amazon EventBridgeby Tiago Oliveira and Rafael Ramos | on02 JUL 2021| inAmazon Athena,Amazon EventBridge,Amazon Simple Storage Service (S3),Architecture,AWS CloudFormation,AWS Lambda,Intermediate (200)|Permalink|ShareOver the past few years, midsize and large enterprises have adoptedvertical integrationas part of their strategy to optimize operations and profitability. Vertical integration consists of separating different stages of the production line from other related departments, such as marketing and logistics. Enterprises implement such strategy to gain full control of their value chain: from the raw material production to the assembly lines and end consumer.To achieve operational efficiency, enterprises must keep a level of independence between departments. However, this can lead to unstandardized operations and communication issues. Moreover, with this kind of autonomy for independent and dynamic verticals, the enterprise may lose some measure of visibility and control. As a result, it becomes challenging to generate a basic report from multiple departments. This blog post provides a high-level solution to integrate your different business verticals, using an event-driven architecture on top ofAmazon EventBridge.Event-driven architectureEvent-driven architectureis an architectural pattern to model communication between services while decoupling applications from each other. Applications scale and fail independently, and a central event bus facilitates the communication between the services in the enterprise. Instead of a particular application sending a request directly to another, it produces an event. The central event router captures it and forwards the message to the proper destinations.For instance, when a customer places a new order on the retail website, the application sends the event to the event bus. Following, the event bus sends the message to the ERP system and the fulfillment center for dispatch. In this scenario, we call the application sending the event, anevent publisher, and the applications receiving the event,event consumers.Because all messages are going through the central event bus, there is clear independence between the applications within the enterprise. Here are some benefits:Application independence occurs even if they belong to the same business workflowYou can plug in more event consumers to receive the same event typeYou can add a data lake to receive all new order events from the retail websiteYou can receive all the events from the payment system and the customer relations departmentThis ensures you can integrate independent departments, increase overall visibility, and make sense of specific processes happening in the organization using the right tools.Implementing event-driven architecture with Amazon EventBridgeEach vertical organically generates lifecycle events. Enterprises can use the event-driven architecture paradigm to make the information flow between the departments by asynchronously exchanging events through the event bus. This way, each department can react to events generated by other departments and initiate processes or actions depending on its business needs.Such an approach creates a dynamic and flexiblechoreographybetween the different participants, which is unique to the enterprise. Such choreography can be followed and monitored using analytics and fine-grained event data collected on the data lake. ReadUsing AWS X-Ray tracing with Amazon EventBridgeto learn how to debug and analyze this kind of distributed application.Figure 1. Architecture diagram depicting enterprise vertical integration with Amazon EventBridgeIn Figure 1, Amazon EventBridge works as the centralevent bus, the core component of this event-driven architecture. Through Amazon EventBridge, each event publisher sends or receives lifecycle events to and from all the other participants. Amazon EventBridge has an advanced routing mechanism using the concept ofrules. Each rule defines up to five targets for the event arriving on the bus. Events are selected based on theevent pattern. You can set up routing rules to determine where to send your data to build application architectures. These will react in real time to your data sources, with event publisher and consumer decoupled.In addition to initiating the heavy routing and distribution of events, Amazon EventBridge can also give real-time insights into how the business runs. Using metrics automatically sent toAmazon CloudWatch, it is possible to see which kinds of events are arriving, and at which rate. You can also see how those events are distributed across the registered targets, and any failures that occur during this distribution. Every event can also be archived using theAmazon EventBridge events archivingfeature.Amazon Simple Storage Service (S3)is the backend storage, ordata lake, for all the events that have ever transited via the event bus. With Amazon S3, customers have a cost-efficient storage service at any scale, with 11 9’s of durability. To help customers manage and secure their data, S3 provides features such asAmazon S3 Lifecycleto optimize costs.S3 Object Lockallows thewrite-once-read-many(WORM) model. You can expand this data and transform it into information using S3. Using services likeAmazon Athena,Amazon Redshift, andAmazon EMR, those events can be transformed, correlated, and aggregated to generate insights on the business. The Amazon S3 data lake can also be the input to adata warehouse, machine learning models, and real-time analytics. Learn more about how to useAmazon S3 as the data lake storage.A critical feature of this solution is the initiation of complex queries on top of the data lake.Amazon API Gatewayprovides one single flexible and elastic API entry point to retrieve data from the data lake. It also can publish events directly to the event bus. For complex queries, Amazon API Gateway can be integrated with an AWS Lambda. It will coordinate the execution of standard SQL queries using Amazon Athena as the query engine. You can read about a fully functional example of such an API calledathena-express.After collecting data from multiple departments, third-party entities, and shop floors, you can use the data to derive business value using cross-organization dashboards. In this way, you can increase visibility over the different entities and make sense of the data from the distributed systems. Even though this design allows you to use your favorite BI tool, we are usingAmazon QuickSightfor this solution. For example, with QuickSight, you can author your interactive dashboards, which includemachine learning-powered insights. Those dashboards can then connect the marketing campaigns data with the sales data. You can measure how effective those campaigns were andforecastthe demand on the production lines.ConclusionIn this blog post, we showed you how to use Amazon EventBridge as an event bus to allow event-driven architectures. This architecture pattern streamlines the adoption of vertical integration. Enterprises can decouple IT systems from each other while retaining visibility into the data they generate. Integrating those systems can happen asynchronously using a choreography approach instead of having an orchestrator as a central component. There are technical challenges to implement this kind of solution, such as maintaining consistency in distributed applications and transactions spanning multiple microservices. Refer to thesaga patternfor microservices-based architecture, andhow to implement it using AWS Step Functions.With a data lake in place to collect all the data produced by IT systems, you can create BI dashboards that provide a holistic view of multiple departments. Moreover, it allows organizations to get better insights into their valuable data and explore other use cases, such as machine learning. To support the data lake creation and management, refer toAWS Lake Formationand aseries of other blog posts.To learn more about Amazon EventBridge from a hands-on perspective, refer to thisEventBridge workshop.TAGS:Event Driven,Vertical IntegrationTiago OliveiraTiago is a Senior Solutions Architect at AWS, where he supports manufacturing customers in Germany. Tiago has over 15 years of experience in the areas of software development, industrial automation, system administration, and DevOps; he is passionate about building solutions to connect the real with the virtual.Rafael RamosRafael is a Solutions Architect at AWS, where he helps ISVs on their journey to the cloud. He spent over 13 years working as a software developer, and is passionate about DevOps and serverless. Outside of work, he enjoys playing tabletop RPG, cooking and running marathons.
Over the past few years, midsize and large enterprises have adopted vertical integration as part of their strategy to optimize operations and profitability. Vertical integration consists of separating different stages of the production line from other related departments, such as marketing and logistics. Enterprises implement such strategy to gain full control of their value chain: from the […]
Tiago Oliveira
Amazon Athena Amazon EventBridge Amazon Simple Storage Service (S3) Architecture AWS CloudFormation AWS Lambda Intermediate (200)