Use AWS Step Functions to Monitor Services Choreography
AWS Architecture BlogUse AWS Step Functions to Monitor Services Choreographyby Vito De Giosa | on22 DEC 2021| inAdvanced (300),Amazon DynamoDB,Amazon EventBridge,Architecture,AWS Lambda,AWS Step Functions,Serverless|Permalink|ShareOrganizations frequently need access to quick visual insight on the status of complex workflows. This involves collaboration across different systems. If your customer requires assistance on an order, you need an overview of the fulfillment process, including payment, inventory, dispatching, packaging, and delivery. If your products are expensive assets such as cars, you must track each item’s journey instantly.Modern applications useevent-driven architecturesto manage the complexity of system integration at scale. These often usechoreographyfor service collaboration. Instead of directly invoking systems to perform tasks, services interact by exchanging events through a centralized broker. Complex workflows are the result of actions each service initiates in response to events produced by other services. Services do not directly depend on each other. This increases flexibility, development speed, and resilience.However, choreography can introduce two main challenges for the visibility of your workflow.It obfuscates the workflow definition.The sequence of events emitted by individual services implicitly defines the workflow. There is no formal statement that describes steps, permitted transitions, and possible failures.It might be harder to understand the status of workflow executions.Services act independently, based on events. You can implement distributed tracing to collect information related to a single execution across services. However, getting visual insights from traces may require custom applications. This increases time to market (TTM) and cost.To address these challenges, we will show you how to useAWS Step Functionsto model choreographies as state machines. The solution enables stakeholders to gain visual insights on workflow executions, identify failures, and troubleshoot directly from theAWS Management Console.ThisGitHub repositoryprovides a Quick Start and examples on how to model choreographies.Modeling choreographies with Step FunctionsMonitoring a choreography requires a formal representation of the distributed system behavior, such as state machines. State machines are mathematical models representing the behavior of systems through states and transitions.Statesmodel situations in which the system can operate.Transitionsdefine which input causes a change from the current state to the next. They occur when a new event happens. Figure 1 shows a state machine modeling an order workflow.Figure 1. Order workflowThe solution in this post usesAmazon State Languageto describe a choreography as a Step Functions state machine. The state machine pauses, usingTaskstates combined with acallback integration pattern. It then waits for the next event to be published on the broker.Choicestates control transitions to the next state by inspecting event payloads. Figure 2 shows how the workflow in Figure 1 translates to a Step Functions state machine.Figure 2. Order workflow translated into Step Functions state machineFigure 3 shows the architecture for monitoring choreographies with Step Functions.Figure 3. Choreography monitoring with AWS Step FunctionsServices involved in the choreography publish events toAmazon EventBridge. There are two configuredrules. The first rule matches the first event of the choreography sequence,Order Placedin the example. The second rule matches any other event of the sequence. Event payloads contain acorrelation id(order_id) to group them by workflow instance.The first rule invokes anAWS Lambdafunction, which starts a new execution of the choreography state machine. The correlation id is passed in thenameparameter, so you can quickly identify an execution in the AWS Management Console.The state machine uses Task states withAWS SDK service integrations, to directly callAmazon DynamoDB. Tasks are configured with a callback pattern. They issue a token, which is stored in DynamoDB with the execution name. Then, the workflow pauses.A service publishes another event on the event bus.The second rule invokes another Lambda function with the event payload.The function uses the correlation id to retrieve the task token from DynamoDB.The function invokes theStep Functions SendTaskSuccess API, with the token and the event payload as parameters.The state machine resumes the execution and uses Choice states to transition to the next state. If the choreography definition expects the received event payload, it selects the next state and the process will restart from Step # 3. The state machine transitions to a Fail state when it receives an unexpected event.Increased visibility with Step Functions consoleModeling service choreographies as Step FunctionsStandard Workflowsincreases visibility with out-of-the-box features.1. You can centrally track events produced by distributed components.Step Functions records full execution history for 90 days after the execution completes. You’ll be able to capture detailed information about the input and output of each state, including event payloads. Additionally, state machines integrate withAmazon CloudWatchto publish execution logs and metrics.2. You can monitor choreographies visually.The Step Functions console displays a list of executions with information such as execution id, status, and start date (see Figure 4).Figure 4. Step Functions workflow dashboardAfter you’ve selected an execution, a graph inspector is displayed (see Figure 5). It shows states, transitions, and marks individual states with colors. This identifies at a glance, successful tasks, failures, and tasks that are still in progress.Figure 5. Step Functions graph inspector3. You can implement event-driven automation.Step Functions enables you to capture execution status changesemitting events directly to EventBridge(see Figure 6). Additionally, AWS gives you the ability to emit events by setting alarms on top of metrics. Step Functions publishes these to CloudWatch. You can respond to events by initiating corrective actions, sending notifications, or integrating with third-party solutions, such as issue tracking systems.Figure 6. Automation with Step Functions, EventBridge, and CloudWatch alarmsEnabling access to AWS Step Functions consoleStakeholders need secure access to the Step Functions console. This requires mechanisms to authenticate users and authorize read-only access to specific Step Functions workflows.AWS Single Sign-Onauthenticates users by directly managing identities or through federation. SSO supports federation with Active Directory andSAML 2.0 compliant external identity providers (IdP). Users gain access to Step Functions state machines by assigning apermission set, which is a collection ofAWS Identity and Access Management (IAM)policies. Additionally, with permission sets, you can configure arelay state, which is a URL to redirect the user after successful authentication. You can authenticate the user through the selected identity provider and immediately show the AWS Step Functions console with the workflow state machine already displayed. Figure 7 shows this process.Figure 7. Access to Step Functions state machine with AWS SSOThe user logs in through the selected identity provider.The SSO user portal uses the SSO endpoint to send the response from the previous step. SSO usesAWS Security Token Service (STS)to get temporary security credentials on behalf of the user. It then creates a console sign-in URL using those credentials and the relay state. Finally, it sends the URL back as a redirect.The browser redirects the user to the Step Functions console.When the identity provider does not support SAML 2.0, SSO is not a viable solution. In this case, you cancreate a URL with a sign-in tokenfor users to securely access the AWS Management Console. This approach uses STSAssumeRoleto get temporary security credentials. Then, it uses credentials to obtain a sign-in token from the AWS federation endpoint. Finally, it constructs a URL for the AWS Management Console, which includes the token. It then distributes this to users to grant access. This is similar to the SSO process. However, it requires custom development.ConclusionThis post shows how you can increase visibility on choreographed business processes using AWS Step Functions. The solution provides detailed visual insights directly from the AWS Management Console, without requiring custom UI development. This reduces TTM and cost.To learn more:Operating Lambda: understanding event-driven architectureChallenges with distributed systemsImplementing Microservices on AWSVito De GiosaVito De Giosa is a Senior Solutions Architect at Amazon Web Services. He helps organizations on their cloud strategy and enables technical teams to build cloud native applications. He loves talking about modernization and serverless, and when he's not doing so he enjoys spending time with family, playing congas, or listening to music.
Organizations frequently need access to quick visual insight on the status of complex workflows. This involves collaboration across different systems. If your customer requires assistance on an order, you need an overview of the fulfillment process, including payment, inventory, dispatching, packaging, and delivery. If your products are expensive assets such as cars, you must track […]
Vito De Giosa
Advanced (300) Amazon DynamoDB Amazon EventBridge Architecture AWS Lambda AWS Step Functions Serverless