Migrating a Database Workflow to Modernized AWS Workflow Services
AWS Architecture BlogMigrating a Database Workflow to Modernized AWS Workflow Servicesby Scott Wainner and Harpreet Virk | on10 DEC 2021| inAmazon DynamoDB,Amazon RDS,Amazon Simple Notification Service (SNS),Amazon Simple Queue Service (SQS),Architecture,AWS Lambda,AWS Step Functions|Permalink|ShareThe relational database is a critical resource in application architecture. Enterprise organizations often use relational database management systems (RDBMS) to provide embedded workflow state management. But this can present problems, such as inefficient use of data storage and compute resources, performance issues, and decreased agility. Add to this the responsibility of managing workflow states through custom triggers and job-based algorithms, which further exacerbate the performance constraints of the database. The complexity of modern workflows, frequency of runtime, and external dependencies encourages us to seek alternatives to using these database mechanisms.This blog describes how to use modernized workflow methods that will mitigate database scalability constraints. We’ll show how transitioning your workflow state management from a legacy database workflow to AWS services enables new capabilities with scale.A workflow system is composed of an ordered set of tasks. Jobs are submitted to the workflow where tasks are initiated in the proper sequence to achieve consistent results. Each task is defined with a taskinput criterion, taskaction, taskoutput, and taskdisposition, see Figure 1.Figure 1. Task with input criteria, an action, task output, and task dispositionEmbedded WorkflowFigure 2 depicts the database serving as the workflow state manager where an external entity submits a job for execution into the database workflow. This can be challenging, as the embedded workflow definition requires the use of well-defined database primitives. In addition, any external tasks require tight coupling with database primitives that constrains workflow agility.Figure 2. Embedded database workflow mechanisms with internal and external task entitiesExternalized workflowA paradigm change is made with use of a modernized workflow management system, where the workflow state exists external to the relational database. A workflow management system is essentially a modernized database specifically designed to manage the workflow state (depicted in Figure 3.)Figure 3. External task manager extracting workflow state, job data, performing the task, and re-inserting the job data back into the databaseAWS offers two workflow state management services:Amazon Simple Workflow Service (Amazon SWF)andAWS Step Functions. The workflow definition and workflow state are no longer stored in a relational database; these workflow attributes are incorporated into the AWS service. The AWS services are highly scalable, enable flexible workflow definition, and integrate tasks from many other systems, including relational databases. These capabilities vastly expand the types of tasks available in a workflow. Migrating the workflow management to an AWS service reduces demand placed upon the database. In this way, the database’s primary value of representing structured and relational data is preserved. AWS Step Functions offers a well-defined set of taskprimitivesfor the workflow designer. The designer can still incorporate tasks that leverage the inherent relational database capabilities.Pull and push workflow modelsFirst, we must differentiate between Amazon SWF and AWS Step Functions to determine which service is optimal for your workflow. Amazon SWF uses an HTTPS APIpull modelwhere external Workers and Deciders execute Tasks and assert the Next-Step, respectively. The workflow state is captured in the Amazon SWF history table. This table tracks the state of jobs and tasks so a common reference exists for all the candidate Workers and Deciders.Amazon SWF does require development of external entities that make the appropriate API calls into Amazon SWF. It inherently supports external tasks that require human intervention. This workflow can tolerate long lead times for task execution. The Amazon SWF pull model is represented in the Figure 4.Figure 4. ‘Pull model’ for workflow definition when using Amazon SWFIn contrast, AWS Step Functions uses apush model, shown in Figure 5, that initiates workflow tasks and integrates seamlessly with other AWS services. AWS Step Functions may also incorporate mechanisms that enable long-running tasks that require human intervention. AWS Step Functions provides the workflow state management, requires minimal coding, and provides traceability of all transactions.Figure 5. ‘Push model’ for workflow definition when using AWS Step FunctionsWorkflow optimizationsThe introduction of an external workflow manager such as AWS Step Functions or Amazon SWF, can effectively handle long-running tasks, computationally complex processes, or large media files. AWS workflow managers support asynchronous call-back mechanisms to track task completion. The state of the workflow is intrinsically captured in the service, and the logging of state transitions is automatically captured. Computationally expensive tasks are addressed by invoking high-performance computational resources.Finally, the AWS workflow manager also improves the handling of large data objects. Previously, jobs would transfer large data objects (images, videos, or audio) into a database’s embedded workflow manager. But this impacts the throughput capacity and consumes database storage.In the new paradigm, large data objects are no longer transferred to the workflow as jobs, but asjob pointers. These are transferred to the workflow whenever tasks must reference external object storage systems. The sequence of state transitions can be traced through CloudWatch Events. This verifies workflow completion, diagnostics of task execution (start, duration, and stop) and metrics on the number of jobs entering the various workflows.Large data objects are best captured in more cost-effective object storage solutions such asAmazon Simple Storage Service (Amazon S3). Data records may be conveyed via a variety of NoSQL storage mechanisms including:Amazon DynamoDB: Scalable and fast data retrieval of key-value task datasetsAmazon Simple Notification Service (SNS): Scalable distribution mechanism for tasksAmazon Simple Queue Service (SQS): Asynchronous processing of data for taskThe workflow manager stores pointer references so tasks can directly access these data objects and perform transformation on the data. It provides pointers to the results without transferring the data objects to the workflow. Transferring pointers in the workflow as opposed to transferring large data objects significantly improves the performance, reduces costs, and dramatically improves scalability. You may continue to use the RDBMS for the storage of structured data and use its SQL capabilities with structured tables, joins, and stored procedures. AWS Step Functions enable indirect integration with relational databases using tools such as the following:AWS Lambda: Short-lived execution of custom code to handle tasksAWS Glue: Data integration enabling combination and preparation of data including SQLAWS Step Functions can be coupled with AWS Lambda, a serverless compute capability. Lambda code can manipulate the job data and incorporate many other AWS services. AWS Lambda can also interact with any relational database includingAmazon Relational Database Service (RDS)orAmazon Auroraas the executor of a task.The modernized architecture shown in Figure 6 offers more flexibility in creating new workflows that can evolve with your business requirements.Figure 6. Using Step Functions as workflow state managerSummarySeveral key advantages are highlighted with this modernized architecture using either Amazon SWF or AWS Step Functions:You can manage multiple versions of a workflow.Backwards compatibility is maintained as capability expands. Previous business requirements using metadata interpretation on job submission is preserved.Tasks leverage loose coupling of external systems.This provides far more data processing and data manipulation capabilities in a workflow.Upgrades can happen independently.A loosely coupled system enables independent upgrade capabilities of the workflow or the external system executing the task.Automatic scaling.Serverless architecture scales automatically with the growth in job submissions.Managed services.AWS provides highly resilient and fault tolerant managed servicesRecovery.Instance recovery mechanisms can manage workflow state machines.The modernized workflow using Amazon SWF or AWS Step Functions offers many key advantages. It enables application agility to adapt to changing business requirements. By using a managed service, the enterprise architect can focus on the workflow requirements and task actions, rather than building out a workflow management system. Finally, critical intellectual property developed in the RDBMS system can be preserved as tasks in the modernized workflow using AWS services.Further reading:Intro to Service Coordination Using AWS Step FunctionsAWS Step Functions Optimized IntegrationsScott WainnerScott Wainner is a Senior Solutions Architect (SA) focused on identifying and developing modernized methods and patterns that accelerate the customer's journey to the AWS. He has over 35 years of experience in designing, building, and operating complex information systems. He has extensive experience in networking, telecommunications, managed services, and management support systems.Harpreet VirkHarpreet Virk is a Senior Migrations Modernization Solutions Architect who is passionate about databases and helping customers migrate and modernize their workload into AWS. He has over 23 years of experience in large enterprises, and extensive experience in Databases and managing large database teams.
The relational database is a critical resource in application architecture. Enterprise organizations often use relational database management systems (RDBMS) to provide embedded workflow state management. But this can present problems, such as inefficient use of data storage and compute resources, performance issues, and decreased agility. Add to this the responsibility of managing workflow states through […]
Scott Wainner
Amazon DynamoDB Amazon RDS Amazon Simple Notification Service (SNS) Amazon Simple Queue Service (SQS) Architecture AWS Lambda AWS Step Functions