Serverless architecture for optimizing Amazon Connect call-recording archival costs
AWS Architecture BlogServerless architecture for optimizing Amazon Connect call-recording archival costsby Brian Maguire, Clarence Choi, Herbert Guerrero, and Agnel Joseph | on24 JUN 2022| inAmazon Connect,Amazon Simple Queue Service (SQS),Amazon Simple Storage Service (S3),Architecture,AWS Lambda,AWS Step Functions|Permalink|ShareIn this post, we provide a serverless solution to cost-optimize the storage of contact-center call recordings. The solution automates the scheduling, storage-tiering, and resampling of call-recording files, resulting in immediate cost savings. The solution is an asynchronous architecture built usingAWS Step Functions,Amazon Simple Queue Service (Amazon SQS), andAWS Lambda.Amazon Connectprovides an omnichannel cloud contact center with the ability to maintain call recordings for compliance and gaining actionable insights usingContact Lens for Amazon ConnectandAWS Contact Center Intelligence Partners. The storage required for call recordings can quickly increase as customers meet compliance retention requirements, often spanning six or more years. This can lead to hundreds of terabytes in long-term storage.Solution overviewWhen an agent completes a customer call, Amazon Connect sends the call recording to anAmazon Simple Storage Solution (Amazon S3)bucket with: a date and contact ID prefix, the file stored in the .WAV format and encoded using bitrate 256 kb/s, pcm_s16le, 8000 Hz, two channels, and 256 kb/s. The call-recording files are approximately 2 Mb/minute optimized for high-quality processing, such as machine learning analysis (see Figure 1).Figure 1. Asynchronous architecture for batch resampling for call-recording files on Amazon S3When a call recording is sent to Amazon S3, downstream post-processing is often performed to generate analytics reports for agents and quality auditors. The downstream processing can include services that provide transcriptions, quality-of-service metrics, and sentiment analysis to create reports and trigger actionable events.While this processing is often completed within minutes, the downstream applications could require processing retries. As audio resampling reduces the quality of the audio files, it is essential to delay resampling until after processing is completed. As processed call recordings are infrequently accessed days after a call is completed, with only a small percentage accessed by agents and call quality auditors, call recordings can benefit from resampling and transitioning to long-term Amazon S3 storage tiers.In Figure 2, multiple AWS services work together to provide an end-to-end cost-optimization solution for your contact center call recordings.Figure 2. AWS Step Function orchestrates the batch resampling of call recordingsAnAmazon EventBridgeschedule rule triggers the step function to perform the batch resampling process for all call recordings from the previous 7 days.In the first step function task, the Lambda function task iterates the S3 bucket using theListObjectsV2API, obtaining the call recordings (1000 objects per iteration) with the date prefix from 7 days ago.The next task invokes a Lambda function inserting the call recording objects into the Amazon SQS queue. The audio-conversion Lambda function receives the Amazon SQS queue events via theevent source mappingLambda integration. Each concurrent Lambda invocation downloads a stored call recording from Amazon S3, resampling the .WAV withffmpegand tagging the S3 object with a “converted=True” tag.Finally, the conversion function uploads the resampled file to Amazon S3, overwriting the original call recording with the resampled recording using a cost-optimized storage class, such asS3 Glacier Instant Retrieval. S3 Glacier Instant Retrieval provides the lowest cost for long-lived data that is rarely accessed and requires milliseconds retrieval, such as for contact-center call-recording playback. By default, Amazon Connect stores call recordings withS3 Versioningenabled, maintaining the original file as a version. You can use lifecycle policies todelete object versionsfrom a version-enabled bucket to permanently remove the original version, as this will minimize the storage of the original call recording.This solution captures failures within the step function workflow with logging and a dead-letter queue, such as when an error occurs with resampling a recording file. A Step Function task monitors the Amazon SQS queue using theAWS Step Function integration with AWS SDKwith SQS and ending the workflow when the queue is emptied. Table 1 demonstrates the default and resampled formats.Figure 3. Detailed AWS Step Functions state machine diagramResamplingTable 1. Default and resampled call recording audio formatsAudio sampling formatsFile size/minuteNotesBitrate 256 kb/s, pcm_s16le, 8000 Hz, 2 channels, 256 kb/s2 MBThe default for Amazon Connect call recordings. Sampled for audio quality and call analytics processing.Bitrate 64 kb/s, pcm_alaw, 8000 Hz, 1 channel, 64 kb/s0.5 MBResampled to mono channel 8 bit. This resampling is not reversible and should only be performed after all call analytics processing has been completed.Cost assessmentFor pricing information for the primary services used in the solution, visit:Amazon S3 pricingAWS Step Functions PricingAWS Lambda PricingAmazon SQS PricingAmazon CloudWatch pricingThe costs incurred by the solution are based on usage and are AWS Free Tier eligible. After the AWS Free Tier allowance is consumed, usage costs are approximately $0.11 per 1000 minutes of call recordings. S3 Standard starts at $0.023 per GB/month; and S3 Glacier Instant Retrieval is $0.004 per GB/month, with $0.003 per GB of data retrieval. During a 6-year compliance retention term, the schedule-based resampling and storage tiering results in significant cost savings.In the 6-year example detailed in Table 2, the S3 Standard storage costs would be approximately $356,664 for 3 million call-recording minutes/month. The audio resampling and S3 Glacier Instant Retrieval tiering reduces the 6-year cost to approximately $41,838.Table 2. Multi-year costs savings scenario (3 million minutes/month) in USDYearTotal minutes (3 million/month)Total storage (TB)Cost of storage, S3 Standard (USD)Cost of running the resampling (USD)Cost of resampling solution with S3 Glacier Instant Retrieval (USD)136,000,0007210,7643,9604,813272,000,00010830,6363,9605,6773108,000,00014450,5083,9606,5414144,000,00018070,3803,9607,4055180,000,00021690,2523,9608,2696216,000,000252110,1243,9609,133Total1,008,000,000972356,66423,76041,838To explore PCA costs for yourself, useAWS Cost Exploreror chooseBill Detailson theAWS Billing Dashboardto see your month-to-date spend by service.Deploying the solutionThe code and documentation for this solution are available by cloning thegit repositoryand can be deployed withAWS Cloud Development Kit (AWS CDK).Bash# clone repositorygit clone https://github.com/aws-samples/amazon-connect-call-recording-cost-optimizer.git# navigate the project directorycd amazon-connect-call-recording-cost-optimizerModify thecdk.context.jsonwith your environment’s configuration setting, such as thebucket_name. Next, install the AWS CDK dependencies and deploy the solution::# ensure you are in the root directory of the repository./cdk-deploy.shOnce deployed, you can test the resampling solution by waiting for the EventBridge schedule rule to execute based on thenum_days_agesetting that is applied. You can also manually run the AWS Step Function with a specified date, for example{"specific_date":"01/01/2022"}.The AWS CDK deployment creates the following resources:AWS Step FunctionAWS Lambda functionAmazon SQS queuesAmazon EventBridge ruleThe solution handles the automation of transitioning a storage tier, such as S3 Glacier Instant Retrieval. In addition,Amazon S3 Lifecyclescan be set manually to transition the call recordings after resampling to alternativeAmazon S3 Storage Classes.CleanupWhen you are finished experimenting with this solution, cleanup your resources by running the command:cdk destroyThis command deletes the AWS CDK-deployed resources. However, the S3 bucket containing your call recordings and CloudWatch log groups are retained.ConclusionThis call recording resampling solution offers an automated, cost-optimized, and scalable architecture to reduce long-term compliance call recording archival costs.Brian MaguireBrian Maguire is a Principal Solutions Architect at Amazon Web Services, where he is focused on helping customers build their ideas in the cloud. He is a technologist, writer, teacher, and student who loves learning. Brian is the co-author of the book Scalable Data Streaming with Amazon Kinesis.Clarence ChoiClarence Choi is a Solution Architect Specialist for Amazon Connect. He has been in the Contact Center technologies industry for 20+ years and previously worked in Oracle, Cisco, and Avaya helping customer improve their customer experience.Herbert GuerreroHerbert Guerrero is a Proserv Consultant at Amazon Web Services, where he is focused on streamlining Contact Centers using Amazon Connect along with other ancillary services. He loves working with customers to establish attainable strategies and build scalable Contact Center solutions.Agnel JosephAgnel Joseph is a Proserv Consultant at Amazon Web Services, where he is focused on helping customers to deploy their contact center with Amazon Connect, along with other supporting services. He is a technologist and student who loves learning and creating new products.
In this post, we provide a serverless solution to cost-optimize the storage of contact-center call recordings. The solution automates the scheduling, storage-tiering, and resampling of call-recording files, resulting in immediate cost savings. The solution is an asynchronous architecture built using AWS Step Functions, Amazon Simple Queue Service (Amazon SQS), and AWS Lambda. Amazon Connect provides an […]
Brian Maguire
Amazon Connect Amazon Simple Queue Service (SQS) Amazon Simple Storage Service (S3) Architecture AWS Lambda AWS Step Functions