How USAA built an Amazon S3 malware scanning solution
AWS Architecture BlogHow USAA built an Amazon S3 malware scanning solutionby Jonathan Nguyen, Chris Wright, Eduardo Wheelock, Jason Ratcliffe, and Steve Simmons | on28 OCT 2022| inAmazon EC2,Amazon Simple Notification Service (SNS),Amazon Simple Queue Service (SQS),Amazon Simple Storage Service (S3),Architecture,AWS Lambda|Permalink|ShareUnited Services Automobile Association(USAA) is a San Antonio-based insurance, financial services, banking, and FinTech company supporting millions of military members and their families. USAA has partnered with Amazon Web Services (AWS) to digitally transform and build multiple USAA solutions that help keep members safe and save members money and time.Why build a S3 malware scanning solution?As complex companies’ businesses continue to grow, there may be an increased need for collaboration and interactions with outside vendors. Prior to developing anAmazon Simple Storage Solution(Amazon S3) scanning solution, a security review and approval process for application teams to ingest data into anAWS Organizationfrom external vendors’ AWS accounts may be warranted, to ensure additional threats are not being introduced. This could result in a lengthy review and exception process, and subsequently, could hinder the velocity of application teams’ collaboration with external vendors.USAA security standards, like those of most companies, require all data from external vendors to be treated as untrusted, and therefore must be scanned by an antivirus or antimalware solution prior to being ingested by downstream processes within the AWS environment. Companies looking to automate the scanning process may want to consider a solution where all incoming external data flow through a demilitarized drop zone to be scanned, and subsequently released to downstream processes if malware and viruses are not detected.S3 malware scanning solution overviewDedicated AWS accounts should be provisioned for specific data classifications and used as a demilitarized zone (DMZ) for an untrusted staging area. The solution discussed in this blog uses a dedicated staging AWS account that controls the release of Amazon S3 objects to other AWS accounts within an AWS Organization. AWS accounts within an AWS Organization should followsecurity best practicesin terms of infrastructure, networking, logging, and security. External vendors shouldexplicitlybe given limited permissions to appropriate resources in their respective staging S3 bucket.A staging S3 bucket should have specific resource policies restricting which applications and identity and access management (IAM) principals can interact with S3 objects using object attributes, such as object tags, to determine whether an object has been scanned, and what the results of that scan are. Additional guardrails are implemented using Service Control Policies (SCP) to restrict authorized IAM principals to create or modify S3 object attributes (Figure 1).Figure 1. Amazon S3 antivirus and antimalware scanning architecture workflowThe external vendor copies an object to the staging S3 bucket.The staging S3 bucket has event notifications configured and generates an event.The S3 PutObject event is sent to an Object CreatedAmazon Simple Queue Service(Amazon SQS) queue topic.AnAmazon Elastic Compute Cloud(Amazon EC2)Auto Scalinggroup is configured to scale based on messages in the Object Created SQS queue.An antivirus and antimalware scanning service application on the Amazon EC2 instances takes the following actions on objects within the Object Created Amazon SQS queue:a. Tag the S3 object with an “In Progress” status.b. Get the object from the Staging S3 bucket and stores it in a local ephemeral file system.c. Scan the copied object using antivirus or antimalware tool.d. Based on the antivirus or antimalware scan results, tag the S3 object with the scan results (for example, No_Malware_Detected vs. Malware_Detected).e. Create and publish a payload to the Object ScannedAmazon Simple Notification Service(Amazon SNS) topic, allowing application team filtering.f. Delete the message from the Object Created SQS queue.Application teams are subscribed to the Object Scanned SNS topic with a filter for their application.For any objects where a virus or malware is detected, a company can use its cyber threat response team to conduct a thorough analysis and take appropriate actions.USAA built a custom anti-virus and anti-malware scanning application using EC2 instances, using a private, hardenedAmazon Machine Image(AMI). For cost-efficacy purposes, the EC2 automatic scaling event can be configured based on Object Created SQS queue depth and Service Level Objective (SLO). A serverless version of an anti-virus and anti-malware solution can be used instead of an EC2 application, depending on your specific use-case and other factors. Some important factors include antivirus and antimalware tool serverless support, resource tuning and configuration requirements, and additional AWS services to manage that could possibly result in a bottleneck. If your enterprise is going with a serverless approach, you can use open-source tools such asClamAV using Lambda functions.In the event of an infected object, proper guardrails and response mechanisms need to be in place. USAA teams have developed playbooks to monitor the health and performance of S3 scanning solution, as well as responding to detected virus or malware.This cloud native, event-driven solution has benefited multiple USAA application teams who have previously requested the ability to ingest data into AWS workloads from teams outside of USAA’s AWS Organization, and allowed additional capabilities and functionality to better serve their members. To enhance this solution even further, USAA’s security team plans to incorporate additional mechanisms to find specific objects that either failed or required additional processing, without having to scan all objects in the buckets. This can be accomplished by including an additionalAWS Lambdafunction andAmazon DynamoDBtable to track object metadata as objects get added to the Object Created SQS queue for processing. The metadata could possibly include information such as S3 bucket origin,S3 object key, version ID, scan status, and the original S3 event payload to replay the event into the Object Created SQS queue. The Lambda function primarily ensures the DynamoDB table is kept up to date as objects are processed, as well as handling issues for objects that may need to be reprocessed. The DynamoDB table also hastime-to-live(TTL) configured to clear records as they expire from the Staging S3 bucket.ConclusionIn this post, we reviewed how USAA’s Public Cloud Security team facilitated collaboration and interactions with external vendors and AWS workloads securely by creating a scalable solution to scan S3 objects for virus and malware prior to releasing objects downstream. The solution uses native AWS services and can be utilized for any use-cases requiring antivirus or antimalware capabilities. Because the S3 object scanning solution uses EC2 instances, you can use your existing antivirus or antimalware enterprise tool.Jonathan NguyenJonathan Nguyen is a Shared Delivery Team Senior Security Consultant at AWS. His background is in AWS Security with a focus on Threat Detection and Incident Response. Today, he helps enterprise customers develop a comprehensive AWS Security strategy and deploy security solutions at scale, and he trains customers on AWS Security best practices.Chris WrightChris Wright is a Lead Cloud Security Engineer at USAA. He has over 25 years of enterprise IT experience in application design, solution architecture, systems integration, and public cloud security - across a variety of industries including healthcare, manufacturing, and financial services. He is currently focused on securely enabling public cloud solutions to continue providing world-class financial services to the military community.Eduardo WheelockEduardo Wheelock is a Software Engineer at USAA with a background in web development. He helps deliver cloud security objectives as part of a team dedicated to furthering USAA's mission and its presence in the cloud.Jason RatcliffeJason Ratcliffe is a Lead Cloud Security Engineer at USAA. He has over 15 years of enterprise security experience focused in Unix/Linux, Container, and Cloud security within the financial services industry. He helps lead and deliver secure and scalable architectures for USAA and its members.Steve SimmonsSteve Simmons is a Senior Software Engineer in USAA's public cloud Cyber Security Engineering team based in Plano, Texas. Over the past six years, his focus on AWS, DevOps, software development with Golang, and container orchestration with Kubernetes has been centered on the financial services industry. He holds all associate-level AWS certifications and continues to push into specialty and professional-level certifications.
United Services Automobile Association (USAA) is a San Antonio-based insurance, financial services, banking, and FinTech company supporting millions of military members and their families. USAA has partnered with Amazon Web Services (AWS) to digitally transform and build multiple USAA solutions that help keep members safe and save members money and time. Why build a S3 […]
Jonathan Nguyen
Amazon EC2 Amazon Simple Notification Service (SNS) Amazon Simple Queue Service (SQS) Amazon Simple Storage Service (S3) Architecture AWS Lambda