Dream11: Blocking application attacks using AWS WAF at scale
AWS Architecture BlogDream11: Blocking application attacks using AWS WAF at scaleby Vatsal Shah, Parag Suchak, Ritesh Kumar, and Sanjana Potdar | on25 MAR 2022| inAmazon Simple Storage Service (S3),Architecture,AWS CloudFormation,AWS Lambda,AWS WAF|Permalink|ShareAs the world’s largest fantasy sports platforms with more than 120 million registered users, Dream11 runs multiple contests simultaneously while processing millions of user requests per minute. Their user-centric and data-driven teams make it a priority to ensure that the Dream11 application (app) remains protected against all kinds of threats and vulnerabilities.Introduction to AWS WAF Security AutomationsAWS WAFis a web application firewall that helps protect apps and APIs against common web exploits and bots. These attacks may affect availability, compromise security, or consume excessive resources. AWS WAF gives you control over how traffic reaches your applications. You can create security rules that control bot traffic and block common attack patterns, such as SQL injection or cross-site scripting (XSS.)AWS WAF Security AutomationsuseAWS CloudFormationto quickly configure AWS WAF rules that help block the following common types of attacks:SQL injectionCross-site scriptingHTTP floodsScanners and probesKnown attacker origins (IP reputation lists)Bots and scrapersIn this blog post, we will explain how Dream11 uses AWS WAF Security Automations to protect its application from scanners and probes attacks.Scanner and probe automationTo understand the scanner and probe automation, let’s look at a realistic attack scenario for a standard app that is protected by AWS WAF. Let’s assume that a malicious user is trying to scan the app and identify loopholes using their custom tool. They plan to conduct injection attacks (such as SQLi, XSS) or directory brute force attacks.The app, secured by AWS WAF, has rules in place to block requests if certain signatures and patterns are matched. AWS WAF cannot have all possible payload lists for each attack vector. This means that after some trial and errors, an attacker may find the payload that doesn’t get blocked by AWS WAF and try to exploit the vulnerability.In this case, what if AWS WAF can detect the behavior of malicious user IPs and block it for a certain time period? Wouldn’t it be great if AWS WAF blocks the IP of a malicious user after receiving a couple of malicious requests? That way, new requests coming from that IP will be blocked without AWS WAF having to check all the rules in the web ACL. Any successful bypass attempts will also get blocked from that IP. Rather than permanently blocking the IP, this feature blocks the offending IP for a certain time period, discouraging the attacker from any further attempts. It acts as a first step of incident response. Here’s where automation can help.Scanner and probe automation monitorsAmazon CloudFrontlogs and analyses HTTP status codes for requests coming from different IPs. Based on the configured threshold of HTTP status codes, scanner and probe automation will update the malicious IP directly to the AWS WAF rule IPSet. It then blocks subsequent requests from that IP for a configured period of time.The AWS WAF Security Automations solution creates an AWS WAF rule, anAWS Lambdafunction, and a Scanner and ProbesAmazon Athenaquery. The Athena query parses Amazon CloudFront orApplication Load Balanceraccess logs at regular intervals. It counts the number of bad requests per minute from unique source IP addresses. The Lambda function updates the AWS WAF IPSet rule to block further scans from IP addresses with a high error rate.Scanner and probe solutionFigure 1. Solution architecture for scanner and probe automation (xxx represents the numbers as defined by the use case)The workflow of the solution is as follows, shown in Figure 1:CloudFront logs are pushed to theAmazon S3bucketLog Parser Lambda will run the Athena query to find the error code threshold for each unique IPIf the HTTP error threshold is crossed for any IP, the Lambda function will update the IP into an AWS WAF IPSet for a certain timeThe IPSet is unblocked automatically after the time period is overCustomizing the AWS WAF Security Automation solutionScanner and probe automation with rules will block traffic if the error rate for a particular IP crosses the threshold. It then adds the IP in the blocked IPSet. This IP is blocked for a configurable amount of time (for example, 12 hours, 2 days, 1 week).During the customization of AWS WAF for Dream11, there were instances which required exceptions to the preceding rule. One was to prevent internal services/gateway IPs from getting blocked by the security automation. We needed to customize the rules for these predefined thresholds. For example: the solution should block the external traffic, butexcludeany internal IP addresses.The Dream11 Security team customized the Lambda logic to approve all internal NAT gateway IPs. Scanner and probe automation ignores these IPs even if there is a high number of errors from the approved IPs. Sample code is as follows:log.info("[update_ip_set] \tIgnore the approved IP ")if ip_type == "IPV4" and source_ip not in outstanding_requesters['ApprovedIPs']:addresses_v4.append(source_ip)elif ip_type == "IPV6" and source_ip not in outstanding_requesters['ApprovedIPs']:                     addresses_v6.append(source_ip)Note:Create a JSON file with list of approved IPs and store it inAPP_ACCESS_LOG_BUCKETWe will use the same S3 bucket to put our office-approved IPs as xyz.json file where we store our CloudFront access logs. This is configurable during CloudFormation template for Security Automation.Code explanation:The custom code first validates the particular IP for which the error threshold is crossed against the approved IPs.If the IP belongs to the IPV4 or IPV6 format and isn’t an approved IP, it will be appended to the blocked IPSet for a certain period of time.The customization of the Lambda function provides a security automation solution that doesn’t block any legitimate request. At the same time, it provides protection against scanner and probe attacks. AWS WAF security automation is an open-source solution and is hosted onGitHub.ConclusionIn this blog post, we’ve given a brief overview of how you can reduce attacks by using AWS WAF Security Automations against scanners and probes. We’ve also illustrated the customization implemented by the Dream11 security team.By automating your security operations, you will improve effective incident response. You can prioritize threats and handle cyber attacks automatically with automated courses of action. This reduces the need for human intervention, reduces response time, and addresses security issues without manual effort.After implementing this at Dream11, we were able to create custom, application-specific rules that blocked attack patterns. This has provided application availability, secure resources, and has prevented excessive resource consumption. With this solution, we are able to provide the best fantasy sports experience for over 120 million users.Read more aboutSecurity Automations in AWS WAF.Vatsal ShahVatsal Shah is a Senior Solutions Architect at AWS based out of Mumbai, India. Vatsal has more than 8 years of industry experience, including roles in cloud architecture, systems engineering, and software design. He currently focuses on enabling large startups to streamline their cloud operations and optimize their cloud spend. His area of interest is in machine learning.Parag SuchakParag Suchak is a Security Engineer at Dream11 in Mumbai, India, with over six years of experience is Web Application Security, Mobile Security, and Automation. His professional expertise includes application penetration testing, secure code review, Network VAPT, and Cloud Security.Ritesh KumarRitesh Kumar is a Security Engineer at Dream11 in Mumbai, India. He has a background in Application Security, and provides expertise in architectures that reduce the likelihood of data leakage, application, and denial-of-service attacks. He enjoys working on complex topics and processes, and breaking them down into simpler components.Sanjana PotdarSanjana Potdar is a cloud Security Engineer at Dream11 in Mumbai, India. She has a master’s degree in Computer Science with a major in System Security. She is passionate about security, cloud services, and building solutions around them. Outside of work, she is an avid reader and traveler.
As the world’s largest fantasy sports platforms with more than 120 million registered users, Dream11 runs multiple contests simultaneously while processing millions of user requests per minute. Their user-centric and data-driven teams make it a priority to ensure that the Dream11 application (app) remains protected against all kinds of threats and vulnerabilities. Introduction to AWS […]
Vatsal Shah
Amazon Simple Storage Service (S3) Architecture AWS CloudFormation AWS Lambda AWS WAF