Field Notes: How to Identify and Block Fake Crawler Bots Using AWS WAF
AWS Architecture BlogField Notes: How to Identify and Block Fake Crawler Bots Using AWS WAFby Vijay Menon and Gaurav Sahi | on04 NOV 2020| inArchitecture,AWS Lambda,AWS WAF,Technical How-to|Permalink|ShareIn this blog post, we focus on how to identify fake bots using these AWS services:AWS WAF,Amazon Kinesis Data Firehose,Amazon S3andAWS Lambda. We use fake Google/Bing bots to demonstrate, but the principles can be applied to other popular crawlers like Slurp Bot from Yahoo, DuckDuckBot from DuckDuckGo, Alexa crawler from Alexa internet ranking service.For industries like media, online retailors, news or social websites, content is critical and often sets them apart from other competitors. These companies put in a significant amount of effort to make the content as visible and accessible as possible. To do that these companies rely on crawler bots, so that legitimate users searching for content can find the content easily. Crawler bots are useful for indexing the site pages and helping make the content more searchable and improve rankings.However, this capability can be misused. So it is important to distinguish between genuine crawler bots and fake ones that are doing more than just indexing your site. It’s important to properly identify good and bad actors so that you can stop the bad ones without impacting the ability of good ones, and at scale. This helps in driving more traffic, visitors, and more revenue from your websites.Identifying botsThere are two primary sources of information required to identify a fake bot:HTTP Header User-Agent:Fake bots try to present themselves as real bots, for example as Google or Bing, by using the same user agent string used by Google or Bing.IP Address:You can look at the source IP address of the incoming request and determine if it belongs to the search engine provider network like Google or Bing. You can do this by performing a forward and reverse look up and comparing the results. These methods are well documented by the search engine providersGoogleandBing.Solution OverviewThe solution leverages the capabilities ofAWS WAF. Our demonstration application is a static website hosted onAmazon S3fronted byAmazon CloudFront. This means that we can provide permission of access to the S3 bucket only to CloudFront using origin access identity.The logs are streamed in near real time usingAmazon Kinesis Data Firehose, inspected using a Lambda function to help identify fake bots before storing the logs on Amazon S3. The Lambda function does two things:Inspect the traffic using the rules to identify bad or fake bots. In this case it uses forward and reverse DNS lookup results of the Client IP address of packets with User-Agent string resembling GoogleBot or BingBot.Once identified as a fake bot, the Lambda function updates AWS WAF IP-Set to permanently block the requests coming from IP addresses of fake bots.Note:For the sake of this demonstration, we are using a static website hosted on Amazon S3 with CloudFront. This requires the AWS WAF and IP-Set used by AWS WAF to be of scope ‘CLOUDFRONT’. You can modify it to use scope ‘REGIONAL’ if you chose to protect your web properties behind Application Load Balancer or API Gateway.Solution ArchitecturePrerequisitesReaders of this blog post should be familiar withHTTPand the following AWS services:AWS CloudFormationAWS LambdaAWS WAFAmazon Kinesis Data FirehoseAmazon S3For this walkthrough, you should have the following:AWS AccountAWS Command Line Interface (AWS CLI): You needAWS CLI installedandconfiguredon the workstation from where you are going to try the steps mentioned below.Credentials configured in AWS CLI should have the required IAM permissions to spin up and modify the resources mentioned in this post.Make sure that you deploy the solution to us-east-1 Region and your AWS CLI default Region is us-east-1. If us-east-1 is not the default Region, reference the Regions explicitly while executing AWS CLI commands using--region us-east-1switch.Amazon S3 bucket in us-east-1 regionWalkthrough1.     Create an Amazon S3 static Website and put it behind an Amazon CloudFront distribution. You can follow thesesteps. Note down the CloudFront distribution ID. We will use it in subsequent steps.2.     Download and unzip the file containing CloudFormation template and lambda function code fromhereto a folder in your local workstation. You will need to run all of the subsequent commands from this folder.3.     Zip the lambda functionlambda_function.pyand upload it to an Amazon S3 bucket of your choice inus-east-1. Note the bucket name as it will be used in the subsequent steps. The blog useswaf-logs-fake-bots-us-east-1for reference as the S3 bucket name.$ zip lambda_function.py lambda_function.py.zip$ aws s3 cp lambda_function.py.zip s3://waf-logs-fake-bots-us-east-1/4.     Create the resources required for this blog post by deploying the AWS CloudFormation template and running the below command:aws cloudformation create-stack \
--stack-name FakeBotBlockBlog \
--template-body file://BotBlog.yml \
--parameters ParameterKey=KinesisBufferIntervalSeconds,ParameterValue=900 ParameterKey=KinesisBufferSizeMB,ParameterValue=3 ParameterKey=IPSetName,ParameterValue=BlockFakeBotIPSet ParameterKey=IPSetScope,ParameterValue=CLOUDFRONT ParameterKey=S3BucketWithDeploymentPackage,ParameterValue=waf-logs-fake-bots-us-east-1 ParameterKey=DeploymentPackageZippedFilename,ParameterValue=lambda_function.py.zip \
--capabilities CAPABILITY_IAM \
--region us-east-1You need to provide the following information, and you can change the parameters based on your specific needs:a.     KinesisBufferIntervalSeconds and KinesisBufferSizeMB. These will define the interval at which Kinesis Firehose ships the logs to Amazon S3, the Default is 900 seconds and 3MB respectively, whichever is met first.b.     IPSetName. Name of the IP Set which will be used to record the client IP address of fake bots. Default value is BlockFakeBotIPSet.c.     IPSetScope. Scope of the IP Set. I am using CLOUDFRONT and associate it with the CloudFront distribution created in step 1. You can choose to make it REGIONAL in which case WebACLassociation will need to be with an ALB or an API Gateway.d.     S3BucketWithDeploymentPackage. Name of S3 bucket used in step 3. The blog assumeswaf-logs-fake-bots-us-east-1.e.     DeploymentPackageZipedFilename. Lambda function filename without the file extension. For example, the blog assumeslambda_function.py.zipis available on the Amazon S3 bucket and uses this value for this parameter.Some stack templates might include resources that can affect permissions in your AWS account, for example, by creating new AWS Identity and Access Management (IAM) role. For those stacks, you must explicitly acknowledge this by specifying CAPABILITY_IAM or CAPABILITY_NAMED_IAM value for the –capabilities parameter.Stack creation will take you approximately 5-7 minutes. Check the status of the stack by executing the below command every few minutes. You should seeStackStatusvalue asCREATE_COMPLETE.Example:aws cloudformation describe-stacks --stack-name FakeBotBlockBlog | grep StackStatusThe CloudFormation template will create the following resources:IP Set for AWS WAFWebACL with rules to block the client IP addresses of fake bots, and anAWS-managed common rule set.Lambda function to help detect fake bots and modify the AWS WAF IP Set to block themKinesis Firehose delivery stream, which will use the above Lambda function for processingIAM roles with required permissions for the Lambda function and Kinesis FirehoseS3 bucket for AWS WAF logs5.     Enable logging for the WebACL using AWS CLI. For this you need the ARN of the WebACL and Kinesis Firehose. You can find that information from the output of the CloudFormation stack created in step 4 using the below AWS CLI commandaws cloudformation describe-stacks --stack-name FakeBotBlockBlogPlease note the 2 ARNs and run the following commands by replacing (1) ResourceArn value with WebACL ARN and (2) LogDestinationConfigs value with Kinesis Firehose delivery stream ARN.Example:aws wafv2 put-logging-configuration –-logging-configuration ResourceArn=arn:aws:wafv2:us-east-1:123456789012:global/webacl/FakeBotWebACL/259ea98f-24ba-4acd-8803-3e7d02e8d482,LogDestinationConfigs=arn:aws:firehose:us-east-1:123456789012:deliverystream/aws-waf-logs-FakeBotBlockBlog --region us-east-16.     Associate CloudFront distribution with this WebACL: Sign in to the AWS Management Console and open the AWS WAF and Shield console at https://console.aws.amazon.com/wafv2/homev2/web-acls?region=global .Click on theWebACLcreated earlier in this procedureNavigate to ‘Associated AWS resources’ tab and selectAdd AWS resourcesIn the subsequent screen, select the CloudFront distribution created in step 1SelectAddNote: If you are using an ALB or API Gateway for your web property, then you need to use REGIONAL WebACL and IP Set.  Review theprocedureto associate an ALB or API Gateway to the WebACL.You can monitor WebACL performance from the Overview section of WebACL from theAWS WAF and Shield console.TestingTo test, you will need to generate some traffic which will trigger the lambda function to detect and block the fake bots created earlier in this blog. The web traffic can be generated from the local machine or from an EC2 instance with access to the internet using curl. Manually set the user agent to resemble Googlebot by running the following command from shell:Replacehttp://www.awsdemodesign.com/with the URL of your CloudFront distribution you created in step 1 of the walkthrough.for i in {1..1000}; do curl -I -A "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" http://www.awsdemodesign.com/; doneInitially you will see HTTP1.1/ 200 OK response. This will trigger Lambda and modify the IP Set to include your public IP address to be blocked. You can verify that by inspecting the IP set from AWS WAF and Shield console.Sign in to the AWS Management Console and open theAWS WAF and Shield consoleClick on IP Set created earlier in this blog. In the subsequent screen you can see your public IP address in the list of IP addresses.If you run the curl command again, you will see that the response now is HTTP/1.1 403 Forbidden.Clean UpDisassociate the CloudFront Distribution from WebACLDelete the S3 bucket and CloudFront Distribution created in Step 1Empty and delete the S3 bucket created by the CloudFormation stack for AWS WAF logs.Delete CloudFormation stack created in Step 4.ConclusionIn this blog post, we demonstrated how you can set up and inspect incoming web traffic using AWS Lambda, AWS WAF native logging capabilities, and Kinesis Firehose to help detect and block bad or fake bots at scale. Furthermore, the solution outlined in this post provides a framework which can be extended to identify similar unwanted traffic impersonating as other good bots.Field Notes provides hands-on technical guidance from AWS Solutions Architects, consultants, and technical account managers, based on their experiences in the field solving real-world business problems for customers.TAGS:Field NotesVijay MenonVijay Menon is a Principal Solutions Architect based out of Bangalore, India with a background in large scale networks and communications infrastructure. He enjoys learning new technologies and helping customers solve complex technical problems by providing solutions using AWS products and services. When he is not helping customers, he likes to go on long runs and spend time with family and friends.Gaurav SahiGaurav Sahi is a Principal Solutions Architect based out of Bangalore, India and has rich experience across embedded, telecom, broadcast & digital video and cloud technologies. In addition to helping customers in their transformation journey to cloud, his current passion is to explore and learn AI/ML services. He likes to travel, enjoy local food and spend time with family and friends in his free time.
In this blog post, we focus on how to identify fake bots using these AWS services: AWS WAF, Amazon Kinesis Data Firehose, Amazon S3 and AWS Lambda. We use fake Google/Bing bots to demonstrate, but the principles can be applied to other popular crawlers like Slurp Bot from Yahoo, DuckDuckBot from DuckDuckGo, Alexa crawler from […]
Vijay Menon
Architecture AWS Lambda AWS WAF Technical How-to