It recently came to my attention (through an outside company) that a host on my internal LAN may be conducting web scraping activity against that outside company's web servers. Due to this activity, the outside company has blocked our ability to access any web server they host.

Anyone have any recommendations for any tools or methods to track down any endpoints on my own internal network that are conducting web scraping against external web sites? We recently purchased a SIEM solution, however it's not yet online, so my only other option is to manually review netflows for excessive http(s) against this company's hosts. Needless to say I'd much rather find a more expeditious option.

I have ~1200 endpoints on my internal LAN.


If you're not above getting your hands dirty with a bit of scripting, you could make a simple Python script using the Scapy library. This script would watch for HTTP traffic to the external company's site and log the IP.

Upon review of the capture file, the most recurring IP address should be the culprit of the scraping.

If this external company's site is accessed frequently by users on your LAN, then you may want to watch for requests that result in a 404 error response from the outside company's site, as this would be a sign of the scraping tool attempting to access URIs that don't exist during the scraping process.

  • This is a fantastic idea, however because the destination I am searching for is basically "any Akamai-hosted website" I might not be able to narrow the script parameters far enough. What I was mostly hoping for was some way to detect on criteria such as number of hits to the same webhost, within a certain time frame, over a sustained period, with X amount of bandwidth consumed, etc. Some kind of behavioral signature that would likely indicate scraping. I'm going to look at Scapy though and see what I can do with it. Thanks! – Zane Dec 2 '16 at 16:00

Check your perimeter firewall for egress traffic logs. If you don't have a perimeter firewall, get a perimeter firewall.

  • Did this - however because the egress traffic I am searching for is "internal node --> external host, http or https" there is, to put it mildly, a superfluous amount of irrelevant matching entries. That basically matches about 85% of the entries and because Akamai (the third party who has issued the restriction) hosts so many different websites I am not really able to filter much further. – Zane Dec 2 '16 at 15:55

Your Answer

