Anyone ever build a scraper for CRE Data sites? (Reonomy, Costar, ProspectNow, etc) - I paid a guy on Fiverr and the scraper is getting blocked…
Anyone here get around this?
[{"id": "ixjkcse", "text": "What is the purpose?", "replies": [], "depth": 1}, {"id": "ixjl39r", "text": "Costar is notorious for suing anyone out of existence if they touch CRE data, claiming some copyright / \"we know it's our data\" argument.\n\nI wrote something to work as a chrome extension, so it helps you save and analyze property data as you naturally use Loopnet/Costar.\n\nWhat type of data are you trying to get, and is there a goal for what you want to do with it?", "replies": [{"id": "ixju033", "text": "Goal is to have the data and market to it for off market deals via sms, Cold call, mail.\nI\u2019m aiming for industrial, office, retail in diff states.", "replies": [{"id": "ixjw0o5", "text": "They are in the business of selling information, including what you are taking.", "replies": [{"id": "ixk0elp", "text": "And I am in the buisness to get data without having to pay an arm and a leg!", "replies": [{"id": "ixkcbhh", "text": "Actually, you\u2019re not in the business of anything. That\u2019s the point. If CoStar has made a business out of data, you don\u2019t think they\u2019ll sue you for scraping their data and repackaging it for commercial uses? You don\u2019t think they\u2019ve thought through the possibility of a bad faith subscriber who just republishes data? \n\nI\u2019m not even a CoStar bootlicker. But there\u2019s a reason they actually are a business. They have a moat around their IP and they\u2019re not gonna let some bumfuck steal their money rofl.", "replies": [{"id": "ixkdesn", "text": "\ud83d\ude02", "replies": [{"id": "ixkfhkt", "text": "You can laugh but he\u2019s right this doesn\u2019t end well for you most likely", "replies": [], "depth": 7}, {"id": "ixlpx4w", "text": "Look up costar vs xceligent", "replies": [], "depth": 7}], "depth": 6}], "depth": 5}, {"id": "ixm8seq", "text": "Hope you also found good lawyers on your hiring spree. You'll need a team to deal with the CoStar attorneys.", "replies": [], "depth": 5}], "depth": 4}], "depth": 3}, {"id": "ixlx3se", "text": "Building that proprietary database takes a lot of work and constant updates.\n\nCostar, loopnet and other databases are mediocre with their data and contact information. REIS and TREPP might be other options. You will find that a modest amount of the contacts listed either don't have the right number, name, or address for the decision makers owning the building. You likely will get flagged for scraping data. I was locked out of loopnet by simply doing many manual searches in a day.\n\nThe only solution is to cold call, and do internet searches to find the true owners using those services as a starting point. Looking at state LLC records also helps. \n\nSource: worked for a broker team that maintained and expanded a list of about 3,000 contacts with a policy contacting that list several times a year for various deals being sold and to try and get listings.", "replies": [], "depth": 3}], "depth": 2}], "depth": 1}, {"id": "ixk540n", "text": "Join a brokerage...", "replies": [], "depth": 1}, {"id": "ixkhnf8", "text": "Use a vpn", "replies": [], "depth": 1}, {"id": "ixl1l5d", "text": "All of the sites have gone to extravagant lengths to prevent this.", "replies": [], "depth": 1}, {"id": "ixl2a4d", "text": "Stealing is a crime.", "replies": [], "depth": 1}, {"id": "ixlxhd8", "text": "Scraping in 2022 is very hard. It's trivial to install e.g., CloudFlare and block bots like yours. Definitely not something you're going to buy off Fiverr. \ud83d\ude02 A Fiverr script can scrape Wikipedia very effectively but that's about all. \n\nMy favorite approach is the one yellowpages.com used to take. They'd let you scrape but they'd start mixing in random listings to screw up your dataset. Pretty brilliant. That was almost 20 years ago though, they probably just block you entirely now.\n\nI know because I run a huge directory site that gets hit by bots all day long (usually disguised as some search engine you've never heard of). They're such a drain on resources, even though we block aggressively. It's a huge nuisance.\n\nOn the off chance they allow scraping via their robots.txt file (lol) then go for it. Otherwise, go jump in a lake.\n\nSite owners: just block all IPs from Ashburn, VA, China, and Russia and that will cut out a huge chunk of these scrapers.", "replies": [], "depth": 1}, {"id": "ixm2dpr", "text": "I make these for my own personal use. I have multiple raspberry pis running. Each randomly cycles through US based VPN after each http request (probably a little excessive) Also to be nice to their servers I intentionally slow down my requests. I still occasionally get blocked. \n\nEdit: I wouldn't mess with costar and friends. They seem to be the most active at blocking and their developers are far smarter than me.", "replies": [], "depth": 1}, {"id": "ixm5t3v", "text": "VPN should work but not in costar obviously", "replies": [], "depth": 1}, {"id": "ixm6u4z", "text": "I think you can use Mozenda to scrape these sites except Costar", "replies": [], "depth": 1}, {"id": "ixmirts", "text": "Costar is incredibly litigious.  They have sued people who uploaded their own photos then downloaded them and used the costar versions in their marketing materials", "replies": [{"id": "ixvxs9n", "text": "People should STOP using Costar for the vey same reasons.", "replies": [{"id": "ixx1yp9", "text": "I don\u2019t disagree but where else are you going to get the data?", "replies": [], "depth": 3}], "depth": 2}], "depth": 1}]