title: How to do Google Search Programatically respecting Robots.txt file
body: <p>I want to search for term in Google and get 100 results using .Net.  We can make a request to <a href="http://www.google.com/search?q=" rel="nofollow noreferrer">http://www.google.com/search?q=</a>{searchterm}&amp;num=100.  But as per Terms Of Service of Google we should not automate the process.  I verified Robots.txt file, we should not send any request to <a href="http://www.google.com/search" rel="nofollow noreferrer">http://www.google.com/search</a>  We have a Google Free Custom Search API. It allows only 100 requests per day.  If we want more request then we need to pay. After paying we can't search for Entire web.  Free Google Custom Search Results are not matching, if we try to search for same word in Google  Please let me know How to get top 100 results for search term using Google legally.  Can we do auomated processing? Will it block our IP?</p>