{"action":"closed","issue":{"url":"https://api.github.com/repos/documentcloud/documentcloud/issues/350","repository_url":"https://api.github.com/repos/documentcloud/documentcloud","labels_url":"https://api.github.com/repos/documentcloud/documentcloud/issues/350/labels{/name}","comments_url":"https://api.github.com/repos/documentcloud/documentcloud/issues/350/comments","events_url":"https://api.github.com/repos/documentcloud/documentcloud/issues/350/events","html_url":"https://github.com/documentcloud/documentcloud/issues/350","id":137327308,"number":350,"title":"Template URLs in resource JSON get crawled, generate lots of 404s","user":{"login":"reefdog","id":4731,"avatar_url":"https://avatars.githubusercontent.com/u/4731?v=3","gravatar_id":"","url":"https://api.github.com/users/reefdog","html_url":"https://github.com/reefdog","followers_url":"https://api.github.com/users/reefdog/followers","following_url":"https://api.github.com/users/reefdog/following{/other_user}","gists_url":"https://api.github.com/users/reefdog/gists{/gist_id}","starred_url":"https://api.github.com/users/reefdog/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/reefdog/subscriptions","organizations_url":"https://api.github.com/users/reefdog/orgs","repos_url":"https://api.github.com/users/reefdog/repos","events_url":"https://api.github.com/users/reefdog/events{/privacy}","received_events_url":"https://api.github.com/users/reefdog/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/documentcloud/documentcloud/labels/discussion","name":"discussion","color":"eb6420"},{"url":"https://api.github.com/repos/documentcloud/documentcloud/labels/platform","name":"platform","color":"fad8c7"}],"state":"closed","locked":false,"assignee":null,"milestone":null,"comments":1,"created_at":"2016-02-29T18:06:54Z","updated_at":"2016-02-29T21:09:14Z","closed_at":"2016-02-29T21:09:14Z","body":"Crawlers (at least, Google Bot) are aware of our [resource JSON bodies](https://www.documentcloud.org/documents/282753-lefler-thesis.json) and follow all the URLs they find. This is a problem for our template URLs like `https://www.documentcloud.org/documents/282753/pages/lefler-thesis-p{page}.txt`, which the bot tries to follow, generating lots of 404 noise.\r\n\r\nThoughts on trying to block template URLs from being indexed with [robots.txt pattern matching](http://stackoverflow.com/questions/6859399/block-google-robots-for-urls-containing-a-certain-word)?\r\n\r\nReference: http://searchengineland.com/a-deeper-look-at-robotstxt-17573"}}