{"action":"created","issue":{"url":"https://api.github.com/repos/commoncrawl/webarchive-indexing/issues/3","repository_url":"https://api.github.com/repos/commoncrawl/webarchive-indexing","labels_url":"https://api.github.com/repos/commoncrawl/webarchive-indexing/issues/3/labels{/name}","comments_url":"https://api.github.com/repos/commoncrawl/webarchive-indexing/issues/3/comments","events_url":"https://api.github.com/repos/commoncrawl/webarchive-indexing/issues/3/events","html_url":"https://github.com/commoncrawl/webarchive-indexing/issues/3","id":187061621,"number":3,"title":"Index non-200 responses and robots.txt","user":{"login":"sebastian-nagel","id":1630582,"avatar_url":"https://avatars.githubusercontent.com/u/1630582?v=3","gravatar_id":"","url":"https://api.github.com/users/sebastian-nagel","html_url":"https://github.com/sebastian-nagel","followers_url":"https://api.github.com/users/sebastian-nagel/followers","following_url":"https://api.github.com/users/sebastian-nagel/following{/other_user}","gists_url":"https://api.github.com/users/sebastian-nagel/gists{/gist_id}","starred_url":"https://api.github.com/users/sebastian-nagel/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/sebastian-nagel/subscriptions","organizations_url":"https://api.github.com/users/sebastian-nagel/orgs","repos_url":"https://api.github.com/users/sebastian-nagel/repos","events_url":"https://api.github.com/users/sebastian-nagel/events{/privacy}","received_events_url":"https://api.github.com/users/sebastian-nagel/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":1,"created_at":"2016-11-03T13:37:59Z","updated_at":"2016-12-15T14:32:40Z","closed_at":"2016-12-15T14:32:40Z","body":"Since August we have datasets with non-200 responses (redirects, 404s, etc.) and robots.txt files [[1](http://commoncrawl.org/2016/09/robotstxt-and-404-redirect-data-sets/)]. They should be added to the [Common Crawl index](http://index.commoncrawl.org/), cf. [[2]](https://groups.google.com/d/msg/common-crawl/19Xs1vMzNHk/EFdWYrL8BAAJ). Trivial to do: add these WARC files to the index input but make sure that paths/filenames do not collide."},"comment":{"url":"https://api.github.com/repos/commoncrawl/webarchive-indexing/issues/comments/267341373","html_url":"https://github.com/commoncrawl/webarchive-indexing/issues/3#issuecomment-267341373","issue_url":"https://api.github.com/repos/commoncrawl/webarchive-indexing/issues/3","id":267341373,"user":{"login":"sebastian-nagel","id":1630582,"avatar_url":"https://avatars.githubusercontent.com/u/1630582?v=3","gravatar_id":"","url":"https://api.github.com/users/sebastian-nagel","html_url":"https://github.com/sebastian-nagel","followers_url":"https://api.github.com/users/sebastian-nagel/followers","following_url":"https://api.github.com/users/sebastian-nagel/following{/other_user}","gists_url":"https://api.github.com/users/sebastian-nagel/gists{/gist_id}","starred_url":"https://api.github.com/users/sebastian-nagel/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/sebastian-nagel/subscriptions","organizations_url":"https://api.github.com/users/sebastian-nagel/orgs","repos_url":"https://api.github.com/users/sebastian-nagel/repos","events_url":"https://api.github.com/users/sebastian-nagel/events{/privacy}","received_events_url":"https://api.github.com/users/sebastian-nagel/received_events","type":"User","site_admin":false},"created_at":"2016-12-15T14:32:40Z","updated_at":"2016-12-15T14:32:40Z","body":"Fixed with [CC-MAIN-2016-50](http://index.commoncrawl.org/CC-MAIN-2016-50):\r\n\r\nCreate list of all WARC files (including robots.txt and non-200 responses) to step 1 (indexwarcsjob.py):\r\n```\r\ncat {warc,robotstxt,non200responses}.paths | shuf >input.paths\r\n```\r\n(list is shuffled so that smaller robots.txt and non-200 WARC files are mixed with large main WARC files)"}}
{"actor":{"display_login":"sebastian-nagel"}}