{"action":"created","issue":{"url":"https://api.github.com/repos/ropensci/robotstxt/issues/27","repository_url":"https://api.github.com/repos/ropensci/robotstxt","labels_url":"https://api.github.com/repos/ropensci/robotstxt/issues/27/labels{/name}","comments_url":"https://api.github.com/repos/ropensci/robotstxt/issues/27/comments","events_url":"https://api.github.com/repos/ropensci/robotstxt/issues/27/events","html_url":"https://github.com/ropensci/robotstxt/issues/27","id":345009510,"node_id":"MDU6SXNzdWUzNDUwMDk1MTA=","number":27,"title":"Second level domains and HTTP redirects","user":{"login":"dmi3kno","id":13419011,"node_id":"MDQ6VXNlcjEzNDE5MDEx","avatar_url":"https://avatars1.githubusercontent.com/u/13419011?v=4","gravatar_id":"","url":"https://api.github.com/users/dmi3kno","html_url":"https://github.com/dmi3kno","followers_url":"https://api.github.com/users/dmi3kno/followers","following_url":"https://api.github.com/users/dmi3kno/following{/other_user}","gists_url":"https://api.github.com/users/dmi3kno/gists{/gist_id}","starred_url":"https://api.github.com/users/dmi3kno/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dmi3kno/subscriptions","organizations_url":"https://api.github.com/users/dmi3kno/orgs","repos_url":"https://api.github.com/users/dmi3kno/repos","events_url":"https://api.github.com/users/dmi3kno/events{/privacy}","received_events_url":"https://api.github.com/users/dmi3kno/received_events","type":"User","site_admin":false},"labels":[{"id":293929370,"node_id":"MDU6TGFiZWwyOTM5MjkzNzA=","url":"https://api.github.com/repos/ropensci/robotstxt/labels/question","name":"question","color":"cc317c","default":true}],"state":"open","locked":false,"assignee":{"login":"petermeissner","id":939139,"node_id":"MDQ6VXNlcjkzOTEzOQ==","avatar_url":"https://avatars0.githubusercontent.com/u/939139?v=4","gravatar_id":"","url":"https://api.github.com/users/petermeissner","html_url":"https://github.com/petermeissner","followers_url":"https://api.github.com/users/petermeissner/followers","following_url":"https://api.github.com/users/petermeissner/following{/other_user}","gists_url":"https://api.github.com/users/petermeissner/gists{/gist_id}","starred_url":"https://api.github.com/users/petermeissner/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/petermeissner/subscriptions","organizations_url":"https://api.github.com/users/petermeissner/orgs","repos_url":"https://api.github.com/users/petermeissner/repos","events_url":"https://api.github.com/users/petermeissner/events{/privacy}","received_events_url":"https://api.github.com/users/petermeissner/received_events","type":"User","site_admin":false},"assignees":[{"login":"petermeissner","id":939139,"node_id":"MDQ6VXNlcjkzOTEzOQ==","avatar_url":"https://avatars0.githubusercontent.com/u/939139?v=4","gravatar_id":"","url":"https://api.github.com/users/petermeissner","html_url":"https://github.com/petermeissner","followers_url":"https://api.github.com/users/petermeissner/followers","following_url":"https://api.github.com/users/petermeissner/following{/other_user}","gists_url":"https://api.github.com/users/petermeissner/gists{/gist_id}","starred_url":"https://api.github.com/users/petermeissner/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/petermeissner/subscriptions","organizations_url":"https://api.github.com/users/petermeissner/orgs","repos_url":"https://api.github.com/users/petermeissner/repos","events_url":"https://api.github.com/users/petermeissner/events{/privacy}","received_events_url":"https://api.github.com/users/petermeissner/received_events","type":"User","site_admin":false}],"milestone":null,"comments":6,"created_at":"2018-07-26T21:19:29Z","updated_at":"2018-09-04T15:15:16Z","closed_at":null,"author_association":"NONE","body":"I wonder how robotstxt deals with redirects\r\n```r\r\n# downloads html, signals a warning\r\nrt <- robotstxt::get_robotstxt(\"github.io\")\r\n#> <!DOCTYPE html>\r\n#> <html lang=\"en\">\r\n#> [...]\r\n#> Warning message:\r\n#> In robotstxt::get_robotstxt(\"github.io\") :\r\n#>   get_robotstxt(): github.io; Not valid robots.txt.\r\n\r\nrobotstxt::is_valid_robotstxt(rt)\r\n#> [1] TRUE\r\n\r\n# what's going on? \r\nhttr::GET(\"github.io\")\r\n#> Response [https://pages.github.com/]\r\n#>  Date: 2018-07-26 20:54\r\n#>  ...\r\n\r\n# this one is also valid\r\nrt <- robotstxt::get_robotstxt(\"pages.github.com\")\r\nrobotstxt::is_valid_robotstxt(rt)\r\n#> [1] TRUE\r\n\r\n# so what did we get?\r\nprint(rt)\r\n#> Sitemap: https://pages.github.com/sitemap.xml\r\n```\r\nCouple of questions:\r\n- How do we handle redirects? A simple GET would indicate that we're looking in the wrong place. Should we warn the user and get the redirected robots.txt instead? Anyway better than rendering html header...\r\n- What happens if robots.txt is not delegated to second-level domain? Clearly github.com has its very proper robots.txt, but none of the second-level domains has its own properly populated file (checked `pages.github.com`, `education.github.com`, `blog.github.com`, `raw.github.com`, with an exception of `gist.github.com`). In my view, file is not valid if there isn't a single rule in it. It is not necessarily corrupt - it just doesn't exist. Shouldn't we, again, warn user and return robots.txt from first-level domain instead?\r\n\r\nIn fact raw.github.com is explicitly disallowed,\r\n```r\r\nrobotstxt::paths_allowed(paths=\"raw\", domain=\"github.com\", bot=\"*\")\r\n#> [1] FALSE\r\n```\r\nbut it is written in slightly unusual way: there's no `github.com/raw` so the rule in main robots.txt actually refers to second level domain.\r\n```r\r\nhttr::GET(\"https://github.com/raw\")\r\n#> Response [https://github.com/raw]\r\n#>   Date: 2018-07-26 21:43\r\n#>   Status: 404\r\n#> [...]\r\n```\r\n"},"comment":{"url":"https://api.github.com/repos/ropensci/robotstxt/issues/comments/418406544","html_url":"https://github.com/ropensci/robotstxt/issues/27#issuecomment-418406544","issue_url":"https://api.github.com/repos/ropensci/robotstxt/issues/27","id":418406544,"node_id":"MDEyOklzc3VlQ29tbWVudDQxODQwNjU0NA==","user":{"login":"dmi3kno","id":13419011,"node_id":"MDQ6VXNlcjEzNDE5MDEx","avatar_url":"https://avatars1.githubusercontent.com/u/13419011?v=4","gravatar_id":"","url":"https://api.github.com/users/dmi3kno","html_url":"https://github.com/dmi3kno","followers_url":"https://api.github.com/users/dmi3kno/followers","following_url":"https://api.github.com/users/dmi3kno/following{/other_user}","gists_url":"https://api.github.com/users/dmi3kno/gists{/gist_id}","starred_url":"https://api.github.com/users/dmi3kno/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dmi3kno/subscriptions","organizations_url":"https://api.github.com/users/dmi3kno/orgs","repos_url":"https://api.github.com/users/dmi3kno/repos","events_url":"https://api.github.com/users/dmi3kno/events{/privacy}","received_events_url":"https://api.github.com/users/dmi3kno/received_events","type":"User","site_admin":false},"created_at":"2018-09-04T15:15:16Z","updated_at":"2018-09-04T15:15:16Z","author_association":"NONE","body":"**Question 3.** Very little is required for robots.txt to be considered \"valid\". This, however, does not mean the file is serving its purpose. Most of the subdomain robot.txt files on github.com are empty (contain only sitemap reference. education.github.com/robots.txt is just a placeholder that was never activated (rules are defined but commented away).\r\n\r\nI think there's a need to distinguish the following possible outcomes:\r\n1) robots.txt is missing - returning 404 or [otherwise ](https://raw.githubusercontent.com/robots.txt)\r\n2) robots.txt is present but not \"full-fledged\" - like we saw in [pages](https://pages.github.com/robots.txt). The criteria here is that we want at least one Allow or Disallow statement.\r\n3) robots.txt is present and well defined - this allows unambiguous resolution of path.\r\n\r\nThoughts?"}}
{"actor":{"display_login":"dmi3kno"}}