{"action":"created","issue":{"url":"https://api.github.com/repos/e107inc/e107/issues/3358","repository_url":"https://api.github.com/repos/e107inc/e107","labels_url":"https://api.github.com/repos/e107inc/e107/issues/3358/labels{/name}","comments_url":"https://api.github.com/repos/e107inc/e107/issues/3358/comments","events_url":"https://api.github.com/repos/e107inc/e107/issues/3358/events","html_url":"https://github.com/e107inc/e107/issues/3358","id":348607703,"node_id":"MDU6SXNzdWUzNDg2MDc3MDM=","number":3358,"title":"question - robots.txt","user":{"login":"Jimmi08","id":5429548,"node_id":"MDQ6VXNlcjU0Mjk1NDg=","avatar_url":"https://avatars2.githubusercontent.com/u/5429548?v=4","gravatar_id":"","url":"https://api.github.com/users/Jimmi08","html_url":"https://github.com/Jimmi08","followers_url":"https://api.github.com/users/Jimmi08/followers","following_url":"https://api.github.com/users/Jimmi08/following{/other_user}","gists_url":"https://api.github.com/users/Jimmi08/gists{/gist_id}","starred_url":"https://api.github.com/users/Jimmi08/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Jimmi08/subscriptions","organizations_url":"https://api.github.com/users/Jimmi08/orgs","repos_url":"https://api.github.com/users/Jimmi08/repos","events_url":"https://api.github.com/users/Jimmi08/events{/privacy}","received_events_url":"https://api.github.com/users/Jimmi08/received_events","type":"User","site_admin":false},"labels":[{"id":19517348,"node_id":"MDU6TGFiZWwxOTUxNzM0OA==","url":"https://api.github.com/repos/e107inc/e107/labels/enhancement","name":"enhancement","color":"84b6eb","default":true}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2018-08-08T07:31:50Z","updated_at":"2018-08-08T10:01:08Z","closed_at":null,"author_association":"CONTRIBUTOR","body":"I got this error from crawl statistics\r\n\r\n![image](https://user-images.githubusercontent.com/5429548/43822841-9f41e63e-9aed-11e8-9666-a1a85f8a3028.png)\r\n\r\nThose urls are direct download links...\r\n\r\nCould be this added to default robots.txt\r\n\r\nSomething like this but I am not sure because it's not real folder or something.\r\n\r\n`Disallow: /download/get/`"},"comment":{"url":"https://api.github.com/repos/e107inc/e107/issues/comments/411353936","html_url":"https://github.com/e107inc/e107/issues/3358#issuecomment-411353936","issue_url":"https://api.github.com/repos/e107inc/e107/issues/3358","id":411353936,"node_id":"MDEyOklzc3VlQ29tbWVudDQxMTM1MzkzNg==","user":{"login":"Jimmi08","id":5429548,"node_id":"MDQ6VXNlcjU0Mjk1NDg=","avatar_url":"https://avatars2.githubusercontent.com/u/5429548?v=4","gravatar_id":"","url":"https://api.github.com/users/Jimmi08","html_url":"https://github.com/Jimmi08","followers_url":"https://api.github.com/users/Jimmi08/followers","following_url":"https://api.github.com/users/Jimmi08/following{/other_user}","gists_url":"https://api.github.com/users/Jimmi08/gists{/gist_id}","starred_url":"https://api.github.com/users/Jimmi08/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Jimmi08/subscriptions","organizations_url":"https://api.github.com/users/Jimmi08/orgs","repos_url":"https://api.github.com/users/Jimmi08/repos","events_url":"https://api.github.com/users/Jimmi08/events{/privacy}","received_events_url":"https://api.github.com/users/Jimmi08/received_events","type":"User","site_admin":false},"created_at":"2018-08-08T10:01:08Z","updated_at":"2018-08-08T10:01:08Z","author_association":"CONTRIBUTOR","body":"e107.robots.txt looks like this:\r\n\r\n```\r\nUser-agent: *\r\nDisallow: /sitedown.php\r\nDisallow: /cron.php\r\nDisallow: /fpw.php\r\nDisallow: /membersonly.php\r\nDisallow: /e107_admin/\r\nDisallow: /e107_core/\r\nDisallow: /e107_docs/\r\nDisallow: /e107_files/cache/\r\nDisallow: /e107_files/public/avatars/\r\nDisallow: /e107_handlers/\r\nDisallow: /e107_languages/\r\nDisallow: /e107_media/\r\nDisallow: /e107_system/\r\nDisallow: /e107_themes/\r\nDisallow: /e107_web/\r\n```\r\n\r\nprobably e107_images should be added too: \r\n\r\n![image](https://user-images.githubusercontent.com/5429548/43830811-873c195a-9b02-11e8-8944-7c660970a0ab.png)\r\n"}}
{"actor":{"display_login":"Jimmi08"}}