{"action":"created","issue":{"url":"https://api.github.com/repos/downthemall/downthemall/issues/83","repository_url":"https://api.github.com/repos/downthemall/downthemall","labels_url":"https://api.github.com/repos/downthemall/downthemall/issues/83/labels{/name}","comments_url":"https://api.github.com/repos/downthemall/downthemall/issues/83/comments","events_url":"https://api.github.com/repos/downthemall/downthemall/issues/83/events","html_url":"https://github.com/downthemall/downthemall/issues/83","id":176438952,"number":83,"title":"Metalink HTTP headers trip CloudFlare scraping, resulting in HTTP 403 responses","user":{"login":"gima","id":1252061,"avatar_url":"https://avatars.githubusercontent.com/u/1252061?v=3","gravatar_id":"","url":"https://api.github.com/users/gima","html_url":"https://github.com/gima","followers_url":"https://api.github.com/users/gima/followers","following_url":"https://api.github.com/users/gima/following{/other_user}","gists_url":"https://api.github.com/users/gima/gists{/gist_id}","starred_url":"https://api.github.com/users/gima/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/gima/subscriptions","organizations_url":"https://api.github.com/users/gima/orgs","repos_url":"https://api.github.com/users/gima/repos","events_url":"https://api.github.com/users/gima/events{/privacy}","received_events_url":"https://api.github.com/users/gima/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/downthemall/downthemall/labels/enhancement","name":"enhancement","color":"1d76db"}],"state":"open","locked":false,"assignee":{"login":"nmaier","id":383665,"avatar_url":"https://avatars.githubusercontent.com/u/383665?v=3","gravatar_id":"","url":"https://api.github.com/users/nmaier","html_url":"https://github.com/nmaier","followers_url":"https://api.github.com/users/nmaier/followers","following_url":"https://api.github.com/users/nmaier/following{/other_user}","gists_url":"https://api.github.com/users/nmaier/gists{/gist_id}","starred_url":"https://api.github.com/users/nmaier/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/nmaier/subscriptions","organizations_url":"https://api.github.com/users/nmaier/orgs","repos_url":"https://api.github.com/users/nmaier/repos","events_url":"https://api.github.com/users/nmaier/events{/privacy}","received_events_url":"https://api.github.com/users/nmaier/received_events","type":"User","site_admin":false},"assignees":[{"login":"nmaier","id":383665,"avatar_url":"https://avatars.githubusercontent.com/u/383665?v=3","gravatar_id":"","url":"https://api.github.com/users/nmaier","html_url":"https://github.com/nmaier","followers_url":"https://api.github.com/users/nmaier/followers","following_url":"https://api.github.com/users/nmaier/following{/other_user}","gists_url":"https://api.github.com/users/nmaier/gists{/gist_id}","starred_url":"https://api.github.com/users/nmaier/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/nmaier/subscriptions","organizations_url":"https://api.github.com/users/nmaier/orgs","repos_url":"https://api.github.com/users/nmaier/repos","events_url":"https://api.github.com/users/nmaier/events{/privacy}","received_events_url":"https://api.github.com/users/nmaier/received_events","type":"User","site_admin":false}],"milestone":null,"comments":6,"created_at":"2016-09-12T17:47:07Z","updated_at":"2016-09-17T22:57:51Z","closed_at":null,"body":"## Description\r\nDownThemAll's new Metalink-support adds certain headers (see below) that makes CloudFlare's security mechanisms deem the requests automated, and not from a user. This trips the CloudFlare's protection features to block the request with `HTTP 403 Forbidden error` response.\r\n\r\nDetails and suggestion for a fix follow.\r\n\r\n---\r\n\r\n### Your DownThemAll! and Browser Version and Language and Operating System\r\nDownThemAll! 3.0.6\r\nFirefox 48.0.2 English\r\nWindows 7 64-bit\r\n\r\n### Expected Behavior\r\nTo download media without a problem using DTA from a server protected by CloudFlare.\r\n\r\n### Actual Behavior\r\nServer responds with `HTTP/1.1 403 Forbidden`.\r\n\r\n### Steps To Reproduce\r\nTry to download media from a CloudFlare-protected HTTP server.\r\n\r\n## Additional info such as log extracts\r\nThe relevant **HTTP request** headers sent by DownThemAll! are:\r\n```\r\nAccept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8, application/metalink4+xml;q=0.9,application/metalink+xml;q=0.8\r\nWant-Digest: SHA256,SHA512,SHA;q=0.4,SHA1;q=0.4,MD5;q=0.3\r\n```\r\n\r\nThis results in the server replying with `HTTP/1.1 403 Forbidden`.\r\n\r\nThis is the reason that the **downloads fail** and DownThemAll! window shows them as `\"403 error *\"`. Interestingly, when the downloads are afterwards manually resumed, different set of HTTP headers are sent by DTA, of which the relevant are:\r\n```\r\nAccept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8\r\nand the absence of \"Want-Digest\" header\r\n```\r\nThis, again interestingly, makes the downloads **succeed**!\r\n\r\nBecause of these observations, the knownledge that CloudFlare has \"scraping protections\" and [damoncloudflare's response on this StackOverflow question][so], I conclude that CloudFlare determines the requesting HTTP client to be a non-user by the headers involving metalink, and thus, makes the HTTP request fail.\r\n\r\n[so]: https://stackoverflow.com/questions/29130137/has-anyone-experienced-cloudflare-403-errors-with-zombie-js-web-scraping\r\n\r\n---\r\n\r\n## Suggestion\r\n\r\nCurrently DTA has option to allow the user to choose whether DTA should include it's presence in the HTTP request's User-Agent header (Advanced -> Miscellaneous -> \"Expose DownThemAll!). This option should, in my opinion, be renamed to be more general, and to be extended to include the metalink headers as well (to choose that they are not sent). Please make it configurable, because currently DTA cannot be used with sites hosted behind CloudFlare.\r\n\r\nApparently this is the relevant part in DTA sourcecode: https://github.com/downthemall/downthemall/blob/6f145f130da0b391fcd1c2b7d7b33e22ec1a1893/modules/manager/connection.js#L204\r\n"},"comment":{"url":"https://api.github.com/repos/downthemall/downthemall/issues/comments/247813211","html_url":"https://github.com/downthemall/downthemall/issues/83#issuecomment-247813211","issue_url":"https://api.github.com/repos/downthemall/downthemall/issues/83","id":247813211,"user":{"login":"nmaier","id":383665,"avatar_url":"https://avatars.githubusercontent.com/u/383665?v=3","gravatar_id":"","url":"https://api.github.com/users/nmaier","html_url":"https://github.com/nmaier","followers_url":"https://api.github.com/users/nmaier/followers","following_url":"https://api.github.com/users/nmaier/following{/other_user}","gists_url":"https://api.github.com/users/nmaier/gists{/gist_id}","starred_url":"https://api.github.com/users/nmaier/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/nmaier/subscriptions","organizations_url":"https://api.github.com/users/nmaier/orgs","repos_url":"https://api.github.com/users/nmaier/repos","events_url":"https://api.github.com/users/nmaier/events{/privacy}","received_events_url":"https://api.github.com/users/nmaier/received_events","type":"User","site_admin":false},"created_at":"2016-09-17T22:57:50Z","updated_at":"2016-09-17T22:57:50Z","body":"We release whenever it makes sense to release. Right now that is \"soon\"\r\n\r\nIn the meantime, we got nightly builds which are signed.\r\nhttps://code.downthemall.net/nightly/\r\nYou need to disable the release DTA version first tho"}}
{"actor":{"display_login":"nmaier"}}