{"action":"created","issue":{"url":"https://api.github.com/repos/mozilla/socorro/issues/3253","repository_url":"https://api.github.com/repos/mozilla/socorro","labels_url":"https://api.github.com/repos/mozilla/socorro/issues/3253/labels{/name}","comments_url":"https://api.github.com/repos/mozilla/socorro/issues/3253/comments","events_url":"https://api.github.com/repos/mozilla/socorro/issues/3253/events","html_url":"https://github.com/mozilla/socorro/pull/3253","id":144229289,"number":3253,"title":"fixes bug 1260216 - Use requests in ftpscraper.py","user":{"login":"peterbe","id":26739,"avatar_url":"https://avatars.githubusercontent.com/u/26739?v=3","gravatar_id":"","url":"https://api.github.com/users/peterbe","html_url":"https://github.com/peterbe","followers_url":"https://api.github.com/users/peterbe/followers","following_url":"https://api.github.com/users/peterbe/following{/other_user}","gists_url":"https://api.github.com/users/peterbe/gists{/gist_id}","starred_url":"https://api.github.com/users/peterbe/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/peterbe/subscriptions","organizations_url":"https://api.github.com/users/peterbe/orgs","repos_url":"https://api.github.com/users/peterbe/repos","events_url":"https://api.github.com/users/peterbe/events{/privacy}","received_events_url":"https://api.github.com/users/peterbe/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":{"login":"willkg","id":820826,"avatar_url":"https://avatars.githubusercontent.com/u/820826?v=3","gravatar_id":"","url":"https://api.github.com/users/willkg","html_url":"https://github.com/willkg","followers_url":"https://api.github.com/users/willkg/followers","following_url":"https://api.github.com/users/willkg/following{/other_user}","gists_url":"https://api.github.com/users/willkg/gists{/gist_id}","starred_url":"https://api.github.com/users/willkg/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/willkg/subscriptions","organizations_url":"https://api.github.com/users/willkg/orgs","repos_url":"https://api.github.com/users/willkg/repos","events_url":"https://api.github.com/users/willkg/events{/privacy}","received_events_url":"https://api.github.com/users/willkg/received_events","type":"User","site_admin":false},"milestone":null,"comments":1,"created_at":"2016-03-29T11:03:38Z","updated_at":"2016-03-30T17:04:21Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/mozilla/socorro/pulls/3253","html_url":"https://github.com/mozilla/socorro/pull/3253","diff_url":"https://github.com/mozilla/socorro/pull/3253.diff","patch_url":"https://github.com/mozilla/socorro/pull/3253.patch"},"body":"To actually run:\r\n```\r\ntime python socorro/cron/jobs/ftpscraper.py --product=thunderbird,firefox,mobile\r\n```\r\n(Takes about 5 minutes and instead of writing into PG, it just prints what it would insert)\r\n\r\nI'm not convinced about the way the test classes are set up with the mocking. Not sure it's particularly important either. "},"comment":{"url":"https://api.github.com/repos/mozilla/socorro/issues/comments/203528382","html_url":"https://github.com/mozilla/socorro/pull/3253#issuecomment-203528382","issue_url":"https://api.github.com/repos/mozilla/socorro/issues/3253","id":203528382,"user":{"login":"willkg","id":820826,"avatar_url":"https://avatars.githubusercontent.com/u/820826?v=3","gravatar_id":"","url":"https://api.github.com/users/willkg","html_url":"https://github.com/willkg","followers_url":"https://api.github.com/users/willkg/followers","following_url":"https://api.github.com/users/willkg/following{/other_user}","gists_url":"https://api.github.com/users/willkg/gists{/gist_id}","starred_url":"https://api.github.com/users/willkg/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/willkg/subscriptions","organizations_url":"https://api.github.com/users/willkg/orgs","repos_url":"https://api.github.com/users/willkg/repos","events_url":"https://api.github.com/users/willkg/events{/privacy}","received_events_url":"https://api.github.com/users/willkg/received_events","type":"User","site_admin":false},"created_at":"2016-03-30T17:04:21Z","updated_at":"2016-03-30T17:04:21Z","body":"I hit a 503 which the assertion dies on:\r\n\r\n```\r\nTraceback (most recent call last):\r\n  File \"/home/vagrant/socorro/socorro-virtualenv/lib/python2.7/site-packages/crontabber/transaction_executor.py\", line 46, in __call__\r\n    result = function(connection, *args, **kwargs)\r\n  File \"/home/vagrant/socorro/socorro/cron/jobs/ftpscraper.py\", line 365, in _scrape_json_releases_and_nightlies\r\n    self.scrape_json_releases(connection, product_name)\r\n  File \"/home/vagrant/socorro/socorro/cron/jobs/ftpscraper.py\", line 400, in scrape_json_releases\r\n    for info in self.get_json_release(release, dirname):\r\n  File \"/home/vagrant/socorro/socorro/cron/jobs/ftpscraper.py\", line 176, in get_json_release\r\n    kvpairs = self.parse_build_json_file(json_url)\r\n  File \"/home/vagrant/socorro/socorro/cron/jobs/ftpscraper.py\", line 69, in parse_build_json_file\r\n    content = self.download(url)\r\n  File \"/home/vagrant/socorro/socorro/cron/jobs/ftpscraper.py\", line 335, in download\r\n    assert response.status_code == 200, response.status_code\r\nAssertionError: 503\r\n```\r\n\r\nIt's this url:\r\n\r\n```\r\n2016-03-30 16:59:13,503 WARNING - SocorroAppBaseClass -  - MainThread - warning, unsupported JSON file: https://archive.mozilla.org/pub/thunderbird/candidates/42.0b2-candidates/build1/linux-i686/en-US/thunderbird-42.0b2.mozinfo.json\r\n```\r\n\r\nWhen I ran ftpscraper on master, I didn't hit this. However, with this branch, I hit it three times in a row.\r\n\r\nCouple of possibilities:\r\n\r\n1. the issue is ephemeral--so we wait a bit and it goes away and yay!\r\n2. there's some difference between the old code and the new code--I'll look into this\r\n3. something changed with those files or that server in the last hour--I'll look into this, too\r\n\r\nThat's the update.\r\n\r\nI still have to look at the code changes."}}