{"action":"closed","issue":{"url":"https://api.github.com/repos/cortadocodes/scraped-lyric-generator/issues/4","repository_url":"https://api.github.com/repos/cortadocodes/scraped-lyric-generator","labels_url":"https://api.github.com/repos/cortadocodes/scraped-lyric-generator/issues/4/labels{/name}","comments_url":"https://api.github.com/repos/cortadocodes/scraped-lyric-generator/issues/4/comments","events_url":"https://api.github.com/repos/cortadocodes/scraped-lyric-generator/issues/4/events","html_url":"https://github.com/cortadocodes/scraped-lyric-generator/issues/4","id":351914127,"node_id":"MDU6SXNzdWUzNTE5MTQxMjc=","number":4,"title":"Errors caused by server blocking scraper should be handled more nicely","user":{"login":"cortadocodes","id":33758634,"node_id":"MDQ6VXNlcjMzNzU4NjM0","avatar_url":"https://avatars2.githubusercontent.com/u/33758634?v=4","gravatar_id":"","url":"https://api.github.com/users/cortadocodes","html_url":"https://github.com/cortadocodes","followers_url":"https://api.github.com/users/cortadocodes/followers","following_url":"https://api.github.com/users/cortadocodes/following{/other_user}","gists_url":"https://api.github.com/users/cortadocodes/gists{/gist_id}","starred_url":"https://api.github.com/users/cortadocodes/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/cortadocodes/subscriptions","organizations_url":"https://api.github.com/users/cortadocodes/orgs","repos_url":"https://api.github.com/users/cortadocodes/repos","events_url":"https://api.github.com/users/cortadocodes/events{/privacy}","received_events_url":"https://api.github.com/users/cortadocodes/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2018-08-19T16:05:01Z","updated_at":"2018-08-19T17:13:23Z","closed_at":"2018-08-19T17:13:23Z","author_association":"OWNER","body":"**Context**\r\nCurrently, when the server gets too many requests from the scraper, it throws this succession of errors:\r\n\r\n```\r\nTraceback (most recent call last):\r\n  File \"/Users/Marcus1/repos/scraped-lyric-generator/venv/lib/python3.6/site-packages/urllib3/connectionpool.py\", line 600, in urlopen\r\n    chunked=chunked)\r\n  File \"/Users/Marcus1/repos/scraped-lyric-generator/venv/lib/python3.6/site-packages/urllib3/connectionpool.py\", line 384, in _make_request\r\n    six.raise_from(e, None)\r\n  File \"<string>\", line 2, in raise_from\r\n  File \"/Users/Marcus1/repos/scraped-lyric-generator/venv/lib/python3.6/site-packages/urllib3/connectionpool.py\", line 380, in _make_request\r\n    httplib_response = conn.getresponse()\r\n  File \"/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/http/client.py\", line 1331, in getresponse\r\n    response.begin()\r\n  File \"/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/http/client.py\", line 297, in begin\r\n    version, status, reason = self._read_status()\r\n  File \"/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/http/client.py\", line 266, in _read_status\r\n    raise RemoteDisconnected(\"Remote end closed connection without\"\r\nhttp.client.RemoteDisconnected: Remote end closed connection without response\r\n\r\nDuring handling of the above exception, another exception occurred:\r\n\r\nTraceback (most recent call last):\r\n  File \"/Users/Marcus1/repos/scraped-lyric-generator/venv/lib/python3.6/site-packages/requests/adapters.py\", line 445, in send\r\n    timeout=timeout\r\n  File \"/Users/Marcus1/repos/scraped-lyric-generator/venv/lib/python3.6/site-packages/urllib3/connectionpool.py\", line 638, in urlopen\r\n    _stacktrace=sys.exc_info()[2])\r\n  File \"/Users/Marcus1/repos/scraped-lyric-generator/venv/lib/python3.6/site-packages/urllib3/util/retry.py\", line 367, in increment\r\n    raise six.reraise(type(error), error, _stacktrace)\r\n  File \"/Users/Marcus1/repos/scraped-lyric-generator/venv/lib/python3.6/site-packages/urllib3/packages/six.py\", line 685, in reraise\r\n    raise value.with_traceback(tb)\r\n  File \"/Users/Marcus1/repos/scraped-lyric-generator/venv/lib/python3.6/site-packages/urllib3/connectionpool.py\", line 600, in urlopen\r\n    chunked=chunked)\r\n  File \"/Users/Marcus1/repos/scraped-lyric-generator/venv/lib/python3.6/site-packages/urllib3/connectionpool.py\", line 384, in _make_request\r\n    six.raise_from(e, None)\r\n  File \"<string>\", line 2, in raise_from\r\n  File \"/Users/Marcus1/repos/scraped-lyric-generator/venv/lib/python3.6/site-packages/urllib3/connectionpool.py\", line 380, in _make_request\r\n    httplib_response = conn.getresponse()\r\n  File \"/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/http/client.py\", line 1331, in getresponse\r\n    response.begin()\r\n  File \"/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/http/client.py\", line 297, in begin\r\n    version, status, reason = self._read_status()\r\n  File \"/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/http/client.py\", line 266, in _read_status\r\n    raise RemoteDisconnected(\"Remote end closed connection without\"\r\nurllib3.exceptions.ProtocolError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response',))\r\n\r\nDuring handling of the above exception, another exception occurred:\r\n\r\nTraceback (most recent call last):\r\n  File \"/Applications/PyCharm.app/Contents/helpers/pydev/pydevd.py\", line 1664, in <module>\r\n    main()\r\n  File \"/Applications/PyCharm.app/Contents/helpers/pydev/pydevd.py\", line 1658, in main\r\n    globals = debugger.run(setup['file'], None, None, is_module)\r\n  File \"/Applications/PyCharm.app/Contents/helpers/pydev/pydevd.py\", line 1068, in run\r\n    pydev_imports.execfile(file, globals, locals)  # execute the script\r\n  File \"/Applications/PyCharm.app/Contents/helpers/pydev/_pydev_imps/_pydev_execfile.py\", line 18, in execfile\r\n    exec(compile(contents+\"\\n\", file, 'exec'), glob, loc)\r\n  File \"/Users/Marcus1/repos/scraped-lyric-generator/scraped_lyric_generator/scrape_song_lyrics.py\", line 15, in <module>\r\n    song_soup = get_soup(link, CACHE_FILE)\r\n  File \"/Users/Marcus1/repos/scraped-lyric-generator/scraped_lyric_generator/get_relevant_pages.py\", line 37, in get_soup\r\n    response = requests.get(url, timeout=5)\r\n  File \"/Users/Marcus1/repos/scraped-lyric-generator/venv/lib/python3.6/site-packages/requests/api.py\", line 72, in get\r\n    return request('get', url, params=params, **kwargs)\r\n  File \"/Users/Marcus1/repos/scraped-lyric-generator/venv/lib/python3.6/site-packages/requests/api.py\", line 58, in request\r\n    return session.request(method=method, url=url, **kwargs)\r\n  File \"/Users/Marcus1/repos/scraped-lyric-generator/venv/lib/python3.6/site-packages/requests/sessions.py\", line 512, in request\r\n    resp = self.send(prep, **send_kwargs)\r\n  File \"/Users/Marcus1/repos/scraped-lyric-generator/venv/lib/python3.6/site-packages/requests/sessions.py\", line 622, in send\r\n    r = adapter.send(request, **kwargs)\r\n  File \"/Users/Marcus1/repos/scraped-lyric-generator/venv/lib/python3.6/site-packages/requests/adapters.py\", line 495, in send\r\n    raise ConnectionError(err, request=request)\r\nrequests.exceptions.ConnectionError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response',))\r\n```\r\n\r\n**Action**\r\nThese should be handled in a nicer way, explaining the high-level problem likely to have caused them."}}
{"actor":{"display_login":"cortadocodes"}}