{"action":"created","issue":{"url":"https://api.github.com/repos/pypa/bandersnatch/issues/47","repository_url":"https://api.github.com/repos/pypa/bandersnatch","labels_url":"https://api.github.com/repos/pypa/bandersnatch/issues/47/labels{/name}","comments_url":"https://api.github.com/repos/pypa/bandersnatch/issues/47/comments","events_url":"https://api.github.com/repos/pypa/bandersnatch/issues/47/events","html_url":"https://github.com/pypa/bandersnatch/issues/47","id":334518735,"node_id":"MDU6SXNzdWUzMzQ1MTg3MzU=","number":47,"title":"Verify failures","user":{"login":"electricworry","id":10690531,"node_id":"MDQ6VXNlcjEwNjkwNTMx","avatar_url":"https://avatars2.githubusercontent.com/u/10690531?v=4","gravatar_id":"","url":"https://api.github.com/users/electricworry","html_url":"https://github.com/electricworry","followers_url":"https://api.github.com/users/electricworry/followers","following_url":"https://api.github.com/users/electricworry/following{/other_user}","gists_url":"https://api.github.com/users/electricworry/gists{/gist_id}","starred_url":"https://api.github.com/users/electricworry/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/electricworry/subscriptions","organizations_url":"https://api.github.com/users/electricworry/orgs","repos_url":"https://api.github.com/users/electricworry/repos","events_url":"https://api.github.com/users/electricworry/events{/privacy}","received_events_url":"https://api.github.com/users/electricworry/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":3,"created_at":"2018-06-21T14:25:26Z","updated_at":"2018-10-30T04:31:07Z","closed_at":"2018-10-30T04:31:07Z","author_association":"NONE","body":"I decided to check out the latest development of bandersnatch to see how the verification is coming along. (I had pointed out [previously](https://bitbucket.org/pypa/bandersnatch/issues/94/package-files-are-never-purged) that with the new package paths it was not straightforward to identify purged files for removal.)\r\n\r\nI can see that you're well on the way to a solution using the json metadata (good idea!). However, I'm finding the running of \"bandersnatch verify\" a bit fragile. It seems that a single exception causes the asyncio event loop to die:\r\n\r\n```\r\n2018-06-21 08:49:59,249 INFO: Finished validating jvc-proxy\r\n2018-06-21 08:49:59,258 INFO: Finished validating djangocms-background-media\r\nTask was destroyed but it is pending!\r\ntask: <Task pending coro=<TCPConnector._resolve_host() running at /opt/mirror/access/bandersnatch-venv/lib/python3.6/site-packages/aiohttp/connector.py:733> wait_for=<Future finished result=[(<AddressFamily.AF_INET: 2>, <SocketKind.SOCK_STREAM: 1>, 6, '', ('151.101.61.63', 443)), (<AddressFamily.AF_INET6: 10>, <SocketKind.SOCK_STREAM: 1>, 6, '', ('2a04:4e42:f::319', 443, 0, 0))]> cb=[shield.<locals>._done_callback() at /usr/lib/python3.6/asyncio/tasks.py:679]>\r\nTraceback (most recent call last):\r\n  File \"/opt/mirror/access/bandersnatch-venv/bin/bandersnatch\", line 11, in <module>\r\n    load_entry_point('bandersnatch', 'console_scripts', 'bandersnatch')()\r\n  File \"/opt/mirror/access/bandersnatch/src/bandersnatch/main.py\", line 152, in main\r\n    loop.run_until_complete(bandersnatch.verify.metadata_verify(config, args))\r\n  File \"/usr/lib/python3.6/asyncio/base_events.py\", line 468, in run_until_complete\r\n    return future.result()\r\n  File \"/opt/mirror/access/bandersnatch/src/bandersnatch/verify.py\", line 174, in metadata_verify\r\n    config, all_package_files, mirror_base, json_files, args, executor\r\n  File \"/opt/mirror/access/bandersnatch/src/bandersnatch/verify.py\", line 148, in async_verify\r\n    await asyncio.gather(*coros)\r\n  File \"/opt/mirror/access/bandersnatch/src/bandersnatch/verify.py\", line 95, in verify\r\n    await url_fetch(jpkg[\"url\"], pkg_file, executor)\r\n  File \"/opt/mirror/access/bandersnatch/src/bandersnatch/verify.py\", line 128, in url_fetch\r\n    async with session.get(url, timeout=timeout) as response:\r\n  File \"/opt/mirror/access/bandersnatch-venv/lib/python3.6/site-packages/aiohttp/client.py\", line 784, in __aenter__\r\n    self._resp = await self._coro\r\n  File \"/opt/mirror/access/bandersnatch-venv/lib/python3.6/site-packages/aiohttp/client.py\", line 409, in _request\r\n    break\r\n  File \"/opt/mirror/access/bandersnatch-venv/lib/python3.6/site-packages/aiohttp/helpers.py\", line 671, in __exit__\r\n    raise asyncio.TimeoutError from None\r\nconcurrent.futures._base.TimeoutError\r\nexception calling callback for <Future at 0x7fbb5e3af0b8 state=finished returned str>\r\nTraceback (most recent call last):\r\n  File \"/usr/lib/python3.6/concurrent/futures/_base.py\", line 324, in _invoke_callbacks\r\n    callback(self)\r\n  File \"/usr/lib/python3.6/asyncio/futures.py\", line 414, in _call_set_state\r\n    dest_loop.call_soon_threadsafe(_set_state, destination, source)\r\n  File \"/usr/lib/python3.6/asyncio/base_events.py\", line 621, in call_soon_threadsafe\r\n    self._check_closed()\r\n  File \"/usr/lib/python3.6/asyncio/base_events.py\", line 358, in _check_closed\r\n    raise RuntimeError('Event loop is closed')\r\nRuntimeError: Event loop is closed\r\nexception calling callback for <Future at 0x7fbb5e265ba8 state=finished returned str>\r\nTraceback (most recent call last):\r\n  File \"/usr/lib/python3.6/concurrent/futures/_base.py\", line 324, in _invoke_callbacks\r\n    callback(self)\r\n  File \"/usr/lib/python3.6/asyncio/futures.py\", line 414, in _call_set_state\r\n    dest_loop.call_soon_threadsafe(_set_state, destination, source)\r\n  File \"/usr/lib/python3.6/asyncio/base_events.py\", line 621, in call_soon_threadsafe\r\n    self._check_closed()\r\n  File \"/usr/lib/python3.6/asyncio/base_events.py\", line 358, in _check_closed\r\n    raise RuntimeError('Event loop is closed')\r\nRuntimeError: Event loop is closed\r\n```\r\n\r\nI'm completely new to asyncio, so I'm not sure what is the best approach. Can asyncio be configured to ignore errors and continue to the next task? Or is diligent handling of failures required as normal?\r\n\r\nThanks. I'd be happy to help out on this one if I can."},"comment":{"url":"https://api.github.com/repos/pypa/bandersnatch/issues/comments/434168452","html_url":"https://github.com/pypa/bandersnatch/issues/47#issuecomment-434168452","issue_url":"https://api.github.com/repos/pypa/bandersnatch/issues/47","id":434168452,"node_id":"MDEyOklzc3VlQ29tbWVudDQzNDE2ODQ1Mg==","user":{"login":"cooperlees","id":3005596,"node_id":"MDQ6VXNlcjMwMDU1OTY=","avatar_url":"https://avatars1.githubusercontent.com/u/3005596?v=4","gravatar_id":"","url":"https://api.github.com/users/cooperlees","html_url":"https://github.com/cooperlees","followers_url":"https://api.github.com/users/cooperlees/followers","following_url":"https://api.github.com/users/cooperlees/following{/other_user}","gists_url":"https://api.github.com/users/cooperlees/gists{/gist_id}","starred_url":"https://api.github.com/users/cooperlees/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/cooperlees/subscriptions","organizations_url":"https://api.github.com/users/cooperlees/orgs","repos_url":"https://api.github.com/users/cooperlees/repos","events_url":"https://api.github.com/users/cooperlees/events{/privacy}","received_events_url":"https://api.github.com/users/cooperlees/received_events","type":"User","site_admin":false},"created_at":"2018-10-30T04:31:07Z","updated_at":"2018-10-30T04:31:07Z","author_association":"CONTRIBUTOR","body":"This should of been fixed with the executors fix."}}
{"actor":{"display_login":"cooperlees"}}