{"action":"created","issue":{"url":"https://api.github.com/repos/boto/boto3/issues/814","repository_url":"https://api.github.com/repos/boto/boto3","labels_url":"https://api.github.com/repos/boto/boto3/issues/814/labels{/name}","comments_url":"https://api.github.com/repos/boto/boto3/issues/814/comments","events_url":"https://api.github.com/repos/boto/boto3/issues/814/events","html_url":"https://github.com/boto/boto3/issues/814","id":177882000,"number":814,"title":"Hang in s3.download_file with Celery worker in version 1.4.0","user":{"login":"dmulter","id":411993,"avatar_url":"https://avatars.githubusercontent.com/u/411993?v=3","gravatar_id":"","url":"https://api.github.com/users/dmulter","html_url":"https://github.com/dmulter","followers_url":"https://api.github.com/users/dmulter/followers","following_url":"https://api.github.com/users/dmulter/following{/other_user}","gists_url":"https://api.github.com/users/dmulter/gists{/gist_id}","starred_url":"https://api.github.com/users/dmulter/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dmulter/subscriptions","organizations_url":"https://api.github.com/users/dmulter/orgs","repos_url":"https://api.github.com/users/dmulter/repos","events_url":"https://api.github.com/users/dmulter/events{/privacy}","received_events_url":"https://api.github.com/users/dmulter/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/boto/boto3/labels/researching","name":"researching","color":"006b75"}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":25,"created_at":"2016-09-19T20:00:58Z","updated_at":"2016-10-17T23:46:32Z","closed_at":null,"body":"I've been using lots of boto3 calls in my Flask app for some time, but the switch to the latest boto3 v1.4.0 has broken my Celery workers. Something that may be unique about my app is that I use S3 to download a secure environment variables file before launching my app or workers. It appears that the new boto3 works with my app, but hangs when launching the Celery worker.\r\n\r\nI would temporarily downgrade my boto3 to avoid the problem, but its been a long time since the last release, and I need the elbv2 support that only comes in 1.4.0.\r\n\r\nI've created a tiny version of my worker (worker2.py) to demonstrate the problem. I've verified that using the previous version boto3 1.3.1 results in the worker launching properly. I see all prints and the Celery worker banner output.\r\n\r\nIf I install boto3 1.4.0, then the second print() statement \"Download complete\" is never reached.  Also note that I tried following the new doc example with boto3.resource and using s3.meta.client, but that fails as well.\r\n\r\n```python\r\n#\r\n# Stub Celery worker to demonstrate bug in Boto3 1.4.0. Works fine with previous version Boto3 1.3.1.\r\n# Test with: celery worker -A worker2.celery\r\n#\r\nfrom flask import Flask\r\nfrom celery import Celery\r\nimport boto3\r\nimport tempfile\r\n\r\ncelery = Celery(__name__, broker='amqp://guest:guest@localhost:5672//')\r\n\r\napp = Flask(__name__)\r\n\r\ns3 = boto3.client('s3', region_name='us-west-1')\r\nenv_file = 'APPNAME.APPSTAGE.env'\r\nwith tempfile.NamedTemporaryFile() as s3_file:\r\n    print(\"Downloading file...\")\r\n    response = s3.download_file('APPBUCKET', env_file, s3_file.name)\r\n    print(\"Download complete!\")\r\n```\r\n\r\nYou can test it by running the following at the command line:\r\n```\r\ncelery worker -A worker2.celery\r\n```\r\n\r\nAlso note that just running the code downloads the file just fine with 1.4.0:\r\n\r\n```\r\npython worker2.py\r\n```"},"comment":{"url":"https://api.github.com/repos/boto/boto3/issues/comments/254366954","html_url":"https://github.com/boto/boto3/issues/814#issuecomment-254366954","issue_url":"https://api.github.com/repos/boto/boto3/issues/814","id":254366954,"user":{"login":"kyleknap","id":4605355,"avatar_url":"https://avatars.githubusercontent.com/u/4605355?v=3","gravatar_id":"","url":"https://api.github.com/users/kyleknap","html_url":"https://github.com/kyleknap","followers_url":"https://api.github.com/users/kyleknap/followers","following_url":"https://api.github.com/users/kyleknap/following{/other_user}","gists_url":"https://api.github.com/users/kyleknap/gists{/gist_id}","starred_url":"https://api.github.com/users/kyleknap/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kyleknap/subscriptions","organizations_url":"https://api.github.com/users/kyleknap/orgs","repos_url":"https://api.github.com/users/kyleknap/repos","events_url":"https://api.github.com/users/kyleknap/events{/privacy}","received_events_url":"https://api.github.com/users/kyleknap/received_events","type":"User","site_admin":false},"created_at":"2016-10-17T23:46:32Z","updated_at":"2016-10-17T23:46:32Z","body":"@dmulter I do not think that setting it to one will help because the main transfer process will not happen in the main thread. The only way to get this to work is by having all of the transfer logic in the main thread. I am currently looking at a way to set this in boto3.\r\n\r\nIn the meantime, I would recommend trying to lazily call the ``download_file`` as suggested before or even look to use ``client.get_object()`` directly to download the object as the object you are downloading is very small and does not really need the managed transfer methods."}}
{"actor":{"display_login":"kyleknap"}}