{"action":"created","issue":{"url":"https://api.github.com/repos/boto/boto3/issues/1670","repository_url":"https://api.github.com/repos/boto/boto3","labels_url":"https://api.github.com/repos/boto/boto3/issues/1670/labels{/name}","comments_url":"https://api.github.com/repos/boto/boto3/issues/1670/comments","events_url":"https://api.github.com/repos/boto/boto3/issues/1670/events","html_url":"https://github.com/boto/boto3/issues/1670","id":352524027,"node_id":"MDU6SXNzdWUzNTI1MjQwMjc=","number":1670,"title":"Excessive memory usage on multithreading","user":{"login":"jbvsmo","id":1024423,"node_id":"MDQ6VXNlcjEwMjQ0MjM=","avatar_url":"https://avatars3.githubusercontent.com/u/1024423?v=4","gravatar_id":"","url":"https://api.github.com/users/jbvsmo","html_url":"https://github.com/jbvsmo","followers_url":"https://api.github.com/users/jbvsmo/followers","following_url":"https://api.github.com/users/jbvsmo/following{/other_user}","gists_url":"https://api.github.com/users/jbvsmo/gists{/gist_id}","starred_url":"https://api.github.com/users/jbvsmo/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jbvsmo/subscriptions","organizations_url":"https://api.github.com/users/jbvsmo/orgs","repos_url":"https://api.github.com/users/jbvsmo/repos","events_url":"https://api.github.com/users/jbvsmo/events{/privacy}","received_events_url":"https://api.github.com/users/jbvsmo/received_events","type":"User","site_admin":false},"labels":[{"id":217065744,"node_id":"MDU6TGFiZWwyMTcwNjU3NDQ=","url":"https://api.github.com/repos/boto/boto3/labels/closing-soon-if-no-response","name":"closing-soon-if-no-response","color":"5319e7","default":false}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":3,"created_at":"2018-08-21T12:50:50Z","updated_at":"2018-08-23T18:00:11Z","closed_at":null,"author_association":"NONE","body":"I have been trying to debug a \"memory leak\" in my newly upgraded boto3 application. I am moving from the original boto 2.49.\r\n\r\nMy application starts a pool of 100 thread and every request is queued and redirected to one of these threads and usual memory for the lifetime of the appication was about 1GB with peaks of 1.5GB depending of the operation.\r\n\r\nAfter the upgrade I added one `boto3.Session` per thread and I access multiple resources and clients from this session which are reused throughout the code. On previous code I would have a boto connection of each kind per thread (I use several services like S3, DynamoDB, SES, SQS, Mturk, SimpleDB) so it is pretty much the same thing.\r\n\r\nExcept that each boto3.Session alone uses increases memory usage immensely and now my application is running on 3GB of memory instead.\r\n\r\nHow do I know it is the boto3 Session, you ask? I created 2 demo experiments with the same 100 threads and the only difference on both is using boto3 in one and not on the other.\r\n\r\nProgram 1: https://pastebin.com/Urkh3TDU\r\nProgram 2: https://pastebin.com/eDWPcS8C  (Same thing with 5 lines regarding boto commented out)\r\n\r\nOutput program 1 (each print happens 5 seconds after the last one):\r\n```\r\nProcess Memory: 39.4 MB\r\nProcess Memory: 261.7 MB\r\nProcess Memory: 518.7 MB\r\nProcess Memory: 788.2 MB\r\nProcess Memory: 944.5 MB\r\nProcess Memory: 940.1 MB\r\nProcess Memory: 944.4 MB\r\nProcess Memory: 948.7 MB\r\nProcess Memory: 959.1 MB\r\nProcess Memory: 957.4 MB\r\nProcess Memory: 958.0 MB\r\nProcess Memory: 959.5 MB\r\n```\r\n\r\nNow with plain multiple threads and no AWS access.\r\nOutput program 2 (each print happens 5 seconds after the last one):\r\n```\r\nProcess Memory: 23.5 MB\r\nProcess Memory: 58.7 MB\r\nProcess Memory: 58.7 MB\r\nProcess Memory: 58.7 MB\r\nProcess Memory: 58.7 MB\r\nProcess Memory: 58.7 MB\r\nProcess Memory: 58.7 MB\r\nProcess Memory: 58.7 MB\r\nProcess Memory: 58.7 MB\r\nProcess Memory: 58.7 MB\r\n```\r\n\r\nAlone the boto3 session object is retaining 10MB per thread in a total of about 1GB. This is not acceptable from an object that should not be doing much more than requesting stuff to the AWS servers only. It means that the Session is keeping lots of unwanted information.\r\n\r\nYou could be wondering if it is not the resource that is keeping live memory. If you move the resource creation to inside the for loop, the program will also hit the 1GB in the exact the same 15 to 20 seconds of existence.\r\n\r\nIn the beginning I tried garbage collecting for cyclic references but it was futile. The decrease in memory was only a couple megabytes.\r\n\r\nI've seen people complaining on botocore project on something similar (maybe not!), so it might be a shared issue.\r\nhttps://github.com/boto/botocore/issues/805\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/boto/boto3/issues/comments/415512630","html_url":"https://github.com/boto/boto3/issues/1670#issuecomment-415512630","issue_url":"https://api.github.com/repos/boto/boto3/issues/1670","id":415512630,"node_id":"MDEyOklzc3VlQ29tbWVudDQxNTUxMjYzMA==","user":{"login":"gridgazer","id":3021645,"node_id":"MDQ6VXNlcjMwMjE2NDU=","avatar_url":"https://avatars2.githubusercontent.com/u/3021645?v=4","gravatar_id":"","url":"https://api.github.com/users/gridgazer","html_url":"https://github.com/gridgazer","followers_url":"https://api.github.com/users/gridgazer/followers","following_url":"https://api.github.com/users/gridgazer/following{/other_user}","gists_url":"https://api.github.com/users/gridgazer/gists{/gist_id}","starred_url":"https://api.github.com/users/gridgazer/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/gridgazer/subscriptions","organizations_url":"https://api.github.com/users/gridgazer/orgs","repos_url":"https://api.github.com/users/gridgazer/repos","events_url":"https://api.github.com/users/gridgazer/events{/privacy}","received_events_url":"https://api.github.com/users/gridgazer/received_events","type":"User","site_admin":false},"created_at":"2018-08-23T18:00:11Z","updated_at":"2018-08-23T18:00:11Z","author_association":"NONE","body":"@joguSD Sorry but that doesn't explain why memory is not being released even with cyclic garbage collection in place.\r\n\r\nAnd the most strange part is if I do `del s3` at the end of the for loop, a really good chunk of memory will go away. No python code is really expect to \"free\" resources.\r\n\r\nThe code you provided is very alike from what I ended up using in my application but besides the 100 threads with the same lifespan as the program, I occasionally run other things in parallel on other threads and those add to total memory that is never freed!\r\n\r\nAfter a few days, my 8GB server is out of memory again. This is *at least* poor memory management on boto3 part. The only solution I am seeing is revert 2 weeks of porting code and go back to boto 2.49\r\n\r\n\r\n\r\n"}}
{"actor":{"display_login":"gridgazer"}}