{"action":"created","issue":{"url":"https://api.github.com/repos/potatolondon/djangae/issues/790","repository_url":"https://api.github.com/repos/potatolondon/djangae","labels_url":"https://api.github.com/repos/potatolondon/djangae/issues/790/labels{/name}","comments_url":"https://api.github.com/repos/potatolondon/djangae/issues/790/comments","events_url":"https://api.github.com/repos/potatolondon/djangae/issues/790/events","html_url":"https://github.com/potatolondon/djangae/issues/790","id":193382255,"number":790,"title":"Issues with defer_iteration","user":{"login":"adamalton","id":270255,"avatar_url":"https://avatars.githubusercontent.com/u/270255?v=3","gravatar_id":"","url":"https://api.github.com/users/adamalton","html_url":"https://github.com/adamalton","followers_url":"https://api.github.com/users/adamalton/followers","following_url":"https://api.github.com/users/adamalton/following{/other_user}","gists_url":"https://api.github.com/users/adamalton/gists{/gist_id}","starred_url":"https://api.github.com/users/adamalton/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/adamalton/subscriptions","organizations_url":"https://api.github.com/users/adamalton/orgs","repos_url":"https://api.github.com/users/adamalton/repos","events_url":"https://api.github.com/users/adamalton/events{/privacy}","received_events_url":"https://api.github.com/users/adamalton/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":2,"created_at":"2016-12-04T22:48:47Z","updated_at":"2016-12-05T10:31:42Z","closed_at":null,"body":"I have been trying to use `djangae.contrib.mappers.defer_iteration` to map over a large number of entities (~350k), and it's not doing very well!..\r\n\r\nThe first problem was that I kept on getting lots of tasks failing with\r\n\r\n```\r\n21:55:42.301\r\nExceeded soft private memory limit of 128 MB with 153 MB after servicing 0 requests total\r\n21:55:42.302\r\nWhile handling this request, the process that handled this request was found to be using too much memory and was terminated. This is likely to cause a new process to be used for the next request to your application. If you see this message frequently, you may have a memory leak in your application.\r\n```\r\n\r\nso I did some modification to `defer_iteration` so that the intial sharding was done on a different queue to the actual processing of the shards, and redployed and re-ran it.\r\n\r\nThat revealed that the sharding task itself was:\r\n\r\n1. Failing and retrying.\r\n2. Multiplying!  I.e. the queue which **only had the sharding on it** was gaining more and more tasks.\r\n\r\nSo looking a bit more closely at `defer_iteration` I noticed that:\r\n\r\n1. It only re-defers the sharding task when it hits `DeadlineExceededError`, so if it hits any other error (like a memory limit) then that error won't be caught and the whole task will just retry and all the sharding will start from the beginning again.\r\n2. When it catches `DeadlineExceededError` it re-defers itself, but it doesn't `return`!  So it will still continue and hit the second (hard) `DeadlineExceededError` and get retried.  Thus the tasks multiply.\r\n\r\nI'm still not sure what was causing the memory limit errors though.  My model and code is really simple, so I'm pretty sure it's something to do with the workings of `defer_iteration`, rather than anything in my code.  I tried disabling the Djangae cache (including with the fixes from #785), but the memory errors still happened.\r\n\r\nI also got some requests with these things, which is odd:\r\n\r\n```\r\nms=10448 cpu_ms=0 cpm_usd=0 loading_request=0\r\nRequest was aborted after waiting too long to attempt to service your request.\r\n```\r\n\r\nPossibly a scaling thing, but possibly related.\r\n\r\n**Conclusion**\r\n\r\nI think that we should:\r\n\r\n1. Have an arbitrary limit to the number of shards that we create before we re-defer the sharding task.  This will:\r\n    * _Possibly_ help it not hit memory errors.  Although possibly not.\r\n    * Prevent the case where we hit `DeadlineExceededError` and do not manage to re-defer before the hard deadline kicks in.\r\n2. Add a `return` statement after we re-defer.\r\n\r\nWe might need to make other fixes too, but I'm not quite sure what yet."},"comment":{"url":"https://api.github.com/repos/potatolondon/djangae/issues/comments/264819298","html_url":"https://github.com/potatolondon/djangae/issues/790#issuecomment-264819298","issue_url":"https://api.github.com/repos/potatolondon/djangae/issues/790","id":264819298,"user":{"login":"adamalton","id":270255,"avatar_url":"https://avatars.githubusercontent.com/u/270255?v=3","gravatar_id":"","url":"https://api.github.com/users/adamalton","html_url":"https://github.com/adamalton","followers_url":"https://api.github.com/users/adamalton/followers","following_url":"https://api.github.com/users/adamalton/following{/other_user}","gists_url":"https://api.github.com/users/adamalton/gists{/gist_id}","starred_url":"https://api.github.com/users/adamalton/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/adamalton/subscriptions","organizations_url":"https://api.github.com/users/adamalton/orgs","repos_url":"https://api.github.com/users/adamalton/repos","events_url":"https://api.github.com/users/adamalton/events{/privacy}","received_events_url":"https://api.github.com/users/adamalton/received_events","type":"User","site_admin":false},"created_at":"2016-12-05T10:31:42Z","updated_at":"2016-12-05T10:31:42Z","body":"Re. the memory issues, I did 2 things which together fixed it:\r\n\r\n* Reduced the shard size.\r\n* Added `.iterator()` to the queryset so that it only instantiates one object at a time.\r\n\r\nNot sure which one had the bigger effect, but we should do the `.iterator()` thing because it can only make things better."}}
{"actor":{"display_login":"adamalton"}}