{"action":"opened","issue":{"url":"https://api.github.com/repos/markovmodel/PyEMMA/issues/776","repository_url":"https://api.github.com/repos/markovmodel/PyEMMA","labels_url":"https://api.github.com/repos/markovmodel/PyEMMA/issues/776/labels{/name}","comments_url":"https://api.github.com/repos/markovmodel/PyEMMA/issues/776/comments","events_url":"https://api.github.com/repos/markovmodel/PyEMMA/issues/776/events","html_url":"https://github.com/markovmodel/PyEMMA/issues/776","id":149474986,"number":776,"title":"Memory usage in parallel minRMSD clustering","user":{"login":"jchodera","id":3656088,"avatar_url":"https://avatars.githubusercontent.com/u/3656088?v=3","gravatar_id":"","url":"https://api.github.com/users/jchodera","html_url":"https://github.com/jchodera","followers_url":"https://api.github.com/users/jchodera/followers","following_url":"https://api.github.com/users/jchodera/following{/other_user}","gists_url":"https://api.github.com/users/jchodera/gists{/gist_id}","starred_url":"https://api.github.com/users/jchodera/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jchodera/subscriptions","organizations_url":"https://api.github.com/users/jchodera/orgs","repos_url":"https://api.github.com/users/jchodera/repos","events_url":"https://api.github.com/users/jchodera/events{/privacy}","received_events_url":"https://api.github.com/users/jchodera/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":0,"created_at":"2016-04-19T14:20:18Z","updated_at":"2016-04-19T14:20:18Z","closed_at":null,"body":"I'm testing the recent #770 parallel clustering additions on a dataset with ~1M snapshots using 32 processes, and find it violates the 96GB memory limit I give it after ~12 hours.\r\n\r\nIs it expected that memory usage should slowly grow during clustering, or are there provisions made to keep memory usage minimal when chunking is in use?\r\n\r\nIf the expectation is that memory usage should remain minimal, I wonder if there is an unintentional memory leak. We frequently encounter cases where relying on Python garbage collection leads to slow growth in memory usage, where we have to explicitly `del` objects after we're done with them, even if all references fall out of scope.\r\n\r\n"}}