{"action":"created","issue":{"url":"https://api.github.com/repos/tensorpack/tensorpack/issues/969","repository_url":"https://api.github.com/repos/tensorpack/tensorpack","labels_url":"https://api.github.com/repos/tensorpack/tensorpack/issues/969/labels{/name}","comments_url":"https://api.github.com/repos/tensorpack/tensorpack/issues/969/comments","events_url":"https://api.github.com/repos/tensorpack/tensorpack/issues/969/events","html_url":"https://github.com/tensorpack/tensorpack/issues/969","id":377499881,"node_id":"MDU6SXNzdWUzNzc0OTk4ODE=","number":969,"title":"Clean up parallel Dataflow (PrefetchDataZMQ)","user":{"login":"sytham","id":5081109,"node_id":"MDQ6VXNlcjUwODExMDk=","avatar_url":"https://avatars2.githubusercontent.com/u/5081109?v=4","gravatar_id":"","url":"https://api.github.com/users/sytham","html_url":"https://github.com/sytham","followers_url":"https://api.github.com/users/sytham/followers","following_url":"https://api.github.com/users/sytham/following{/other_user}","gists_url":"https://api.github.com/users/sytham/gists{/gist_id}","starred_url":"https://api.github.com/users/sytham/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/sytham/subscriptions","organizations_url":"https://api.github.com/users/sytham/orgs","repos_url":"https://api.github.com/users/sytham/repos","events_url":"https://api.github.com/users/sytham/events{/privacy}","received_events_url":"https://api.github.com/users/sytham/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":2,"created_at":"2018-11-05T17:17:53Z","updated_at":"2018-11-06T08:07:19Z","closed_at":null,"author_association":"NONE","body":"**Problem**\r\nProcesses spawned by `PrefetchDataZMQ` stay alive. I'm doing a hyperparameter search, and since the dataflow is not reentrant, my understanding is I need to create a new one for each hyperparam evaluation. This leads to a very large amount of live processes, eventually causing my machine to run out of memory.\r\nI create my dataflow as:\r\n```\r\ndef create_dataflow(input_data, target_data):\r\n    ds = DataFromList(zip(input_data, target_data), shuffle=True)\r\n    ds = BatchData(ds, BATCH_SIZE)\r\n    ds = PrefetchDataZMQ(ds, nr_proc=NUM_PARALLEL_FETCHERS)\r\n    return ds\r\n```\r\n\r\n**What have I tried**\r\n\r\n- reuse the same dataflow. This leads to the reentrant eror.\r\n- in each hyperparam evaluation, do the following:\r\n```\r\ndataflow_train = create_dataflow(train_inp, train_targ)\r\n<do stuff>\r\ndel dataflow_train\r\ndel model # holds a reference to dataflow_train\r\ngc.collect()\r\n```\r\nSince it seems the `_MultiProcessZMQDataFlow.__del__` method should do the cleanup. However, I never see the \"successfully cleaned-up\" message that this method prints if successful. It seems this could either be because `__del__` isn't called, or because `_MultiProcessZMQDataFlow._reset_done = False` (I wouldn't understand why).\r\n\r\nIn any case, is there a better way to do this?\r\n"},"comment":{"url":"https://api.github.com/repos/tensorpack/tensorpack/issues/comments/436164644","html_url":"https://github.com/tensorpack/tensorpack/issues/969#issuecomment-436164644","issue_url":"https://api.github.com/repos/tensorpack/tensorpack/issues/969","id":436164644,"node_id":"MDEyOklzc3VlQ29tbWVudDQzNjE2NDY0NA==","user":{"login":"ppwwyyxx","id":1381301,"node_id":"MDQ6VXNlcjEzODEzMDE=","avatar_url":"https://avatars3.githubusercontent.com/u/1381301?v=4","gravatar_id":"","url":"https://api.github.com/users/ppwwyyxx","html_url":"https://github.com/ppwwyyxx","followers_url":"https://api.github.com/users/ppwwyyxx/followers","following_url":"https://api.github.com/users/ppwwyyxx/following{/other_user}","gists_url":"https://api.github.com/users/ppwwyyxx/gists{/gist_id}","starred_url":"https://api.github.com/users/ppwwyyxx/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ppwwyyxx/subscriptions","organizations_url":"https://api.github.com/users/ppwwyyxx/orgs","repos_url":"https://api.github.com/users/ppwwyyxx/repos","events_url":"https://api.github.com/users/ppwwyyxx/events{/privacy}","received_events_url":"https://api.github.com/users/ppwwyyxx/received_events","type":"User","site_admin":false},"created_at":"2018-11-06T08:07:19Z","updated_at":"2018-11-06T08:07:19Z","author_association":"COLLABORATOR","body":"Thanks for the short repro! There might be a bug somewhere that mistakenly keeps reference to the dataflow. I'll take a look at that!"}}
{"actor":{"display_login":"ppwwyyxx"}}