{"action":"created","issue":{"url":"https://api.github.com/repos/keras-team/keras/issues/11578","repository_url":"https://api.github.com/repos/keras-team/keras","labels_url":"https://api.github.com/repos/keras-team/keras/issues/11578/labels{/name}","comments_url":"https://api.github.com/repos/keras-team/keras/issues/11578/comments","events_url":"https://api.github.com/repos/keras-team/keras/issues/11578/events","html_url":"https://github.com/keras-team/keras/issues/11578","id":377268788,"node_id":"MDU6SXNzdWUzNzcyNjg3ODg=","number":11578,"title":"fit_generator Multiprocessing Overhead - Performance Optimization Ideas?","user":{"login":"mharradon","id":308108,"node_id":"MDQ6VXNlcjMwODEwOA==","avatar_url":"https://avatars0.githubusercontent.com/u/308108?v=4","gravatar_id":"","url":"https://api.github.com/users/mharradon","html_url":"https://github.com/mharradon","followers_url":"https://api.github.com/users/mharradon/followers","following_url":"https://api.github.com/users/mharradon/following{/other_user}","gists_url":"https://api.github.com/users/mharradon/gists{/gist_id}","starred_url":"https://api.github.com/users/mharradon/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mharradon/subscriptions","organizations_url":"https://api.github.com/users/mharradon/orgs","repos_url":"https://api.github.com/users/mharradon/repos","events_url":"https://api.github.com/users/mharradon/events{/privacy}","received_events_url":"https://api.github.com/users/mharradon/received_events","type":"User","site_admin":false},"labels":[{"id":192940625,"node_id":"MDU6TGFiZWwxOTI5NDA2MjU=","url":"https://api.github.com/repos/keras-team/keras/labels/Enhancement","name":"Enhancement","color":"84b6eb","default":false}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2018-11-05T06:40:20Z","updated_at":"2018-11-05T10:05:51Z","closed_at":null,"author_association":"NONE","body":"Please make sure that the boxes below are checked before you submit your issue.\r\nIf your issue is an **implementation question**, please ask your question on [StackOverflow](http://stackoverflow.com/questions/tagged/keras) or [on the Keras Slack channel](https://keras-slack-autojoin.herokuapp.com/) instead of opening a GitHub issue.\r\n\r\nThank you!\r\n\r\n- [x] Check that you are up-to-date with the master branch of Keras. You can update with:\r\n`pip install git+git://github.com/keras-team/keras.git --upgrade --no-deps`\r\n\r\n- [x] Check that your version of TensorFlow is up-to-date. The installation instructions can be found [here](https://www.tensorflow.org/get_started/os_setup).\r\n\r\n- [] Provide a link to a GitHub Gist of a Python script that can reproduce your issue (or just copy the script here if it is short).\r\n\r\nI'm training a model using fit_generator with multiprocessing. I've been working on optimizing training performance, and I've found via some profiling that roughly half of the time training is spent in various multiprocessing _recv methods (the other half waiting on tensorflow for GPU callbacks). It appears that significant time is spent transferring bytes and pickling/unpickling (I just benchmarked, and the pickling/unpickling is sufficient to explain the time).\r\n\r\nMy workload is moderately extreme in terms of batch size and number of GPUs, but I would imagine many people are unknowingly getting limited by this. I was curious if anyone had suggestions or if there was interest in contributions here - I'm considering implementing something to improve this situation, though I'm not sure what yet."},"comment":{"url":"https://api.github.com/repos/keras-team/keras/issues/comments/435819876","html_url":"https://github.com/keras-team/keras/issues/11578#issuecomment-435819876","issue_url":"https://api.github.com/repos/keras-team/keras/issues/11578","id":435819876,"node_id":"MDEyOklzc3VlQ29tbWVudDQzNTgxOTg3Ng==","user":{"login":"gabrieldemarmiesse","id":12891691,"node_id":"MDQ6VXNlcjEyODkxNjkx","avatar_url":"https://avatars1.githubusercontent.com/u/12891691?v=4","gravatar_id":"","url":"https://api.github.com/users/gabrieldemarmiesse","html_url":"https://github.com/gabrieldemarmiesse","followers_url":"https://api.github.com/users/gabrieldemarmiesse/followers","following_url":"https://api.github.com/users/gabrieldemarmiesse/following{/other_user}","gists_url":"https://api.github.com/users/gabrieldemarmiesse/gists{/gist_id}","starred_url":"https://api.github.com/users/gabrieldemarmiesse/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/gabrieldemarmiesse/subscriptions","organizations_url":"https://api.github.com/users/gabrieldemarmiesse/orgs","repos_url":"https://api.github.com/users/gabrieldemarmiesse/repos","events_url":"https://api.github.com/users/gabrieldemarmiesse/events{/privacy}","received_events_url":"https://api.github.com/users/gabrieldemarmiesse/received_events","type":"User","site_admin":false},"created_at":"2018-11-05T10:05:51Z","updated_at":"2018-11-05T10:05:51Z","author_association":"MEMBER","body":"I suppose that would be something to discuss with @Dref360 . It's not really my area of expertise."}}
{"actor":{"display_login":"gabrieldemarmiesse"}}