{"action":"created","issue":{"url":"https://api.github.com/repos/matterport/Mask_RCNN/issues/581","repository_url":"https://api.github.com/repos/matterport/Mask_RCNN","labels_url":"https://api.github.com/repos/matterport/Mask_RCNN/issues/581/labels{/name}","comments_url":"https://api.github.com/repos/matterport/Mask_RCNN/issues/581/comments","events_url":"https://api.github.com/repos/matterport/Mask_RCNN/issues/581/events","html_url":"https://github.com/matterport/Mask_RCNN/issues/581","id":324987587,"number":581,"title":"Trick to double the training speed of this repo / keras (improvement)","user":{"login":"zungam","id":11159388,"avatar_url":"https://avatars0.githubusercontent.com/u/11159388?v=4","gravatar_id":"","url":"https://api.github.com/users/zungam","html_url":"https://github.com/zungam","followers_url":"https://api.github.com/users/zungam/followers","following_url":"https://api.github.com/users/zungam/following{/other_user}","gists_url":"https://api.github.com/users/zungam/gists{/gist_id}","starred_url":"https://api.github.com/users/zungam/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/zungam/subscriptions","organizations_url":"https://api.github.com/users/zungam/orgs","repos_url":"https://api.github.com/users/zungam/repos","events_url":"https://api.github.com/users/zungam/events{/privacy}","received_events_url":"https://api.github.com/users/zungam/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":5,"created_at":"2018-05-21T16:57:03Z","updated_at":"2018-05-23T08:51:35Z","closed_at":null,"author_association":"NONE","body":"Hi, this is not a matterport issue, actually its an improvement to Keras so I will post this on Keras soon,\r\nbut it actually doubled the training speed of this repo in particular, so I will share it! I analyzed this repo for bottlenecks, and deduced the biggest bottleneck down to the `model.data_generator` which fetch batches ready on a seperate thread to improve speed. However, on my training job it didnt help much, because some batches took 0.1 seconds to load, and others 20 seconds. This means that if loading a batch takes longer than the GPU uses to train on the last batch, the GPU will halt to wait on a batch to load. \r\n\r\nTo make the GPU allways have an image ready to load into GPU memory without waiting time, I propose having a FIFO queue fetching batches non-stop. This means that while the GPU is busy doing what it does, the CPU will always work in the background to have images ready (forever). This litle trick doubled the training time of my network (which uses 704x704 images as input and between 5 and 2000 masks on each image).\r\n\r\n**How to:**  \r\nIn [keras/engine/training_generator.py](https://github.com/keras-team/keras/blob/master/keras/engine/training_generator.py), add this class:\r\n\r\n```\r\nimport threading\r\nclass ThreadingBatches(object):\r\n    \"\"\" Threading example class\r\n    The run() method will be started and it will run in the background\r\n    until the application exits.\r\n    \"\"\"\r\n    def __init__(self, generator=None):\r\n        \"\"\" Constructor\r\n        :type interval: int\r\n        :param interval: Check interval, in seconds\r\n        \"\"\"\r\n        self.generator = generator\r\n        self.stack = []\r\n        self.queue_size = 50 #This number should be experimented with based on how much RAM you have\r\n        thread = threading.Thread(target=self.run, args=())\r\n        thread.daemon = True                    # Daemonize thread\r\n        thread.start()                                  # Start the execution\r\n        \r\n        \r\n\r\n    def run(self):\r\n        \"\"\" Method that runs forever \"\"\"\r\n        while True:\r\n            # Do something\r\n            if(len(self.stack) < self.queue_size):\r\n                try:\r\n                    batch = next(self.generator)\r\n                    self.stack.append(batch)\r\n                except ValueError:\r\n                    None\r\n    def popNextBatch(self):\r\n        while len(self.stack) == 0:\r\n            None\r\n\r\n        return self.stack.pop(0)\r\n```\r\n\r\nNow, add this below the line 145:\r\n`batch_thread = ThreadingBatches(generator=output_generator) `\r\nand overwrite the line 155 from:\r\n`generator_output = next(output_generator)`\r\nto\r\n`generator_output = batch_thread.popNextBatch()`\r\n\r\nAnd voala, superfast training! :)"},"comment":{"url":"https://api.github.com/repos/matterport/Mask_RCNN/issues/comments/391271515","html_url":"https://github.com/matterport/Mask_RCNN/issues/581#issuecomment-391271515","issue_url":"https://api.github.com/repos/matterport/Mask_RCNN/issues/581","id":391271515,"user":{"login":"ziyigogogo","id":9452949,"avatar_url":"https://avatars2.githubusercontent.com/u/9452949?v=4","gravatar_id":"","url":"https://api.github.com/users/ziyigogogo","html_url":"https://github.com/ziyigogogo","followers_url":"https://api.github.com/users/ziyigogogo/followers","following_url":"https://api.github.com/users/ziyigogogo/following{/other_user}","gists_url":"https://api.github.com/users/ziyigogogo/gists{/gist_id}","starred_url":"https://api.github.com/users/ziyigogogo/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ziyigogogo/subscriptions","organizations_url":"https://api.github.com/users/ziyigogogo/orgs","repos_url":"https://api.github.com/users/ziyigogogo/repos","events_url":"https://api.github.com/users/ziyigogogo/events{/privacy}","received_events_url":"https://api.github.com/users/ziyigogogo/received_events","type":"User","site_admin":false},"created_at":"2018-05-23T08:51:35Z","updated_at":"2018-05-23T08:51:35Z","author_association":"NONE","body":"@benqua \r\nmaybe, i could only say, since my cpu usage is not that high either. Did you try this PR and gain a significant improve?\r\nAnd if so, would please share you hardware condition for reference? Thank you so much\r\n\r\n@cklat \r\nI believe that **waleedka** is currently on his vocation or something since his github activity stopped from the end of april."}}
{"actor":{"display_login":"ziyigogogo"}}