{"action":"created","issue":{"url":"https://api.github.com/repos/uber/horovod/issues/602","repository_url":"https://api.github.com/repos/uber/horovod","labels_url":"https://api.github.com/repos/uber/horovod/issues/602/labels{/name}","comments_url":"https://api.github.com/repos/uber/horovod/issues/602/comments","events_url":"https://api.github.com/repos/uber/horovod/issues/602/events","html_url":"https://github.com/uber/horovod/issues/602","id":375612576,"node_id":"MDU6SXNzdWUzNzU2MTI1NzY=","number":602,"title":"GPU Memory leak when XLA is activated","user":{"login":"DEKHTIARJonathan","id":10923599,"node_id":"MDQ6VXNlcjEwOTIzNTk5","avatar_url":"https://avatars2.githubusercontent.com/u/10923599?v=4","gravatar_id":"","url":"https://api.github.com/users/DEKHTIARJonathan","html_url":"https://github.com/DEKHTIARJonathan","followers_url":"https://api.github.com/users/DEKHTIARJonathan/followers","following_url":"https://api.github.com/users/DEKHTIARJonathan/following{/other_user}","gists_url":"https://api.github.com/users/DEKHTIARJonathan/gists{/gist_id}","starred_url":"https://api.github.com/users/DEKHTIARJonathan/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/DEKHTIARJonathan/subscriptions","organizations_url":"https://api.github.com/users/DEKHTIARJonathan/orgs","repos_url":"https://api.github.com/users/DEKHTIARJonathan/repos","events_url":"https://api.github.com/users/DEKHTIARJonathan/events{/privacy}","received_events_url":"https://api.github.com/users/DEKHTIARJonathan/received_events","type":"User","site_admin":false},"labels":[{"id":665452437,"node_id":"MDU6TGFiZWw2NjU0NTI0Mzc=","url":"https://api.github.com/repos/uber/horovod/labels/question","name":"question","color":"cc317c","default":true}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":6,"created_at":"2018-10-30T17:39:53Z","updated_at":"2018-11-29T23:35:11Z","closed_at":null,"author_association":"NONE","body":"Hi everyone,\r\n\r\nI am working using Horovod with various models using XLA. We have identified a problem when Horovod is used with XLA activated.\r\n\r\nTo keep it short, if you enable XLA, each Horovod process will allocate a little memory on each GPU.\r\nWhen using DGX1 (8 GPUs) or even worse DGX2 (16 GPUs) this becomes a **huge** issue.\r\n\r\nEach horovod thread takes between 250-500MB of VRAM on all GPUs + Network Memory (this time only on the correct GPU).\r\n\r\nI have been able to reproduce this issue with:\r\n- TF: 1.8 1.9 1.10 1.11 and 1.12rc2\r\n- Horovod 0.12 0.13 0.14 and 0.15\r\n\r\nHere is the command I use to launch the code on an 8 GPUs machine (DGX1):\r\n```bash\r\n#!/usr/bin/env bash\r\n\r\nmpirun \\\r\n    -np 8 \\\r\n    -H localhost:8 \\\r\n    -bind-to none \\\r\n    -map-by slot \\\r\n    -x NCCL_DEBUG=INFO \\\r\n    -x LD_LIBRARY_PATH \\\r\n    -x PATH \\\r\n    -mca pml ob1 -mca btl ^openib \\\r\n    --allow-run-as-root \\\r\n    python horovod_issue.py\r\n```\r\n\r\nAnd the minimalistic python file `horovod_issue.py` to reproduce:\r\n\r\n```python\r\nimport time\r\n\r\nimport tensorflow as tf\r\nimport horovod.tensorflow as hvd\r\n\r\nif __name__ == '__main__':\r\n\r\n    USE_XLA = True\r\n\r\n    # Horovod: initialize Horovod.\r\n    hvd.init()\r\n\r\n    input_plh = tf.placeholder(tf.float32, [100, 32, 32, 3])\r\n\r\n    net = tf.layers.conv2d(\r\n        inputs=input_plh, filters=32, kernel_size=[5, 5], padding=\"same\", activation=tf.nn.relu\r\n    )\r\n\r\n    def configure_tf_session(use_xla=False):\r\n\r\n        config = tf.ConfigProto(\r\n            allow_soft_placement=False,\r\n            log_device_placement=False\r\n        )\r\n\r\n        config.gpu_options.allow_growth = True\r\n        config.gpu_options.visible_device_list = str(hvd.local_rank())\r\n\r\n        # Turns on XLA JIT compilation.\r\n        if use_xla:\r\n            jit_level = tf.OptimizerOptions.ON_1\r\n            print(\"XLA is ACTIVATED\")\r\n        else:\r\n            jit_level = tf.OptimizerOptions.OFF\r\n            print(\"XLA is DEACTIVATED\")\r\n\r\n        config.graph_options.optimizer_options.global_jit_level = jit_level\r\n\r\n        # config.intra_op_parallelism_threads = 1  # Setting this to `1` does not solve the issue\r\n        # config.inter_op_parallelism_threads = 1  # Setting this to `1` does not solve the issue\r\n\r\n        return config\r\n\r\n    config = configure_tf_session(use_xla=USE_XLA)\r\n\r\n    with tf.Session(config=config) as sess:\r\n\r\n        sess.run(tf.global_variables_initializer())\r\n\r\n        print(\"Sleeping for 30 sec ... Check `nvidia-smi` now ...\")\r\n        time.sleep(30)\r\n```\r\n\r\nDuring the sleep, launch `nvidia-smi` and look at the GPU processes section.\r\n\r\n### Without XLA enabled - No Issue\r\n\r\n![horovod_without_xla_issue](https://user-images.githubusercontent.com/10923599/47738248-6d70d780-dc7b-11e8-96cc-7493491b8671.png)\r\n\r\n### With XLA enabled - 1 Process per Horovod Thread per GPU\r\n\r\n![horovod_xla_issue](https://user-images.githubusercontent.com/10923599/47738276-7f527a80-dc7b-11e8-861d-d6370ea8d105.png)\r\n\r\n*Disclaimer: The image has be truncated*\r\n\r\nI hope this will be well enough explained.\r\n\r\nIf you need any more information, I'll be glad to help,\r\n\r\nThanks for the help\r\n"},"comment":{"url":"https://api.github.com/repos/uber/horovod/issues/comments/443034134","html_url":"https://github.com/uber/horovod/issues/602#issuecomment-443034134","issue_url":"https://api.github.com/repos/uber/horovod/issues/602","id":443034134,"node_id":"MDEyOklzc3VlQ29tbWVudDQ0MzAzNDEzNA==","user":{"login":"tgaddair","id":1742912,"node_id":"MDQ6VXNlcjE3NDI5MTI=","avatar_url":"https://avatars1.githubusercontent.com/u/1742912?v=4","gravatar_id":"","url":"https://api.github.com/users/tgaddair","html_url":"https://github.com/tgaddair","followers_url":"https://api.github.com/users/tgaddair/followers","following_url":"https://api.github.com/users/tgaddair/following{/other_user}","gists_url":"https://api.github.com/users/tgaddair/gists{/gist_id}","starred_url":"https://api.github.com/users/tgaddair/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tgaddair/subscriptions","organizations_url":"https://api.github.com/users/tgaddair/orgs","repos_url":"https://api.github.com/users/tgaddair/repos","events_url":"https://api.github.com/users/tgaddair/events{/privacy}","received_events_url":"https://api.github.com/users/tgaddair/received_events","type":"User","site_admin":false},"created_at":"2018-11-29T23:35:11Z","updated_at":"2018-11-29T23:35:11Z","author_association":"COLLABORATOR","body":"Hey @DEKHTIARJonathan, I believe I've been able to reproduce the memory leak, but no progress on fixing it on our end as of yet.  Sounds like this may be a TensorFlow issue according to your other issue."}}
{"actor":{"display_login":"tgaddair"}}