{"action":"created","issue":{"url":"https://api.github.com/repos/tensorflow/tensorflow/issues/30321","repository_url":"https://api.github.com/repos/tensorflow/tensorflow","labels_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/30321/labels{/name}","comments_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/30321/comments","events_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/30321/events","html_url":"https://github.com/tensorflow/tensorflow/issues/30321","id":463372796,"node_id":"MDU6SXNzdWU0NjMzNzI3OTY=","number":30321,"title":"tensorflow 2.0 keras multi_gpu_model only utilizing one GPU","user":{"login":"mketcha","id":10420606,"node_id":"MDQ6VXNlcjEwNDIwNjA2","avatar_url":"https://avatars1.githubusercontent.com/u/10420606?v=4","gravatar_id":"","url":"https://api.github.com/users/mketcha","html_url":"https://github.com/mketcha","followers_url":"https://api.github.com/users/mketcha/followers","following_url":"https://api.github.com/users/mketcha/following{/other_user}","gists_url":"https://api.github.com/users/mketcha/gists{/gist_id}","starred_url":"https://api.github.com/users/mketcha/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mketcha/subscriptions","organizations_url":"https://api.github.com/users/mketcha/orgs","repos_url":"https://api.github.com/users/mketcha/repos","events_url":"https://api.github.com/users/mketcha/events{/privacy}","received_events_url":"https://api.github.com/users/mketcha/received_events","type":"User","site_admin":false},"labels":[{"id":1392900578,"node_id":"MDU6TGFiZWwxMzkyOTAwNTc4","url":"https://api.github.com/repos/tensorflow/tensorflow/labels/2.0.0-beta0","name":"2.0.0-beta0","color":"d93f0b","default":false},{"id":1097546578,"node_id":"MDU6TGFiZWwxMDk3NTQ2NTc4","url":"https://api.github.com/repos/tensorflow/tensorflow/labels/comp:keras","name":"comp:keras","color":"0052cc","default":false},{"id":473172988,"node_id":"MDU6TGFiZWw0NzMxNzI5ODg=","url":"https://api.github.com/repos/tensorflow/tensorflow/labels/type:bug/performance","name":"type:bug/performance","color":"159b2e","default":false}],"state":"open","locked":false,"assignee":{"login":"ymodak","id":42785357,"node_id":"MDQ6VXNlcjQyNzg1MzU3","avatar_url":"https://avatars1.githubusercontent.com/u/42785357?v=4","gravatar_id":"","url":"https://api.github.com/users/ymodak","html_url":"https://github.com/ymodak","followers_url":"https://api.github.com/users/ymodak/followers","following_url":"https://api.github.com/users/ymodak/following{/other_user}","gists_url":"https://api.github.com/users/ymodak/gists{/gist_id}","starred_url":"https://api.github.com/users/ymodak/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ymodak/subscriptions","organizations_url":"https://api.github.com/users/ymodak/orgs","repos_url":"https://api.github.com/users/ymodak/repos","events_url":"https://api.github.com/users/ymodak/events{/privacy}","received_events_url":"https://api.github.com/users/ymodak/received_events","type":"User","site_admin":false},"assignees":[{"login":"ymodak","id":42785357,"node_id":"MDQ6VXNlcjQyNzg1MzU3","avatar_url":"https://avatars1.githubusercontent.com/u/42785357?v=4","gravatar_id":"","url":"https://api.github.com/users/ymodak","html_url":"https://github.com/ymodak","followers_url":"https://api.github.com/users/ymodak/followers","following_url":"https://api.github.com/users/ymodak/following{/other_user}","gists_url":"https://api.github.com/users/ymodak/gists{/gist_id}","starred_url":"https://api.github.com/users/ymodak/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ymodak/subscriptions","organizations_url":"https://api.github.com/users/ymodak/orgs","repos_url":"https://api.github.com/users/ymodak/repos","events_url":"https://api.github.com/users/ymodak/events{/privacy}","received_events_url":"https://api.github.com/users/ymodak/received_events","type":"User","site_admin":false}],"milestone":null,"comments":2,"created_at":"2019-07-02T18:25:44Z","updated_at":"2019-07-10T05:09:43Z","closed_at":null,"author_association":"NONE","body":"**System information**\r\n- Have I written custom code (as opposed to using a stock example script provided in TensorFlow): No\r\n- OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Windows 10\r\n- Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue happens on mobile device: N/A\r\n- TensorFlow installed from (source or binary): Binary, pip install\r\n- TensorFlow version (use command below): tensorflow-gpu==2.0.0-beta1\r\n- Python version: 3.6\r\n- Bazel version (if compiling from source): N/A\r\n- GCC/Compiler version (if compiling from source): N/A\r\n- CUDA/cuDNN version: cudatoolkit 10.0.130, cudnn 7.6.0\r\n- GPU model and memory: 4x NVIDIA GeForce GTX 1080 Ti\r\n\r\n**Describe the current behavior**\r\nWhen using multi_gpu_model (i.e., tf.keras.utils.multi_gpu_model) in tensorflow 2.0 to distribute a job across multiple gpus (4), only one gpu appears to be used. That is when monitoring the GPU usage only one GPU shows substantial dedicated GPU memory usage and GPU utility. \r\n\r\n**Describe the expected behavior**\r\nEach of the 4 GPUs should indicate that memory is being copied to the device and processed.\r\n\r\n**Code to reproduce the issue**\r\nWhile my issue arises with custom code using model.fit_generator, I was able to replicate the issue using model.fit with documentation code provided at https://www.tensorflow.org/versions/r2.0/api_docs/python/tf/keras/utils/multi_gpu_model\r\n\r\n```\r\nimport tensorflow as tf\r\nfrom tensorflow import keras\r\nfrom tensorflow.keras.applications import Xception\r\nfrom tensorflow.keras.utils import multi_gpu_model\r\nimport numpy as np\r\n\r\nnum_samples = 1000\r\nheight = 224\r\nwidth = 224\r\nnum_classes = 1000\r\n\r\n# Instantiate the base model (or \"template\" model).\r\n# We recommend doing this with under a CPU device scope,\r\n# so that the model's weights are hosted on CPU memory.\r\n# Otherwise they may end up hosted on a GPU, which would\r\n# complicate weight sharing.\r\nwith tf.device('/cpu:0'):\r\n    model = Xception(weights=None,\r\n                     input_shape=(height, width, 3),\r\n                     classes=num_classes)\r\n\r\n# Replicates the model on 8 GPUs.\r\n# This assumes that your machine has 8 available GPUs.\r\nparallel_model = multi_gpu_model(model, gpus=4) # gpus changed to 4\r\nparallel_model.compile(loss='categorical_crossentropy', optimizer='rmsprop')\r\n\r\n# Generate dummy data.\r\nx = np.random.random((num_samples, height, width, 3))\r\ny = np.random.random((num_samples, num_classes))\r\n\r\nparallel_model.summary()\r\n# This `fit` call will be distributed on 8 GPUs.\r\n# Since the batch size is 256, each GPU will process 32 samples.\r\nparallel_model.fit(x, y, epochs=20, batch_size=16) #batch_sized changed to 16\r\n```\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/tensorflow/tensorflow/issues/comments/509910250","html_url":"https://github.com/tensorflow/tensorflow/issues/30321#issuecomment-509910250","issue_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/30321","id":509910250,"node_id":"MDEyOklzc3VlQ29tbWVudDUwOTkxMDI1MA==","user":{"login":"IuriiZhakun","id":517387,"node_id":"MDQ6VXNlcjUxNzM4Nw==","avatar_url":"https://avatars1.githubusercontent.com/u/517387?v=4","gravatar_id":"","url":"https://api.github.com/users/IuriiZhakun","html_url":"https://github.com/IuriiZhakun","followers_url":"https://api.github.com/users/IuriiZhakun/followers","following_url":"https://api.github.com/users/IuriiZhakun/following{/other_user}","gists_url":"https://api.github.com/users/IuriiZhakun/gists{/gist_id}","starred_url":"https://api.github.com/users/IuriiZhakun/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/IuriiZhakun/subscriptions","organizations_url":"https://api.github.com/users/IuriiZhakun/orgs","repos_url":"https://api.github.com/users/IuriiZhakun/repos","events_url":"https://api.github.com/users/IuriiZhakun/events{/privacy}","received_events_url":"https://api.github.com/users/IuriiZhakun/received_events","type":"User","site_admin":false},"created_at":"2019-07-10T05:09:43Z","updated_at":"2019-07-10T05:09:43Z","author_association":"NONE","body":"Any plans or advise how to use multiple gpus with fit_generator method?\r\n\r\ntensorflow/python/keras/engine/training.py\", line 1157, in fit_generator\r\n    raise NotImplementedError('`fit_generator` is not supported for '\r\nNotImplementedError: `fit_generator` is not supported for models compiled with tf.distribute.Strategy.\r\n"}}
{"actor":{"display_login":"IuriiZhakun"}}