{"action":"created","issue":{"url":"https://api.github.com/repos/lengstrom/fast-style-transfer/issues/22","repository_url":"https://api.github.com/repos/lengstrom/fast-style-transfer","labels_url":"https://api.github.com/repos/lengstrom/fast-style-transfer/issues/22/labels{/name}","comments_url":"https://api.github.com/repos/lengstrom/fast-style-transfer/issues/22/comments","events_url":"https://api.github.com/repos/lengstrom/fast-style-transfer/issues/22/events","html_url":"https://github.com/lengstrom/fast-style-transfer/issues/22","id":187584162,"number":22,"title":"Training seems to be very slow on Tesla K40","user":{"login":"michaelhuang74","id":23154573,"avatar_url":"https://avatars.githubusercontent.com/u/23154573?v=3","gravatar_id":"","url":"https://api.github.com/users/michaelhuang74","html_url":"https://github.com/michaelhuang74","followers_url":"https://api.github.com/users/michaelhuang74/followers","following_url":"https://api.github.com/users/michaelhuang74/following{/other_user}","gists_url":"https://api.github.com/users/michaelhuang74/gists{/gist_id}","starred_url":"https://api.github.com/users/michaelhuang74/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/michaelhuang74/subscriptions","organizations_url":"https://api.github.com/users/michaelhuang74/orgs","repos_url":"https://api.github.com/users/michaelhuang74/repos","events_url":"https://api.github.com/users/michaelhuang74/events{/privacy}","received_events_url":"https://api.github.com/users/michaelhuang74/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":1,"created_at":"2016-11-06T19:50:14Z","updated_at":"2016-11-06T19:55:25Z","closed_at":null,"body":"I am trying to generate a model file using the following,\r\n\r\npython style.py --style  examples/style/la_muse.jpg --checkpoint-dir checkpoint/check2  --content-weight 1.5e1 --checkpoint-iterations 1 --batch-size 20\r\n\r\n**At very beginning of the training process, it took about 20 seconds to finish one iteration. However, as the training process goes on, the time of one iteration becomes much longer. Currently the training process has been going on for more than 14 hours and only 621 iterations have finished.** \r\n\r\nThe followings are the screen output during the training process.\r\n\r\nI tensorflow/stream_executor/dso_loader.cc:111] successfully opened CUDA library libcublas.so locally\r\nI tensorflow/stream_executor/dso_loader.cc:111] successfully opened CUDA library libcudnn.so locally\r\nI tensorflow/stream_executor/dso_loader.cc:111] successfully opened CUDA library libcufft.so locally\r\nI tensorflow/stream_executor/dso_loader.cc:111] successfully opened CUDA library libcuda.so.1 locally\r\nI tensorflow/stream_executor/dso_loader.cc:111] successfully opened CUDA library libcurand.so locally\r\nTrain set has been trimmed slightly..\r\n(1, 512, 512, 3)\r\n\r\nI tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:925] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\r\nI tensorflow/core/common_runtime/gpu/gpu_device.cc:951] Found device 0 with properties: \r\nname: Tesla K40c\r\nmajor: 3 minor: 5 memoryClockRate (GHz) 0.745\r\npciBusID 0000:01:00.0\r\nTotal memory: 11.17GiB\r\nFree memory: 11.10GiB\r\nW tensorflow/stream_executor/cuda/cuda_driver.cc:572] creating context when one is currently active; existing: 0x2a55840\r\nI tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:925] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\r\nI tensorflow/core/common_runtime/gpu/gpu_device.cc:951] Found device 1 with properties: \r\nname: Tesla K40c\r\nmajor: 3 minor: 5 memoryClockRate (GHz) 0.745\r\npciBusID 0000:02:00.0\r\nTotal memory: 11.17GiB\r\nFree memory: 11.10GiB\r\nI tensorflow/core/common_runtime/gpu/gpu_device.cc:972] DMA: 0 1 \r\nI tensorflow/core/common_runtime/gpu/gpu_device.cc:982] 0:   Y Y \r\nI tensorflow/core/common_runtime/gpu/gpu_device.cc:982] 1:   Y Y \r\nI tensorflow/core/common_runtime/gpu/gpu_device.cc:1041] Creating TensorFlow device (/gpu:0) -> (device: 0, name: Tesla K40c, pci bus id: 0000:01:00.0)\r\nI tensorflow/core/common_runtime/gpu/gpu_device.cc:1041] Creating TensorFlow device (/gpu:1) -> (device: 1, name: Tesla K40c, pci bus id: 0000:02:00.0)\r\nI tensorflow/core/common_runtime/gpu/gpu_device.cc:1041] Creating TensorFlow device (/gpu:0) -> (device: 0, name: Tesla K40c, pci bus id: 0000:01:00.0)\r\nI tensorflow/core/common_runtime/gpu/gpu_device.cc:1041] Creating TensorFlow device (/gpu:1) -> (device: 1, name: Tesla K40c, pci bus id: 0000:02:00.0)\r\nUID: 11\r\nEpoch 0, Iteration: 1, Loss: 1.60785e+08\r\nstyle: 1.4523e+08, content:8.83568e+06, tv: 6.71974e+06\r\nEpoch 0, Iteration: 2, Loss: 1.08762e+08\r\nstyle: 9.43464e+07, content:7.88181e+06, tv: 6.5336e+06\r\n.............................................\r\nEpoch 0, Iteration: 620, Loss: 1.08479e+07\r\nstyle: 3.26121e+06, content:6.84982e+06, tv: 736917.0\r\nEpoch 0, Iteration: 621, Loss: 1.12474e+07\r\nstyle: 3.23448e+06, content:7.27868e+06, tv: 734276.0\r\n\r\n**Only 621 iteratons have finished after more than 14 hours. Currently, one iteration takes a couple of minutes. \r\n\r\nAny idea why the training process is so slow on Tealsa K40?\r\nIn addition, how can I adjust the parameters (e.g., batch-size) in order to speedup the training process without compromising the quality of the model file too much? Thanks.**  \r\n"},"comment":{"url":"https://api.github.com/repos/lengstrom/fast-style-transfer/issues/comments/258705945","html_url":"https://github.com/lengstrom/fast-style-transfer/issues/22#issuecomment-258705945","issue_url":"https://api.github.com/repos/lengstrom/fast-style-transfer/issues/22","id":258705945,"user":{"login":"lengstrom","id":760865,"avatar_url":"https://avatars.githubusercontent.com/u/760865?v=3","gravatar_id":"","url":"https://api.github.com/users/lengstrom","html_url":"https://github.com/lengstrom","followers_url":"https://api.github.com/users/lengstrom/followers","following_url":"https://api.github.com/users/lengstrom/following{/other_user}","gists_url":"https://api.github.com/users/lengstrom/gists{/gist_id}","starred_url":"https://api.github.com/users/lengstrom/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/lengstrom/subscriptions","organizations_url":"https://api.github.com/users/lengstrom/orgs","repos_url":"https://api.github.com/users/lengstrom/repos","events_url":"https://api.github.com/users/lengstrom/events{/privacy}","received_events_url":"https://api.github.com/users/lengstrom/received_events","type":"User","site_admin":false},"created_at":"2016-11-06T19:55:25Z","updated_at":"2016-11-06T19:55:25Z","body":"`checkpoint_iterations=1` forces your program to save a checkpoint every iteration, so you should make that parameter larger if you want to train a reasonable amount of time. "}}
{"actor":{"display_login":"lengstrom"}}