{"action":"created","issue":{"url":"https://api.github.com/repos/Theano/Theano/issues/4740","repository_url":"https://api.github.com/repos/Theano/Theano","labels_url":"https://api.github.com/repos/Theano/Theano/issues/4740/labels{/name}","comments_url":"https://api.github.com/repos/Theano/Theano/issues/4740/comments","events_url":"https://api.github.com/repos/Theano/Theano/issues/4740/events","html_url":"https://github.com/Theano/Theano/issues/4740","id":165622038,"number":4740,"title":"Theano and other GPU app conflict","user":{"login":"eduardohenriquearnold","id":4097245,"avatar_url":"https://avatars.githubusercontent.com/u/4097245?v=3","gravatar_id":"","url":"https://api.github.com/users/eduardohenriquearnold","html_url":"https://github.com/eduardohenriquearnold","followers_url":"https://api.github.com/users/eduardohenriquearnold/followers","following_url":"https://api.github.com/users/eduardohenriquearnold/following{/other_user}","gists_url":"https://api.github.com/users/eduardohenriquearnold/gists{/gist_id}","starred_url":"https://api.github.com/users/eduardohenriquearnold/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/eduardohenriquearnold/subscriptions","organizations_url":"https://api.github.com/users/eduardohenriquearnold/orgs","repos_url":"https://api.github.com/users/eduardohenriquearnold/repos","events_url":"https://api.github.com/users/eduardohenriquearnold/events{/privacy}","received_events_url":"https://api.github.com/users/eduardohenriquearnold/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":3,"created_at":"2016-07-14T18:01:55Z","updated_at":"2016-07-14T19:03:07Z","closed_at":null,"body":"I am working on a computer vision application developed with OpenCV in C++, we use a camera that comes with a third-party library which uses CUDA for image processing. I also use Keras for Deep Learning with Theano as backend. I am calling all Keras functionality from C++ through the Python C++ API.\r\n\r\nThe problem is that whenever I am using both the Camera and the Keras procedures, I receive the following errors:\r\n```\r\nUsing Theano backend.\r\nUsing gpu device 0: NVS 510 (CNMeM is disabled, cuDNN not available)\r\nTraceback (most recent call last):\r\n  File \"./DeepClassifier.py\", line 107, in classify\r\n    res = model.predict(im)\r\n  File \"/usr/local/lib/python2.7/dist-packages/keras/models.py\", line 464, in predict\r\n    return self.model.predict(x, batch_size=batch_size, verbose=verbose)\r\n  File \"/usr/local/lib/python2.7/dist-packages/keras/engine/training.py\", line 1155, in predict\r\n    batch_size=batch_size, verbose=verbose)\r\n  File \"/usr/local/lib/python2.7/dist-packages/keras/engine/training.py\", line 855, in _predict_loop\r\n    batch_outs = f(ins_batch)\r\n  File \"/usr/local/lib/python2.7/dist-packages/keras/backend/theano_backend.py\", line 531, in __call__\r\n    return self.function(*inputs)\r\n  File \"/usr/local/lib/python2.7/dist-packages/theano/compile/function_module.py\", line 871, in __call__\r\n    storage_map=getattr(self.fn, 'storage_map', None))\r\n  File \"/usr/local/lib/python2.7/dist-packages/theano/gof/link.py\", line 314, in raise_with_op\r\n    reraise(exc_type, exc_value, exc_trace)\r\n  File \"/usr/local/lib/python2.7/dist-packages/theano/compile/function_module.py\", line 859, in __call__\r\n    outputs = self.fn()\r\nRuntimeError: cublasSgemm failed (14) an internal operation failed\r\n unit=0 N=0, c.dims=[1 512], a.dim=[1 3600], alpha=%f, beta=%f, a=%p, b=%p, c=%p sa_0=%d, sa_1=%d, sb_0=%d, sb_1=%d, sc_0=%d, sc_1=%d\r\nApply node that caused the error: GpuDot22(GpuElemwise{Composite{(i0 * (i1 + Abs(i1)))}}[(0, 1)].0, dense_1_W)\r\nToposort index: 14\r\nInputs types: [CudaNdarrayType(float32, matrix), CudaNdarrayType(float32, matrix)]\r\nInputs shapes: [(1, 3600), (3600, 512)]\r\nInputs strides: [(0, 1), (512, 1)]\r\nInputs values: ['not shown', 'not shown']\r\nOutputs clients: [[GpuElemwise{Composite{(i0 * ((i1 + i2) + Abs((i1 + i2))))}}[(0, 1)](CudaNdarrayConstant{[[ 0.5]]}, GpuDot22.0, GpuDimShuffle{x,0}.0)]]\r\n\r\nDebugprint of the apply node: \r\nGpuDot22 [id A] <CudaNdarrayType(float32, matrix)> ''   \r\n |GpuElemwise{Composite{(i0 * (i1 + Abs(i1)))}}[(0, 1)] [id B] <CudaNdarrayType(float32, matrix)> ''   \r\n | |CudaNdarrayConstant{[[ 0.5]]} [id C] <CudaNdarrayType(float32, (True, True))>\r\n | |GpuReshape{2} [id D] <CudaNdarrayType(float32, matrix)> ''   \r\n |   |GpuFromHost [id E] <CudaNdarrayType(float32, 4D)> ''   \r\n |   | |flatten_input_1 [id F] <TensorType(float32, 4D)>\r\n |   |MakeVector{dtype='int64'} [id G] <TensorType(int64, vector)> ''   \r\n |     |Shape_i{0} [id H] <TensorType(int64, scalar)> ''   \r\n |     | |flatten_input_1 [id F] <TensorType(float32, 4D)>\r\n |     |Elemwise{IntDiv}[(0, 0)] [id I] <TensorType(int64, scalar)> ''   \r\n |       |Prod{acc_dtype=int64} [id J] <TensorType(int64, scalar)> ''   \r\n |       | |MakeVector{dtype='int64'} [id K] <TensorType(int64, vector)> ''   \r\n |       |   |Shape_i{0} [id H] <TensorType(int64, scalar)> ''   \r\n |       |   |Shape_i{1} [id L] <TensorType(int64, scalar)> ''   \r\n |       |   | |flatten_input_1 [id F] <TensorType(float32, 4D)>\r\n |       |   |Shape_i{2} [id M] <TensorType(int64, scalar)> ''   \r\n |       |   | |flatten_input_1 [id F] <TensorType(float32, 4D)>\r\n |       |   |Shape_i{3} [id N] <TensorType(int64, scalar)> ''   \r\n |       |     |flatten_input_1 [id F] <TensorType(float32, 4D)>\r\n |       |Shape_i{0} [id H] <TensorType(int64, scalar)> ''   \r\n |dense_1_W [id O] <CudaNdarrayType(float32, matrix)>\r\n\r\nStorage map footprint:\r\n - dense_1_W, Shared Input, Shape: (3600, 512), ElemSize: 4 Byte(s), TotalSize: 7372800 Byte(s)\r\n - dense_2_W, Shared Input, Shape: (512, 256), ElemSize: 4 Byte(s), TotalSize: 524288 Byte(s)\r\n - GpuElemwise{Composite{(i0 * (i1 + Abs(i1)))}}[(0, 1)].0, Shape: (1, 3600), ElemSize: 4 Byte(s), TotalSize: 14400 Byte(s)\r\n - flatten_input_1, Input, Shape: (1, 1, 60, 60), ElemSize: 4 Byte(s), TotalSize: 14400 Byte(s)\r\n - dense_1_b, Shared Input, Shape: (512,), ElemSize: 4 Byte(s), TotalSize: 2048 Byte(s)\r\n - dense_2_b, Shared Input, Shape: (256,), ElemSize: 4 Byte(s), TotalSize: 1024 Byte(s)\r\n - dense_3_W, Shared Input, Shape: (256, 1), ElemSize: 4 Byte(s), TotalSize: 1024 Byte(s)\r\n - dense_3_b, Shared Input, Shape: (1,), ElemSize: 4 Byte(s), TotalSize: 4 Byte(s)\r\n - CudaNdarrayConstant{[[ 0.5]]}, Shape: (1, 1), ElemSize: 4 Byte(s), TotalSize: 4 Byte(s)\r\n TotalSize: 7929992 Byte(s) 0.007 GB\r\n TotalSize inputs: 7915592 Byte(s) 0.007 GB\r\n```\r\nIf am using Keras alone (without initializing the camera object), no errors occur. Also, if I disable Theano GPU support, it all runs fine, but with significant performance loss.\r\n\r\nI have also tried running two process in parallel (one which uses the camera and other that uses Keras) and both work just fine.\r\n\r\nGiven these information, my hypothesis is that when calling the Python functionality from C++ I am running everything under the same PID, so there is no distinction of CUDA contexts between Theano objects and the camera 3rd party algorithm. \r\n\r\nDoes anyone have a suggestion on how to proceed here?"},"comment":{"url":"https://api.github.com/repos/Theano/Theano/issues/comments/232761026","html_url":"https://github.com/Theano/Theano/issues/4740#issuecomment-232761026","issue_url":"https://api.github.com/repos/Theano/Theano/issues/4740","id":232761026,"user":{"login":"nouiz","id":180987,"avatar_url":"https://avatars.githubusercontent.com/u/180987?v=3","gravatar_id":"","url":"https://api.github.com/users/nouiz","html_url":"https://github.com/nouiz","followers_url":"https://api.github.com/users/nouiz/followers","following_url":"https://api.github.com/users/nouiz/following{/other_user}","gists_url":"https://api.github.com/users/nouiz/gists{/gist_id}","starred_url":"https://api.github.com/users/nouiz/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/nouiz/subscriptions","organizations_url":"https://api.github.com/users/nouiz/orgs","repos_url":"https://api.github.com/users/nouiz/repos","events_url":"https://api.github.com/users/nouiz/events{/privacy}","received_events_url":"https://api.github.com/users/nouiz/received_events","type":"User","site_admin":false},"created_at":"2016-07-14T19:03:06Z","updated_at":"2016-07-14T19:03:06Z","body":"Did you try to init first Theano, then init the 3rd party and the reverse?\nMaybe one order work.\n\nFor context, how it work is that it is a stack of context. So you can load\none software, pop the context, load the other software and set the good\ncontext before calling it.\n\nI didn't try it, but this is how normal multi-context work for Theano\nmulti-GPU in the same process work.\n\nCUDA have function to pop and push the context on the stack.\n\nOn Thu, Jul 14, 2016 at 2:29 PM, Eduardo Henrique Arnold <\nnotifications@github.com> wrote:\n\n> Thanks for the quick reply. Unfortunately it is a 3rd party library with\n> closed source, so I have no way of knowing how they initialize the GPU, but\n> I have access to the following function CUcontext getCUDAContext()\n>\n> I believe that the *cublasSgemm* error is caused because both this 3rd\n> party library and Theano are sharing the GPU within the same context, since\n> they run under the same process. So I would like to force them to use\n> different contexts.\n>\n> I am not sure I can modify the 3rd party CUDA context, though. Do you know\n> a way of ensuring Theano has a different CUDA context?\n>\n> —\n> You are receiving this because you commented.\n> Reply to this email directly, view it on GitHub\n> <https://github.com/Theano/Theano/issues/4740#issuecomment-232751366>, or mute\n> the thread\n> <https://github.com/notifications/unsubscribe/AALC-1M0Tu-FWqTdLgyu0TMUQRmPWtudks5qVn_ygaJpZM4JMsWI>\n> .\n>\n"}}
{"actor":{"display_login":"nouiz"}}