{"action":"created","issue":{"url":"https://api.github.com/repos/tensorflow/tensorflow/issues/21338","repository_url":"https://api.github.com/repos/tensorflow/tensorflow","labels_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/21338/labels{/name}","comments_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/21338/comments","events_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/21338/events","html_url":"https://github.com/tensorflow/tensorflow/issues/21338","id":346963930,"node_id":"MDU6SXNzdWUzNDY5NjM5MzA=","number":21338,"title":"Max pooling cause error on empty batch","user":{"login":"wrongtest","id":7600935,"node_id":"MDQ6VXNlcjc2MDA5MzU=","avatar_url":"https://avatars2.githubusercontent.com/u/7600935?v=4","gravatar_id":"","url":"https://api.github.com/users/wrongtest","html_url":"https://github.com/wrongtest","followers_url":"https://api.github.com/users/wrongtest/followers","following_url":"https://api.github.com/users/wrongtest/following{/other_user}","gists_url":"https://api.github.com/users/wrongtest/gists{/gist_id}","starred_url":"https://api.github.com/users/wrongtest/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/wrongtest/subscriptions","organizations_url":"https://api.github.com/users/wrongtest/orgs","repos_url":"https://api.github.com/users/wrongtest/repos","events_url":"https://api.github.com/users/wrongtest/events{/privacy}","received_events_url":"https://api.github.com/users/wrongtest/received_events","type":"User","site_admin":false},"labels":[{"id":404586594,"node_id":"MDU6TGFiZWw0MDQ1ODY1OTQ=","url":"https://api.github.com/repos/tensorflow/tensorflow/labels/stat:awaiting%20tensorflower","name":"stat:awaiting tensorflower","color":"b60205","default":false},{"id":299643928,"node_id":"MDU6TGFiZWwyOTk2NDM5Mjg=","url":"https://api.github.com/repos/tensorflow/tensorflow/labels/stat:contributions%20welcome","name":"stat:contributions welcome","color":"009800","default":false}],"state":"open","locked":false,"assignee":{"login":"tfboyd","id":23486130,"node_id":"MDQ6VXNlcjIzNDg2MTMw","avatar_url":"https://avatars1.githubusercontent.com/u/23486130?v=4","gravatar_id":"","url":"https://api.github.com/users/tfboyd","html_url":"https://github.com/tfboyd","followers_url":"https://api.github.com/users/tfboyd/followers","following_url":"https://api.github.com/users/tfboyd/following{/other_user}","gists_url":"https://api.github.com/users/tfboyd/gists{/gist_id}","starred_url":"https://api.github.com/users/tfboyd/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tfboyd/subscriptions","organizations_url":"https://api.github.com/users/tfboyd/orgs","repos_url":"https://api.github.com/users/tfboyd/repos","events_url":"https://api.github.com/users/tfboyd/events{/privacy}","received_events_url":"https://api.github.com/users/tfboyd/received_events","type":"User","site_admin":false},"assignees":[{"login":"tfboyd","id":23486130,"node_id":"MDQ6VXNlcjIzNDg2MTMw","avatar_url":"https://avatars1.githubusercontent.com/u/23486130?v=4","gravatar_id":"","url":"https://api.github.com/users/tfboyd","html_url":"https://github.com/tfboyd","followers_url":"https://api.github.com/users/tfboyd/followers","following_url":"https://api.github.com/users/tfboyd/following{/other_user}","gists_url":"https://api.github.com/users/tfboyd/gists{/gist_id}","starred_url":"https://api.github.com/users/tfboyd/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tfboyd/subscriptions","organizations_url":"https://api.github.com/users/tfboyd/orgs","repos_url":"https://api.github.com/users/tfboyd/repos","events_url":"https://api.github.com/users/tfboyd/events{/privacy}","received_events_url":"https://api.github.com/users/tfboyd/received_events","type":"User","site_admin":false}],"milestone":null,"comments":13,"created_at":"2018-08-02T11:04:05Z","updated_at":"2018-10-04T20:15:53Z","closed_at":null,"author_association":"NONE","body":"### System information\r\n- **Have I written custom code (as opposed to using a stock example script provided in TensorFlow)**: yes\r\n- **OS Platform and Distribution (e.g., Linux Ubuntu 16.04)**: 3.10.0-693.2.2.el7.x86_64\r\n- **TensorFlow installed from (source or binary)**: binary\r\n- **TensorFlow version (use command below)**: 1.8.0\r\n- **Python version**: Python 2.7.14 :: Anaconda\r\n- **Bazel version (if compiling from source)**: None\r\n- **GCC/Compiler version (if compiling from source)**: None\r\n- **CUDA/cuDNN version**: cuda==9.0, cudnn==7.0.4\r\n- **GPU model and memory**: None\r\n- **Exact command to reproduce**: See below\r\n\r\n### Describe the problem\r\nWhen batch_size is 0, max pooling operation seems to produce an unhandled cudaError_t status. It may cause subsequent operations fail with odd error message. That is extremely difficult to debug.\r\n\r\n(This corner case bothers us, where we first extract some bounding boxes and then run traditional convolution operations on areas specified by them. The above error occurs in case that no bounding boxes are detected thus batch_size becomes 0. However, the python exception will be randomly thrown at following operation or following session run steps)\r\n\r\n```python\r\nimport tensorflow as tf\r\nimport numpy as np\r\n\r\nx = tf.placeholder(dtype=tf.float32, shape=[None, 4, 4, 1])\r\npool_op = tf.nn.pool(x, pooling_type=\"MAX\", window_shape=[2, 2], strides=[1, 1], padding=\"SAME\")\r\n\r\ny = tf.placeholder(dtype=tf.float32, shape=[None])\r\nother_op = tf.where(tf.equal(y, 1.0))\r\n\r\nnormal_data = np.zeros([1, 4, 4, 1], dtype=\"float32\")\r\nempty_data = np.zeros([0, 4, 4, 1], dtype=\"float32\")\r\n\r\n# cudaError is thread local, limit thread pool size to make it easy to reproduce\r\nconfig = tf.ConfigProto()\r\nconfig.inter_op_parallelism_threads = 1\r\nwith tf.Session(config=config) as sess:\r\n    # run other_op success\r\n    print sess.run(other_op, {y: [1.0, 2.0, 3.0, 4.0]})  # [[0]]\r\n\r\n    # run pooling on datas success\r\n    print sess.run(pool_op, {x: normal_data}).shape  # (1, 4, 4, 1)\r\n    print sess.run(pool_op, {x: empty_data}).shape  # (0, 4, 4, 1)\r\n\r\n    # run other_op now failed\r\n    print sess.run(other_op, {y: [1.0, 2.0, 3.0, 4.0]})  # err\r\n``` \r\n\r\nAbove code report error:\r\ntensorflow.python.framework.errors_impl.InternalError: WhereOp: Could not launch cub::DeviceReduce::Sum to count number of true / nonzero indices.  temp_storage_bytes: 1, status: invalid configuration argument\r\n\r\n\"invalid configuration argument\" seems to be message return by cudaGetError, which indicates a failed kernel launch due to zero or too large number of block threads.\r\n\r\n### Source code / logs\r\n![image](https://user-images.githubusercontent.com/7600935/43579974-6dd4967a-9686-11e8-9b22-8288159d155c.png)\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/tensorflow/tensorflow/issues/comments/427154396","html_url":"https://github.com/tensorflow/tensorflow/issues/21338#issuecomment-427154396","issue_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/21338","id":427154396,"node_id":"MDEyOklzc3VlQ29tbWVudDQyNzE1NDM5Ng==","user":{"login":"tfboyd","id":23486130,"node_id":"MDQ6VXNlcjIzNDg2MTMw","avatar_url":"https://avatars1.githubusercontent.com/u/23486130?v=4","gravatar_id":"","url":"https://api.github.com/users/tfboyd","html_url":"https://github.com/tfboyd","followers_url":"https://api.github.com/users/tfboyd/followers","following_url":"https://api.github.com/users/tfboyd/following{/other_user}","gists_url":"https://api.github.com/users/tfboyd/gists{/gist_id}","starred_url":"https://api.github.com/users/tfboyd/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tfboyd/subscriptions","organizations_url":"https://api.github.com/users/tfboyd/orgs","repos_url":"https://api.github.com/users/tfboyd/repos","events_url":"https://api.github.com/users/tfboyd/events{/privacy}","received_events_url":"https://api.github.com/users/tfboyd/received_events","type":"User","site_admin":false},"created_at":"2018-10-04T20:15:53Z","updated_at":"2018-10-04T20:15:53Z","author_association":"MEMBER","body":"@asimshankar   Can you review this with the changes to the team?  I was not sure which github was Alek.  "}}
{"actor":{"display_login":"tfboyd"}}