{"action":"created","issue":{"url":"https://api.github.com/repos/tensorflow/tensorflow/issues/19155","repository_url":"https://api.github.com/repos/tensorflow/tensorflow","labels_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/19155/labels{/name}","comments_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/19155/comments","events_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/19155/events","html_url":"https://github.com/tensorflow/tensorflow/issues/19155","id":321270159,"node_id":"MDU6SXNzdWUzMjEyNzAxNTk=","number":19155,"title":"[Bug] Silent memory failure","user":{"login":"unrealwill","id":11304248,"node_id":"MDQ6VXNlcjExMzA0MjQ4","avatar_url":"https://avatars0.githubusercontent.com/u/11304248?v=4","gravatar_id":"","url":"https://api.github.com/users/unrealwill","html_url":"https://github.com/unrealwill","followers_url":"https://api.github.com/users/unrealwill/followers","following_url":"https://api.github.com/users/unrealwill/following{/other_user}","gists_url":"https://api.github.com/users/unrealwill/gists{/gist_id}","starred_url":"https://api.github.com/users/unrealwill/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/unrealwill/subscriptions","organizations_url":"https://api.github.com/users/unrealwill/orgs","repos_url":"https://api.github.com/users/unrealwill/repos","events_url":"https://api.github.com/users/unrealwill/events{/privacy}","received_events_url":"https://api.github.com/users/unrealwill/received_events","type":"User","site_admin":false},"labels":[{"id":404586594,"node_id":"MDU6TGFiZWw0MDQ1ODY1OTQ=","url":"https://api.github.com/repos/tensorflow/tensorflow/labels/stat:awaiting%20tensorflower","name":"stat:awaiting tensorflower","color":"f4b400","default":false},{"id":473172988,"node_id":"MDU6TGFiZWw0NzMxNzI5ODg=","url":"https://api.github.com/repos/tensorflow/tensorflow/labels/type:bug/performance","name":"type:bug/performance","color":"159b2e","default":false}],"state":"closed","locked":false,"assignee":{"login":"zheng-xq","id":15736910,"node_id":"MDQ6VXNlcjE1NzM2OTEw","avatar_url":"https://avatars0.githubusercontent.com/u/15736910?v=4","gravatar_id":"","url":"https://api.github.com/users/zheng-xq","html_url":"https://github.com/zheng-xq","followers_url":"https://api.github.com/users/zheng-xq/followers","following_url":"https://api.github.com/users/zheng-xq/following{/other_user}","gists_url":"https://api.github.com/users/zheng-xq/gists{/gist_id}","starred_url":"https://api.github.com/users/zheng-xq/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/zheng-xq/subscriptions","organizations_url":"https://api.github.com/users/zheng-xq/orgs","repos_url":"https://api.github.com/users/zheng-xq/repos","events_url":"https://api.github.com/users/zheng-xq/events{/privacy}","received_events_url":"https://api.github.com/users/zheng-xq/received_events","type":"User","site_admin":false},"assignees":[{"login":"zheng-xq","id":15736910,"node_id":"MDQ6VXNlcjE1NzM2OTEw","avatar_url":"https://avatars0.githubusercontent.com/u/15736910?v=4","gravatar_id":"","url":"https://api.github.com/users/zheng-xq","html_url":"https://github.com/zheng-xq","followers_url":"https://api.github.com/users/zheng-xq/followers","following_url":"https://api.github.com/users/zheng-xq/following{/other_user}","gists_url":"https://api.github.com/users/zheng-xq/gists{/gist_id}","starred_url":"https://api.github.com/users/zheng-xq/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/zheng-xq/subscriptions","organizations_url":"https://api.github.com/users/zheng-xq/orgs","repos_url":"https://api.github.com/users/zheng-xq/repos","events_url":"https://api.github.com/users/zheng-xq/events{/privacy}","received_events_url":"https://api.github.com/users/zheng-xq/received_events","type":"User","site_admin":false}],"milestone":null,"comments":9,"created_at":"2018-05-08T17:06:52Z","updated_at":"2018-11-05T13:29:09Z","closed_at":"2018-07-16T07:56:08Z","author_association":"NONE","body":"### System information\r\n- **Have I written custom code (as opposed to using a stock example script provided in TensorFlow)**: yes (see below)\r\n- **OS Platform and Distribution (e.g., Linux Ubuntu 16.04)**:Lubuntu 16.04\r\n- **TensorFlow installed from (source or binary)**: binary\r\n- **TensorFlow version (use command below)**: v1.7.0-3-g024aecf414 1.7.0\r\n- **Python version**: Python 3.5.2\r\n **Bazel version (if compiling from source)**: N/A\r\n- **GCC/Compiler version (if compiling from source)**: N/A\r\n- **CUDA/cuDNN version**:  Driver Version: 390.30 CUDA Driver Version = 9.1\r\n- **GPU model and memory**: NVidia GTX 1080Ti (11Go)\r\n- **Exact command to reproduce**: copy paste the given code and call the only function (if running on a  GPU with different memory please fiddle with the size of the arrays accordingly)\r\n\r\n### Describe the problem\r\nWhen run on GPU the program runs fine except results are silently zeroed for elements in the last 25% of result. It only happens when I it uses a lot of memory but instead of failing it tries to compute and silently mess the results.\r\n\r\n### Source code / logs\r\n\r\n```\r\nimport tensorflow as tf\r\ndef bugTensorflow():\r\n    #bug with GTX 1080 Ti\r\n    #Lubuntu 16.04\r\n    #Driver Version: 390.30\r\n    #CUDA Driver Version = 9.1\r\n    nbpoints1 = 100000\r\n    nbpoints2 = 4000\r\n    dim = 3\r\n\r\n    #works fine on cpu and bugs on second gpu as well\r\n    #with tf.device(\"/cpu:0\"):\r\n    #with tf.device(\"/gpu:1\"):\r\n\r\n    with tf.device(\"/gpu:0\"):\r\n        points = tf.random_normal( (nbpoints1,1,dim) )\r\n        traj = tf.random_normal((1,nbpoints2,dim) )\r\n        diff= points-traj\r\n        dist = tf.reduce_sum( diff*diff,axis=-1)\r\n    sess = tf.Session()\r\n    with(sess.as_default()):\r\n        _dist = sess.run(dist)\r\n        print(\"dist.shape\")\r\n        print(dist.shape)\r\n        print(\"dist[0]\")\r\n        print(_dist[0])\r\n        print(\"dist[75000]\")\r\n        print(_dist[75000])\r\n        print(\"tensorflow version\")\r\n        print(tf.__version__)\r\n```\r\nResults : \r\n\r\n> dist.shape\r\n> (100000, 4000)\r\n> dist[0]\r\n> [ 3.918147   8.78706    3.0947132 ...  0.8662497  0.6857513 11.894537 ]\r\n> dist[75000]\r\n> [0. 0. 0. ... 0. 0. 0.]\r\n> tensorflow version\r\n> 1.7.0\r\n> \r\n\r\n\r\n\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/tensorflow/tensorflow/issues/comments/435873363","html_url":"https://github.com/tensorflow/tensorflow/issues/19155#issuecomment-435873363","issue_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/19155","id":435873363,"node_id":"MDEyOklzc3VlQ29tbWVudDQzNTg3MzM2Mw==","user":{"login":"chsigg","id":7523982,"node_id":"MDQ6VXNlcjc1MjM5ODI=","avatar_url":"https://avatars3.githubusercontent.com/u/7523982?v=4","gravatar_id":"","url":"https://api.github.com/users/chsigg","html_url":"https://github.com/chsigg","followers_url":"https://api.github.com/users/chsigg/followers","following_url":"https://api.github.com/users/chsigg/following{/other_user}","gists_url":"https://api.github.com/users/chsigg/gists{/gist_id}","starred_url":"https://api.github.com/users/chsigg/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/chsigg/subscriptions","organizations_url":"https://api.github.com/users/chsigg/orgs","repos_url":"https://api.github.com/users/chsigg/repos","events_url":"https://api.github.com/users/chsigg/events{/privacy}","received_events_url":"https://api.github.com/users/chsigg/received_events","type":"User","site_admin":false},"created_at":"2018-11-05T13:29:09Z","updated_at":"2018-11-05T13:29:09Z","author_association":"NONE","body":"Sorry for ignoring your issue. I've reproduced the issue and will provide updates in #22123."}}
{"actor":{"display_login":"chsigg"}}