{"action":"created","issue":{"url":"https://api.github.com/repos/tensorflow/tensorflow/issues/20094","repository_url":"https://api.github.com/repos/tensorflow/tensorflow","labels_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/20094/labels{/name}","comments_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/20094/comments","events_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/20094/events","html_url":"https://github.com/tensorflow/tensorflow/issues/20094","id":333257622,"node_id":"MDU6SXNzdWUzMzMyNTc2MjI=","number":20094,"title":"Wrong Results for tf.matmul(X,Y) with X.shape[0]>2^21","user":{"login":"jopege","id":40358486,"node_id":"MDQ6VXNlcjQwMzU4NDg2","avatar_url":"https://avatars0.githubusercontent.com/u/40358486?v=4","gravatar_id":"","url":"https://api.github.com/users/jopege","html_url":"https://github.com/jopege","followers_url":"https://api.github.com/users/jopege/followers","following_url":"https://api.github.com/users/jopege/following{/other_user}","gists_url":"https://api.github.com/users/jopege/gists{/gist_id}","starred_url":"https://api.github.com/users/jopege/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jopege/subscriptions","organizations_url":"https://api.github.com/users/jopege/orgs","repos_url":"https://api.github.com/users/jopege/repos","events_url":"https://api.github.com/users/jopege/events{/privacy}","received_events_url":"https://api.github.com/users/jopege/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2018-06-18T12:55:58Z","updated_at":"2018-06-18T13:02:09Z","closed_at":null,"author_association":"NONE","body":"\r\n\r\n------------------------\r\n\r\n### System information\r\n- **OS Platform and Distribution (e.g., Linux Ubuntu 16.04)**: Win 10, 64bit\r\n- **TensorFlow installed from (source or binary)**: source\r\n- **TensorFlow version (use command below)**: 1.8.0 (same issue with 1.5.x)\r\n- **Python version**: 3.5.3\r\n- **GPU model and memory**: GTX 1080 \r\n\r\n\r\n### Describe the problem\r\nWhen using `tf.matmul(X,Y)` with `X.shape==[n,2]` ,`Y.shape==[2,2]` **AND** `n>2^21 #=2097152` only the first `[0:2^21 , : ]` results are correct\r\n\r\n- Does only occur on GPU\r\n- Does not occur if Y==tf.eye(2)\r\n- it is possible to circumvent the problem by splitting the operation into chunks of `n<=2**21`\r\n\r\n-> i guess a solution would be to automaticaly split the operation or warn the user \r\n\r\n### Source code\r\n`import tensorflow as tf`\r\n\r\n`X=tf.ones([10+2**21,2],dtype=tf.float32)`\r\n`Y=tf.constant([[1,0],[0.01,1]],dtype=tf.float32)`\r\n`mul=tf.matmul(X,Y)`\r\n\r\n`sess=tf.Session()`\r\n`sess.run(tf.global_variables_initializer())`\r\n`result=sess.run(mul)`\r\n`sess.close()`\r\n\r\n`print(\"These are ok: \" + str(result[2**21-2:2**21,:]))`\r\n\r\n`print(\"These are wrong: \" + str(result[2**21:2**21+2,:]))`\r\n\r\n\r\n### Output Log\r\n\r\n_2018-06-18 14:49:53.627009: I T:\\src\\github\\tensorflow\\tensorflow\\core\\platform\\cpu_feature_guard.cc:140] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2\r\n2018-06-18 14:49:53.868050: I T:\\src\\github\\tensorflow\\tensorflow\\core\\common_runtime\\gpu\\gpu_device.cc:1356] Found device 0 with properties: \r\nname: GeForce GTX 1080 major: 6 minor: 1 memoryClockRate(GHz): 1.7715\r\npciBusID: 0000:03:00.0\r\ntotalMemory: 8.00GiB freeMemory: 6.61GiB\r\n2018-06-18 14:49:53.868514: I T:\\src\\github\\tensorflow\\tensorflow\\core\\common_runtime\\gpu\\gpu_device.cc:1435] Adding visible gpu devices: 0\r\n2018-06-18 14:49:54.450846: I T:\\src\\github\\tensorflow\\tensorflow\\core\\common_runtime\\gpu\\gpu_device.cc:923] Device interconnect StreamExecutor with strength 1 edge matrix:\r\n2018-06-18 14:49:54.451158: I T:\\src\\github\\tensorflow\\tensorflow\\core\\common_runtime\\gpu\\gpu_device.cc:929]      0 \r\n2018-06-18 14:49:54.451346: I T:\\src\\github\\tensorflow\\tensorflow\\core\\common_runtime\\gpu\\gpu_device.cc:942] 0:   N \r\n2018-06-18 14:49:54.451641: I T:\\src\\github\\tensorflow\\tensorflow\\core\\common_runtime\\gpu\\gpu_device.cc:1053] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 6384 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1080, pci bus id: 0000:03:00.0, compute capability: 6.1)_\r\n\r\nThese are ok: [[1.01 1.  ]\r\n [1.01 1.  ]]\r\nThese are wrong: [[0. 0.]\r\n [0. 0.]]\r\n"},"comment":{"url":"https://api.github.com/repos/tensorflow/tensorflow/issues/comments/398046293","html_url":"https://github.com/tensorflow/tensorflow/issues/20094#issuecomment-398046293","issue_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/20094","id":398046293,"node_id":"MDEyOklzc3VlQ29tbWVudDM5ODA0NjI5Mw==","user":{"login":"jopege","id":40358486,"node_id":"MDQ6VXNlcjQwMzU4NDg2","avatar_url":"https://avatars0.githubusercontent.com/u/40358486?v=4","gravatar_id":"","url":"https://api.github.com/users/jopege","html_url":"https://github.com/jopege","followers_url":"https://api.github.com/users/jopege/followers","following_url":"https://api.github.com/users/jopege/following{/other_user}","gists_url":"https://api.github.com/users/jopege/gists{/gist_id}","starred_url":"https://api.github.com/users/jopege/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jopege/subscriptions","organizations_url":"https://api.github.com/users/jopege/orgs","repos_url":"https://api.github.com/users/jopege/repos","events_url":"https://api.github.com/users/jopege/events{/privacy}","received_events_url":"https://api.github.com/users/jopege/received_events","type":"User","site_admin":false},"created_at":"2018-06-18T13:02:09Z","updated_at":"2018-06-18T13:02:09Z","author_association":"NONE","body":"Example for the workaround is to use this function (might not be perfect, but it works):\r\n```\r\ndef large_koord_tensor_mul(tensor1, tensor2):\r\n    steps = int(int(tensor1.shape[0]) / 2097152)\r\n    if steps == 0:\r\n        output = tf.matmul(tensor1, tensor2)\r\n    else:\r\n        output = tf.matmul(tensor1[0:2097152, :], tensor2)\r\n        for i in range(1, steps):\r\n            addon_output = tf.matmul(tensor1[i * 2097152:(i + 1) * 2097152, :], tensor2)\r\n            output = tf.concat([output, addon_output], 0)\r\n        addon_output = tf.matmul(tensor1[steps * 2097152:, :], tensor2)\r\n        output = tf.concat([output, addon_output], 0)\r\n    return output\r\n```\r\n\r\n\r\n\r\n\r\n```\r\nX=tf.ones([10+2**21,2],dtype=tf.float32)\r\nY=tf.constant([[1,0],[0.01,1]],dtype=tf.float32)\r\n\r\n\r\nmul=large_koord_tensor_mul(X,Y)\r\n\r\nsess=tf.Session()\r\nsess.run(tf.global_variables_initializer())\r\n\r\nresult=sess.run(mul)\r\nsess.close()\r\n\r\nprint(\"These are ok:\" + str(result[2**21-2:2**21,:]))\r\n\r\nprint(\"These are NOT wrong\" + str(result[2**21:2**21+2,:]))\r\n```\r\n\r\n\r\n### Result\r\n\r\n\r\nThese are ok:[[1.01 1.  ]\r\n [1.01 1.  ]]\r\nThese are NOT wrong[[1.01 1.  ]\r\n [1.01 1.  ]]"}}
{"actor":{"display_login":"jopege"}}