{"action":"created","issue":{"url":"https://api.github.com/repos/tensorflow/tensorflow/issues/22013","repository_url":"https://api.github.com/repos/tensorflow/tensorflow","labels_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/22013/labels{/name}","comments_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/22013/comments","events_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/22013/events","html_url":"https://github.com/tensorflow/tensorflow/issues/22013","id":356314231,"node_id":"MDU6SXNzdWUzNTYzMTQyMzE=","number":22013,"title":"tf.scatter_nd_update - Segmentation fault (core dumped)","user":{"login":"mpekalski","id":2975068,"node_id":"MDQ6VXNlcjI5NzUwNjg=","avatar_url":"https://avatars1.githubusercontent.com/u/2975068?v=4","gravatar_id":"","url":"https://api.github.com/users/mpekalski","html_url":"https://github.com/mpekalski","followers_url":"https://api.github.com/users/mpekalski/followers","following_url":"https://api.github.com/users/mpekalski/following{/other_user}","gists_url":"https://api.github.com/users/mpekalski/gists{/gist_id}","starred_url":"https://api.github.com/users/mpekalski/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mpekalski/subscriptions","organizations_url":"https://api.github.com/users/mpekalski/orgs","repos_url":"https://api.github.com/users/mpekalski/repos","events_url":"https://api.github.com/users/mpekalski/events{/privacy}","received_events_url":"https://api.github.com/users/mpekalski/received_events","type":"User","site_admin":false},"labels":[{"id":386191887,"node_id":"MDU6TGFiZWwzODYxOTE4ODc=","url":"https://api.github.com/repos/tensorflow/tensorflow/labels/stat:awaiting%20response","name":"stat:awaiting response","color":"5319e7","default":false}],"state":"open","locked":false,"assignee":{"login":"rohan100jain","id":144114,"node_id":"MDQ6VXNlcjE0NDExNA==","avatar_url":"https://avatars2.githubusercontent.com/u/144114?v=4","gravatar_id":"","url":"https://api.github.com/users/rohan100jain","html_url":"https://github.com/rohan100jain","followers_url":"https://api.github.com/users/rohan100jain/followers","following_url":"https://api.github.com/users/rohan100jain/following{/other_user}","gists_url":"https://api.github.com/users/rohan100jain/gists{/gist_id}","starred_url":"https://api.github.com/users/rohan100jain/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/rohan100jain/subscriptions","organizations_url":"https://api.github.com/users/rohan100jain/orgs","repos_url":"https://api.github.com/users/rohan100jain/repos","events_url":"https://api.github.com/users/rohan100jain/events{/privacy}","received_events_url":"https://api.github.com/users/rohan100jain/received_events","type":"User","site_admin":false},"assignees":[{"login":"rohan100jain","id":144114,"node_id":"MDQ6VXNlcjE0NDExNA==","avatar_url":"https://avatars2.githubusercontent.com/u/144114?v=4","gravatar_id":"","url":"https://api.github.com/users/rohan100jain","html_url":"https://github.com/rohan100jain","followers_url":"https://api.github.com/users/rohan100jain/followers","following_url":"https://api.github.com/users/rohan100jain/following{/other_user}","gists_url":"https://api.github.com/users/rohan100jain/gists{/gist_id}","starred_url":"https://api.github.com/users/rohan100jain/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/rohan100jain/subscriptions","organizations_url":"https://api.github.com/users/rohan100jain/orgs","repos_url":"https://api.github.com/users/rohan100jain/repos","events_url":"https://api.github.com/users/rohan100jain/events{/privacy}","received_events_url":"https://api.github.com/users/rohan100jain/received_events","type":"User","site_admin":false}],"milestone":null,"comments":0,"created_at":"2018-09-02T21:09:33Z","updated_at":"2018-09-03T06:35:19Z","closed_at":null,"author_association":"NONE","body":"-----------------------\r\n\r\n### System information\r\n- **Have I written custom code (as opposed to using a stock example script provided in TensorFlow)**:\r\nyes\r\n- **OS Platform and Distribution (e.g., Linux Ubuntu 16.04)**:\r\nLinux Ubuntu 16.04\r\n- **Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue happens on mobile device**:\r\n- **TensorFlow installed from (source or binary)**:\r\nsource\r\n- **TensorFlow version (use command below)**:\r\n\r\nTF checkpoint I have built\r\n```\r\n/tmp/tensorflow# git log   \r\ncommit 09792df012c22622324f085f46edde33006c7355\r\nAuthor: A. Unique TensorFlower <gardener@tensorflow.org>\r\nDate:   Sun Aug 26 02:07:11 2018 -0700\r\n\r\n    compat: Update forward compatibility horizon to 2018-08-26\r\n    \r\n    PiperOrigin-RevId: 210266798\r\n```\r\n\r\n```\r\n== cat /etc/issue ===============================================\r\nLinux 3bed2f328777 4.13.0-38-generic #43~16.04.1-Ubuntu SMP Wed Mar 14 17:48:43 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux\r\nVERSION=\"16.04.5 LTS (Xenial Xerus)\"\r\nVERSION_ID=\"16.04\"\r\nVERSION_CODENAME=xenial\r\n\r\n== are we in docker =============================================\r\nYes\r\n\r\n== compiler =====================================================\r\nc++ (Ubuntu 5.4.0-6ubuntu1~16.04.10) 5.4.0 20160609\r\nCopyright (C) 2015 Free Software Foundation, Inc.\r\nThis is free software; see the source for copying conditions.  There is NO\r\nwarranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.\r\n\r\n\r\n== uname -a =====================================================\r\nLinux 3bed2f328777 4.13.0-38-generic #43~16.04.1-Ubuntu SMP Wed Mar 14 17:48:43 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux\r\n\r\n== check pips ===================================================\r\nnumpy (1.14.5)\r\nprotobuf (3.6.1)\r\ntensorflow (1.10.0)\r\n\r\n== check for virtualenv =========================================\r\nFalse\r\n\r\n== tensorflow import ============================================\r\ntf.VERSION = 1.10.0\r\ntf.GIT_VERSION = b'unknown'\r\ntf.COMPILER_VERSION = b'unknown'\r\nSanity check: array([1], dtype=int32)\r\n\r\n== env ==========================================================\r\nLD_LIBRARY_PATH /usr/local/cuda/lib64/stubs:/usr/local/cuda/extras/CUPTI/lib64:/usr/local/cuda/lib64/stubs:/usr/local/cuda/lib64/stubs:/usr/local/cuda/extras/CUPTI/lib64:/lib/amd64/server/:/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server:/opt/boost/lib:/opt/conda/lib/:/usr/local/cuda/lib64/:/opt/conda/lib/R/lib/:/usr/local/nvidia/lib64/:/usr/local/nvidia/lib:/lib/x86_64-linux-gnu:/usr/local/cuda/extras/CUPTI/lib64:/usr/lib/x86_64-linux-gnu:/opt/opencv/lib\r\nDYLD_LIBRARY_PATH is unset\r\n\r\n== nvidia-smi ===================================================\r\nWed Aug 29 19:57:14 2018       \r\n+-----------------------------------------------------------------------------+\r\n| NVIDIA-SMI 396.26                 Driver Version: 396.26                    |\r\n|-------------------------------+----------------------+----------------------+\r\n| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |\r\n| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |\r\n|===============================+======================+======================|\r\n|   0  GeForce GTX 108...  Off  | 00000000:01:00.0  On |                  N/A |\r\n|  0%   41C    P0    76W / 250W |    708MiB / 11177MiB |      0%      Default |\r\n+-------------------------------+----------------------+----------------------+\r\n                                                                               \r\n+-----------------------------------------------------------------------------+\r\n| Processes:                                                       GPU Memory |\r\n|  GPU       PID   Type   Process name                             Usage      |\r\n|=============================================================================|\r\n+-----------------------------------------------------------------------------+\r\n\r\n== cuda libs  ===================================================\r\n/usr/local/cuda-9.2/targets/x86_64-linux/lib/libcudart.so.9.2.148\r\n/usr/local/cuda-9.2/targets/x86_64-linux/lib/libcudart_static.a\r\n\r\n```\r\n**Bazel** version\r\n```\r\n$ bazel version\r\nWARNING: --batch mode is deprecated. Please instead explicitly shut down your Bazel server using the command \"bazel shutdown\".\r\nBuild label: 0.16.0\r\nBuild target: bazel-out/k8-opt/bin/src/main/java/com/google/devtools/build/lib/bazel/BazelServer_deploy.jar\r\nBuild time: Tue Jul 31 17:01:24 2018 (1533056484)\r\nBuild timestamp: 1533056484\r\nBuild timestamp as int: 1533056484\r\n```\r\n\r\n**CUDNN** version:\r\n```\r\n$ nvcc --version\r\nnvcc: NVIDIA (R) Cuda compiler driver\r\nCopyright (c) 2005-2018 NVIDIA Corporation\r\nBuilt on Tue_Jun_12_23:07:04_CDT_2018\r\nCuda compilation tools, release 9.2, V9.2.148\r\n```\r\n\r\n**GPU**: GEFORCE GTX 1080Ti, 11GB, GIGABYTE AORUS\r\n\r\n- **Exact command to reproduce**:\r\n```\r\nfrom __future__ import absolute_import\r\nfrom __future__ import division\r\nfrom __future__ import print_function\r\n\r\nimport tensorflow as tf\r\n\r\ndef scope_1():\r\n    print(\"DS1 SCOPE =============\")\r\n    with tf.variable_scope(\"scope_1\", reuse=tf.AUTO_REUSE):\r\n        x = tf.get_variable(\"x\", initializer=lambda: tf.zeros(shape=[], dtype=tf.float32), dtype=tf.float32\r\n                               , trainable=False, use_resource=True)             \r\n        print(\"graph: {}\".format(x.graph))\r\n        print(\"scope: {}\".format(tf.get_variable_scope().name))\r\n        print(\" name: {}\".format(x.name))\r\n        print(\"  var: {}\".format(str(x)))\r\n        current_scope = tf.get_variable_scope()       \r\n        assign_one = tf.assign(x, 1.0, name=\"x_is_one\")\r\n    \r\n    def scope_2(inputs, label):        \r\n        print(\"initial scope: {}\".format(tf.get_variable_scope().name))\r\n        print(\"DS1 SCOPE =============\")\r\n        #with tf.variable_scope(\"scope_1\", reuse=tf.AUTO_REUSE):\r\n        with tf.variable_scope(current_scope, reuse=tf.AUTO_REUSE):\r\n            y = tf.get_variable(\"x\", initializer=lambda: tf.zeros(shape=[], dtype=tf.float32), dtype=tf.float32\r\n                                   , trainable=False, use_resource=True)         \r\n            print(\"graph: {}\".format(y.graph))\r\n            print(\"scope: {}\".format(tf.get_variable_scope().name))\r\n            print(\" name: {}\".format(y.name))\r\n            print(\"  var: {}\".format(str(y)))\r\n            print(\"=============\")\r\n            print(y)\r\n            print(inputs)\r\n            #assign_two = tf.assign(y, tf.add(tf.cast(inputs, dtype=tf.float32),1.0), name=\"inputs_plus_1\")\r\n            assign_two = tf.identity(tf.assign(y, tf.add(tf.cast(inputs, dtype=tf.float32), 1.0)))\r\n            with tf.control_dependencies([assign_two]):\r\n                with tf.control_dependencies([tf.scatter_nd_update(y, [[0]], [0.22])]):\r\n                    return y.read_value(), label\r\n            #return x,label\r\n    \r\n    # test that original x is mutable\r\n    with tf.control_dependencies([assign_one]):\r\n        dataset = (tf.data.Dataset.from_tensor_slices(([1,2,3,4,5], [-1,-2,-3,-4,-5]))\r\n                    .map(scope_2)\r\n                    .batch(1)\r\n                    .repeat(1)        \r\n                    )\r\n    return dataset\r\n    \r\n                \r\nwith tf.variable_scope(\"scope_0\"):\r\n        dataset_fn = scope_1()\r\n\r\nwith tf.variable_scope(\"iterator\"):\r\n    # Define iterator from_string_handle. In general it is useful to have\r\n    # this kind of iterator if one wants to switch between train and validation\r\n    # within the training loop.        \r\n    iterator_t = dataset_fn.make_initializable_iterator()\r\n    iterator_handle = tf.placeholder(tf.string, shape=[], name=\"iterator_handle\")\r\n    iterator = tf.data.Iterator.from_string_handle(iterator_handle, \r\n                                                iterator_t.output_types,\r\n                                                iterator_t.output_shapes)\r\n    \r\n    def get_next_item():\r\n        next_elem = iterator.get_next(name=\"next_element\")\r\n        x, y = tf.cast(next_elem[0], tf.float32), next_elem[1]# tf.cast(next_elem[1], tf.int32)\r\n        return x, y    \r\nwith tf.Session() as sess:\r\n\r\n    sess.run([tf.global_variables_initializer(),tf.local_variables_initializer()])\r\n    handle_t = sess.run(iterator_t.string_handle())\r\n    # Run data iterator initialisation\r\n    sess.run(iterator_t.initializer)\r\n    print(sess.graph.get_operations()) \r\n    while True:\r\n        try:\r\n            print(sess.run(get_next_item(), feed_dict={iterator_handle:handle_t}))\r\n        except tf.errors.OutOfRangeError:\r\n                        print(\"End of training dataset.\")\r\n                        break        \r\n    print()\r\n    print(\"global vars: {}\".format(tf.global_variables()))\r\n    print(\"local vars: {}\".format(tf.local_variables()))\r\n    print(tf.get_default_graph().get_name_scope())\r\n\r\n```\r\n\r\n### Describe the problem\r\nI am trying to create a function that would modify a Tensor within a pipeline of Dataset API.\r\nThe scoping may seem weird, but that is minimal example that shows the problem that I created from my project. After adding `tf.scatter_nd_update(y, [[0]], [0.22])` I started to get segmentation fault.\r\nA minimal example with `tf.add` instead of `tf.scatter_nd_update` worked, see https://github.com/tensorflow/tensorflow/issues/22009\r\n\r\nI tried disabling GPU with `config=tf.ConfigProto(device_count={'GPU': 0})` and `CUDA_VISIBLE_DEVICES=\"\"` but the result was the same.\r\n\r\nI will recompile TF overnight (from the current master) with `--copt=-g` and try to provide a stacktrace \r\n\r\n```\r\nTF_BUILD_INFO = {container_type: \"gpu\", command: \"bazel build --config=opt --config=cuda --copt=-march=native --copt=-mfpmath=both --copt=-mtune=native --copt=-g --verbose_failures --cxxopt=-D_GLIBCXX_USE_CXX11_ABI=0 --jobs=8 --config=mkl --action_env=LD_LIBRARY_PATH=/usr/local/cuda/lib64/stubs:/usr/local/cuda/extras/CUPTI/lib64:/usr/local/cuda/lib64/stubs:/usr/local/cuda/lib64/stubs:/usr/local/cuda/extras/CUPTI/lib64:/lib/amd64/server/:/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server:/opt/boost/lib:/opt/conda/lib/:/usr/local/cuda/lib64/:/opt/conda/lib/R/lib/:/usr/local/nvidia/lib64/:/usr/local/nvidia/lib:/lib/x86_64-linux-gnu:/usr/local/cuda/extras/CUPTI/lib64:/usr/lib/x86_64-linux-gnu:/opt/opencv/lib tensorflow/tools/pip_package:build_pip_package\", source_HEAD: \"201be3d514d7239aa19496dba4dd0c85303b03f1\", source_remote_origin: \"https://github.com/tensorflow/tensorflow\", OS: \"Linux\", kernel: \"4.13.0-38-generic\", architecture: \"x86_64\", processor: \"Intel(R) Core(TM) i7-4770K CPU @ 3.50GHz\", processor_count: \"8\", memory_total: \"32877820 kB\", swap_total: \"69444596 kB\", Bazel_version: \"Build label: 0.16.0\", Java_version: \"1.8.0_181\", Python_version: \"3.6.2\", gpp_version: \"g++ (Ubuntu 5.4.0-6ubuntu1~16.04.10) 5.4.0 20160609\", swig_version: \"\", NVIDIA_driver_version: \"396.26\", CUDA_device_count: \"1\", CUDA_device_names: \"GeForce GTX 1080 Ti   (*PrimaryCard),\", CUDA_toolkit_version: \"V9.2.148\"}\r\n```\r\n\r\n### Source code / logs\r\n\r\n```\r\nDS1 SCOPE =============\r\ngraph: <tensorflow.python.framework.ops.Graph object at 0x7f0940f582b0>\r\nscope: scope_0/scope_1\r\n name: scope_0/scope_1/x:0\r\n  var: <tf.Variable 'scope_0/scope_1/x:0' shape=() dtype=float32>\r\ninitial scope: \r\nDS1 SCOPE =============\r\ngraph: <tensorflow.python.framework.ops.Graph object at 0x7f0940f582b0>\r\nscope: scope_0/scope_1\r\n name: scope_0/scope_1/x_1:0\r\n  var: <tf.Variable 'scope_0/scope_1/x_1:0' shape=() dtype=float32>\r\n=============\r\n<tf.Variable 'scope_0/scope_1/x_1:0' shape=() dtype=float32>\r\nTensor(\"arg0:0\", shape=(), dtype=int32)\r\n2018-09-02 20:52:39.456276: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:964] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\r\n2018-09-02 20:52:39.456710: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1406] Found device 0 with properties: \r\nname: GeForce GTX 1080 Ti major: 6 minor: 1 memoryClockRate(GHz): 1.721\r\npciBusID: 0000:01:00.0\r\ntotalMemory: 10.92GiB freeMemory: 10.23GiB\r\n2018-09-02 20:52:39.456728: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1485] Adding visible gpu devices: 0\r\n2018-09-02 20:52:39.665323: I tensorflow/core/common_runtime/gpu/gpu_device.cc:966] Device interconnect StreamExecutor with strength 1 edge matrix:\r\n2018-09-02 20:52:39.665362: I tensorflow/core/common_runtime/gpu/gpu_device.cc:972]      0 \r\n2018-09-02 20:52:39.665369: I tensorflow/core/common_runtime/gpu/gpu_device.cc:985] 0:   N \r\n2018-09-02 20:52:39.665731: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1098] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 9887 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1080 Ti, pci bus id: 0000:01:00.0, compute capability: 6.1)\r\n2018-09-02 20:52:39.756397: I tensorflow/core/common_runtime/process_util.cc:69] Creating new thread pool with default inter op setting: 2. Tune using inter_op_parallelism_threads for best performance.\r\nSegmentation fault (core dumped)\r\n```\r\n\r\nlog of run on CPU\r\n```\r\nCUDA_VISIBLE_DEVICES=\"\" python3 bug.py \r\nDS1 SCOPE =============\r\ngraph: <tensorflow.python.framework.ops.Graph object at 0x7ffaa61de390>\r\nscope: scope_0/scope_1\r\n name: scope_0/scope_1/x:0\r\n  var: <tf.Variable 'scope_0/scope_1/x:0' shape=() dtype=float32>\r\ninitial scope: \r\nDS1 SCOPE =============\r\ngraph: <tensorflow.python.framework.ops.Graph object at 0x7ffaa61de390>\r\nscope: scope_0/scope_1\r\n name: scope_0/scope_1/x_1:0\r\n  var: <tf.Variable 'scope_0/scope_1/x_1:0' shape=() dtype=float32>\r\n=============\r\n<tf.Variable 'scope_0/scope_1/x_1:0' shape=() dtype=float32>\r\nTensor(\"arg0:0\", shape=(), dtype=int32)\r\n2018-09-02 20:54:41.271567: E tensorflow/stream_executor/cuda/cuda_driver.cc:300] failed call to cuInit: CUDA_ERROR_NO_DEVICE: no CUDA-capable device is detected\r\n2018-09-02 20:54:41.271604: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:163] retrieving CUDA diagnostic information for host: 3bed2f328777\r\n2018-09-02 20:54:41.271614: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:170] hostname: 3bed2f328777\r\n2018-09-02 20:54:41.271655: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:194] libcuda reported version is: 396.26.0\r\n2018-09-02 20:54:41.271686: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:198] kernel reported version is: 396.26.0\r\n2018-09-02 20:54:41.271694: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:305] kernel version seems to match DSO: 396.26.0\r\n2018-09-02 20:54:41.271962: I tensorflow/core/common_runtime/process_util.cc:69] Creating new thread pool with default inter op setting: 2. Tune using inter_op_parallelism_threads for best performance.\r\nSegmentation fault (core dumped)\r\n```"},"comment":{"url":"https://api.github.com/repos/tensorflow/tensorflow/issues/comments/418014302","html_url":"https://github.com/tensorflow/tensorflow/issues/22013#issuecomment-418014302","issue_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/22013","id":418014302,"node_id":"MDEyOklzc3VlQ29tbWVudDQxODAxNDMwMg==","user":{"login":"tensorflowbutler","id":28546240,"node_id":"MDQ6VXNlcjI4NTQ2MjQw","avatar_url":"https://avatars2.githubusercontent.com/u/28546240?v=4","gravatar_id":"","url":"https://api.github.com/users/tensorflowbutler","html_url":"https://github.com/tensorflowbutler","followers_url":"https://api.github.com/users/tensorflowbutler/followers","following_url":"https://api.github.com/users/tensorflowbutler/following{/other_user}","gists_url":"https://api.github.com/users/tensorflowbutler/gists{/gist_id}","starred_url":"https://api.github.com/users/tensorflowbutler/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tensorflowbutler/subscriptions","organizations_url":"https://api.github.com/users/tensorflowbutler/orgs","repos_url":"https://api.github.com/users/tensorflowbutler/repos","events_url":"https://api.github.com/users/tensorflowbutler/events{/privacy}","received_events_url":"https://api.github.com/users/tensorflowbutler/received_events","type":"User","site_admin":false},"created_at":"2018-09-03T06:35:19Z","updated_at":"2018-09-03T06:35:19Z","author_association":"MEMBER","body":"Thank you for your post. We noticed you have not filled out the following field in the issue template. Could you update them if they are relevant in your case, or leave them as N/A? Thanks.\nCUDA/cuDNN version\nGPU model and memory"}}
{"actor":{"display_login":"tensorflowbutler"}}