{"action":"created","issue":{"url":"https://api.github.com/repos/tensorflow/tensorflow/issues/22013","repository_url":"https://api.github.com/repos/tensorflow/tensorflow","labels_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/22013/labels{/name}","comments_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/22013/comments","events_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/22013/events","html_url":"https://github.com/tensorflow/tensorflow/issues/22013","id":356314231,"node_id":"MDU6SXNzdWUzNTYzMTQyMzE=","number":22013,"title":"tf.scatter_nd_update - Segmentation fault (core dumped)","user":{"login":"mpekalski","id":2975068,"node_id":"MDQ6VXNlcjI5NzUwNjg=","avatar_url":"https://avatars1.githubusercontent.com/u/2975068?v=4","gravatar_id":"","url":"https://api.github.com/users/mpekalski","html_url":"https://github.com/mpekalski","followers_url":"https://api.github.com/users/mpekalski/followers","following_url":"https://api.github.com/users/mpekalski/following{/other_user}","gists_url":"https://api.github.com/users/mpekalski/gists{/gist_id}","starred_url":"https://api.github.com/users/mpekalski/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mpekalski/subscriptions","organizations_url":"https://api.github.com/users/mpekalski/orgs","repos_url":"https://api.github.com/users/mpekalski/repos","events_url":"https://api.github.com/users/mpekalski/events{/privacy}","received_events_url":"https://api.github.com/users/mpekalski/received_events","type":"User","site_admin":false},"labels":[{"id":386191887,"node_id":"MDU6TGFiZWwzODYxOTE4ODc=","url":"https://api.github.com/repos/tensorflow/tensorflow/labels/stat:awaiting%20response","name":"stat:awaiting response","color":"5319e7","default":false}],"state":"open","locked":false,"assignee":{"login":"rohan100jain","id":144114,"node_id":"MDQ6VXNlcjE0NDExNA==","avatar_url":"https://avatars2.githubusercontent.com/u/144114?v=4","gravatar_id":"","url":"https://api.github.com/users/rohan100jain","html_url":"https://github.com/rohan100jain","followers_url":"https://api.github.com/users/rohan100jain/followers","following_url":"https://api.github.com/users/rohan100jain/following{/other_user}","gists_url":"https://api.github.com/users/rohan100jain/gists{/gist_id}","starred_url":"https://api.github.com/users/rohan100jain/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/rohan100jain/subscriptions","organizations_url":"https://api.github.com/users/rohan100jain/orgs","repos_url":"https://api.github.com/users/rohan100jain/repos","events_url":"https://api.github.com/users/rohan100jain/events{/privacy}","received_events_url":"https://api.github.com/users/rohan100jain/received_events","type":"User","site_admin":false},"assignees":[{"login":"rohan100jain","id":144114,"node_id":"MDQ6VXNlcjE0NDExNA==","avatar_url":"https://avatars2.githubusercontent.com/u/144114?v=4","gravatar_id":"","url":"https://api.github.com/users/rohan100jain","html_url":"https://github.com/rohan100jain","followers_url":"https://api.github.com/users/rohan100jain/followers","following_url":"https://api.github.com/users/rohan100jain/following{/other_user}","gists_url":"https://api.github.com/users/rohan100jain/gists{/gist_id}","starred_url":"https://api.github.com/users/rohan100jain/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/rohan100jain/subscriptions","organizations_url":"https://api.github.com/users/rohan100jain/orgs","repos_url":"https://api.github.com/users/rohan100jain/repos","events_url":"https://api.github.com/users/rohan100jain/events{/privacy}","received_events_url":"https://api.github.com/users/rohan100jain/received_events","type":"User","site_admin":false}],"milestone":null,"comments":1,"created_at":"2018-09-02T21:09:33Z","updated_at":"2018-09-03T07:44:02Z","closed_at":null,"author_association":"NONE","body":"-----------------------\r\n\r\n### System information\r\n- **Have I written custom code (as opposed to using a stock example script provided in TensorFlow)**:\r\nyes\r\n- **OS Platform and Distribution (e.g., Linux Ubuntu 16.04)**:\r\nLinux Ubuntu 16.04\r\n- **Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue happens on mobile device**:\r\n- **TensorFlow installed from (source or binary)**:\r\nsource\r\n- **TensorFlow version (use command below)**:\r\n\r\nTF checkpoint I have built\r\n```\r\n/tmp/tensorflow# git log   \r\ncommit 09792df012c22622324f085f46edde33006c7355\r\nAuthor: A. Unique TensorFlower <gardener@tensorflow.org>\r\nDate:   Sun Aug 26 02:07:11 2018 -0700\r\n\r\n    compat: Update forward compatibility horizon to 2018-08-26\r\n    \r\n    PiperOrigin-RevId: 210266798\r\n```\r\n\r\n```\r\n== cat /etc/issue ===============================================\r\nLinux 3bed2f328777 4.13.0-38-generic #43~16.04.1-Ubuntu SMP Wed Mar 14 17:48:43 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux\r\nVERSION=\"16.04.5 LTS (Xenial Xerus)\"\r\nVERSION_ID=\"16.04\"\r\nVERSION_CODENAME=xenial\r\n\r\n== are we in docker =============================================\r\nYes\r\n\r\n== compiler =====================================================\r\nc++ (Ubuntu 5.4.0-6ubuntu1~16.04.10) 5.4.0 20160609\r\nCopyright (C) 2015 Free Software Foundation, Inc.\r\nThis is free software; see the source for copying conditions.  There is NO\r\nwarranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.\r\n\r\n\r\n== uname -a =====================================================\r\nLinux 3bed2f328777 4.13.0-38-generic #43~16.04.1-Ubuntu SMP Wed Mar 14 17:48:43 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux\r\n\r\n== check pips ===================================================\r\nnumpy (1.14.5)\r\nprotobuf (3.6.1)\r\ntensorflow (1.10.0)\r\n\r\n== check for virtualenv =========================================\r\nFalse\r\n\r\n== tensorflow import ============================================\r\ntf.VERSION = 1.10.0\r\ntf.GIT_VERSION = b'unknown'\r\ntf.COMPILER_VERSION = b'unknown'\r\nSanity check: array([1], dtype=int32)\r\n\r\n== env ==========================================================\r\nLD_LIBRARY_PATH /usr/local/cuda/lib64/stubs:/usr/local/cuda/extras/CUPTI/lib64:/usr/local/cuda/lib64/stubs:/usr/local/cuda/lib64/stubs:/usr/local/cuda/extras/CUPTI/lib64:/lib/amd64/server/:/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server:/opt/boost/lib:/opt/conda/lib/:/usr/local/cuda/lib64/:/opt/conda/lib/R/lib/:/usr/local/nvidia/lib64/:/usr/local/nvidia/lib:/lib/x86_64-linux-gnu:/usr/local/cuda/extras/CUPTI/lib64:/usr/lib/x86_64-linux-gnu:/opt/opencv/lib\r\nDYLD_LIBRARY_PATH is unset\r\n\r\n== nvidia-smi ===================================================\r\nWed Aug 29 19:57:14 2018       \r\n+-----------------------------------------------------------------------------+\r\n| NVIDIA-SMI 396.26                 Driver Version: 396.26                    |\r\n|-------------------------------+----------------------+----------------------+\r\n| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |\r\n| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |\r\n|===============================+======================+======================|\r\n|   0  GeForce GTX 108...  Off  | 00000000:01:00.0  On |                  N/A |\r\n|  0%   41C    P0    76W / 250W |    708MiB / 11177MiB |      0%      Default |\r\n+-------------------------------+----------------------+----------------------+\r\n                                                                               \r\n+-----------------------------------------------------------------------------+\r\n| Processes:                                                       GPU Memory |\r\n|  GPU       PID   Type   Process name                             Usage      |\r\n|=============================================================================|\r\n+-----------------------------------------------------------------------------+\r\n\r\n== cuda libs  ===================================================\r\n/usr/local/cuda-9.2/targets/x86_64-linux/lib/libcudart.so.9.2.148\r\n/usr/local/cuda-9.2/targets/x86_64-linux/lib/libcudart_static.a\r\n\r\n```\r\n**Bazel** version\r\n```\r\n$ bazel version\r\nWARNING: --batch mode is deprecated. Please instead explicitly shut down your Bazel server using the command \"bazel shutdown\".\r\nBuild label: 0.16.0\r\nBuild target: bazel-out/k8-opt/bin/src/main/java/com/google/devtools/build/lib/bazel/BazelServer_deploy.jar\r\nBuild time: Tue Jul 31 17:01:24 2018 (1533056484)\r\nBuild timestamp: 1533056484\r\nBuild timestamp as int: 1533056484\r\n```\r\n\r\n**CUDNN** version:\r\n```\r\n$ nvcc --version\r\nnvcc: NVIDIA (R) Cuda compiler driver\r\nCopyright (c) 2005-2018 NVIDIA Corporation\r\nBuilt on Tue_Jun_12_23:07:04_CDT_2018\r\nCuda compilation tools, release 9.2, V9.2.148\r\n```\r\n\r\n**GPU**: GEFORCE GTX 1080Ti, 11GB, GIGABYTE AORUS\r\n\r\n- **Exact command to reproduce**:\r\n```\r\nfrom __future__ import absolute_import\r\nfrom __future__ import division\r\nfrom __future__ import print_function\r\n\r\nimport tensorflow as tf\r\n\r\ndef scope_1():\r\n    print(\"DS1 SCOPE =============\")\r\n    with tf.variable_scope(\"scope_1\", reuse=tf.AUTO_REUSE):\r\n        x = tf.get_variable(\"x\", initializer=lambda: tf.zeros(shape=[], dtype=tf.float32), dtype=tf.float32\r\n                               , trainable=False, use_resource=True)             \r\n        print(\"graph: {}\".format(x.graph))\r\n        print(\"scope: {}\".format(tf.get_variable_scope().name))\r\n        print(\" name: {}\".format(x.name))\r\n        print(\"  var: {}\".format(str(x)))\r\n        current_scope = tf.get_variable_scope()       \r\n        assign_one = tf.assign(x, 1.0, name=\"x_is_one\")\r\n    \r\n    def scope_2(inputs, label):        \r\n        print(\"initial scope: {}\".format(tf.get_variable_scope().name))\r\n        print(\"DS1 SCOPE =============\")\r\n        #with tf.variable_scope(\"scope_1\", reuse=tf.AUTO_REUSE):\r\n        with tf.variable_scope(current_scope, reuse=tf.AUTO_REUSE):\r\n            y = tf.get_variable(\"x\", initializer=lambda: tf.zeros(shape=[], dtype=tf.float32), dtype=tf.float32\r\n                                   , trainable=False, use_resource=True)         \r\n            print(\"graph: {}\".format(y.graph))\r\n            print(\"scope: {}\".format(tf.get_variable_scope().name))\r\n            print(\" name: {}\".format(y.name))\r\n            print(\"  var: {}\".format(str(y)))\r\n            print(\"=============\")\r\n            print(y)\r\n            print(inputs)\r\n            #assign_two = tf.assign(y, tf.add(tf.cast(inputs, dtype=tf.float32),1.0), name=\"inputs_plus_1\")\r\n            assign_two = tf.identity(tf.assign(y, tf.add(tf.cast(inputs, dtype=tf.float32), 1.0)))\r\n            with tf.control_dependencies([assign_two]):\r\n                with tf.control_dependencies([tf.scatter_nd_update(y, [[0]], [0.22])]):\r\n                    return y.read_value(), label\r\n            #return x,label\r\n    \r\n    # test that original x is mutable\r\n    with tf.control_dependencies([assign_one]):\r\n        dataset = (tf.data.Dataset.from_tensor_slices(([1,2,3,4,5], [-1,-2,-3,-4,-5]))\r\n                    .map(scope_2)\r\n                    .batch(1)\r\n                    .repeat(1)        \r\n                    )\r\n    return dataset\r\n    \r\n                \r\nwith tf.variable_scope(\"scope_0\"):\r\n        dataset_fn = scope_1()\r\n\r\nwith tf.variable_scope(\"iterator\"):\r\n    # Define iterator from_string_handle. In general it is useful to have\r\n    # this kind of iterator if one wants to switch between train and validation\r\n    # within the training loop.        \r\n    iterator_t = dataset_fn.make_initializable_iterator()\r\n    iterator_handle = tf.placeholder(tf.string, shape=[], name=\"iterator_handle\")\r\n    iterator = tf.data.Iterator.from_string_handle(iterator_handle, \r\n                                                iterator_t.output_types,\r\n                                                iterator_t.output_shapes)\r\n    \r\n    def get_next_item():\r\n        next_elem = iterator.get_next(name=\"next_element\")\r\n        x, y = tf.cast(next_elem[0], tf.float32), next_elem[1]# tf.cast(next_elem[1], tf.int32)\r\n        return x, y    \r\nwith tf.Session() as sess:\r\n\r\n    sess.run([tf.global_variables_initializer(),tf.local_variables_initializer()])\r\n    handle_t = sess.run(iterator_t.string_handle())\r\n    # Run data iterator initialisation\r\n    sess.run(iterator_t.initializer)\r\n    print(sess.graph.get_operations()) \r\n    while True:\r\n        try:\r\n            print(sess.run(get_next_item(), feed_dict={iterator_handle:handle_t}))\r\n        except tf.errors.OutOfRangeError:\r\n                        print(\"End of training dataset.\")\r\n                        break        \r\n    print()\r\n    print(\"global vars: {}\".format(tf.global_variables()))\r\n    print(\"local vars: {}\".format(tf.local_variables()))\r\n    print(tf.get_default_graph().get_name_scope())\r\n\r\n```\r\n\r\n### Describe the problem\r\nI am trying to create a function that would modify a Tensor within a pipeline of Dataset API.\r\nThe scoping may seem weird, but that is minimal example that shows the problem that I created from my project. After adding `tf.scatter_nd_update(y, [[0]], [0.22])` I started to get segmentation fault.\r\nA minimal example with `tf.add` instead of `tf.scatter_nd_update` worked, see https://github.com/tensorflow/tensorflow/issues/22009\r\n\r\nI tried disabling GPU with `config=tf.ConfigProto(device_count={'GPU': 0})` and `CUDA_VISIBLE_DEVICES=\"\"` but the result was the same.\r\n\r\nI will recompile TF overnight (from the current master) with `--copt=-g` and try to provide a stacktrace \r\n\r\n```\r\nTF_BUILD_INFO = {container_type: \"gpu\", command: \"bazel build --config=opt --config=cuda --copt=-march=native --copt=-mfpmath=both --copt=-mtune=native --copt=-g --verbose_failures --cxxopt=-D_GLIBCXX_USE_CXX11_ABI=0 --jobs=8 --config=mkl --action_env=LD_LIBRARY_PATH=/usr/local/cuda/lib64/stubs:/usr/local/cuda/extras/CUPTI/lib64:/usr/local/cuda/lib64/stubs:/usr/local/cuda/lib64/stubs:/usr/local/cuda/extras/CUPTI/lib64:/lib/amd64/server/:/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server:/opt/boost/lib:/opt/conda/lib/:/usr/local/cuda/lib64/:/opt/conda/lib/R/lib/:/usr/local/nvidia/lib64/:/usr/local/nvidia/lib:/lib/x86_64-linux-gnu:/usr/local/cuda/extras/CUPTI/lib64:/usr/lib/x86_64-linux-gnu:/opt/opencv/lib tensorflow/tools/pip_package:build_pip_package\", source_HEAD: \"201be3d514d7239aa19496dba4dd0c85303b03f1\", source_remote_origin: \"https://github.com/tensorflow/tensorflow\", OS: \"Linux\", kernel: \"4.13.0-38-generic\", architecture: \"x86_64\", processor: \"Intel(R) Core(TM) i7-4770K CPU @ 3.50GHz\", processor_count: \"8\", memory_total: \"32877820 kB\", swap_total: \"69444596 kB\", Bazel_version: \"Build label: 0.16.0\", Java_version: \"1.8.0_181\", Python_version: \"3.6.2\", gpp_version: \"g++ (Ubuntu 5.4.0-6ubuntu1~16.04.10) 5.4.0 20160609\", swig_version: \"\", NVIDIA_driver_version: \"396.26\", CUDA_device_count: \"1\", CUDA_device_names: \"GeForce GTX 1080 Ti   (*PrimaryCard),\", CUDA_toolkit_version: \"V9.2.148\"}\r\n```\r\n\r\n### Source code / logs\r\n\r\n```\r\nDS1 SCOPE =============\r\ngraph: <tensorflow.python.framework.ops.Graph object at 0x7f0940f582b0>\r\nscope: scope_0/scope_1\r\n name: scope_0/scope_1/x:0\r\n  var: <tf.Variable 'scope_0/scope_1/x:0' shape=() dtype=float32>\r\ninitial scope: \r\nDS1 SCOPE =============\r\ngraph: <tensorflow.python.framework.ops.Graph object at 0x7f0940f582b0>\r\nscope: scope_0/scope_1\r\n name: scope_0/scope_1/x_1:0\r\n  var: <tf.Variable 'scope_0/scope_1/x_1:0' shape=() dtype=float32>\r\n=============\r\n<tf.Variable 'scope_0/scope_1/x_1:0' shape=() dtype=float32>\r\nTensor(\"arg0:0\", shape=(), dtype=int32)\r\n2018-09-02 20:52:39.456276: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:964] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\r\n2018-09-02 20:52:39.456710: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1406] Found device 0 with properties: \r\nname: GeForce GTX 1080 Ti major: 6 minor: 1 memoryClockRate(GHz): 1.721\r\npciBusID: 0000:01:00.0\r\ntotalMemory: 10.92GiB freeMemory: 10.23GiB\r\n2018-09-02 20:52:39.456728: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1485] Adding visible gpu devices: 0\r\n2018-09-02 20:52:39.665323: I tensorflow/core/common_runtime/gpu/gpu_device.cc:966] Device interconnect StreamExecutor with strength 1 edge matrix:\r\n2018-09-02 20:52:39.665362: I tensorflow/core/common_runtime/gpu/gpu_device.cc:972]      0 \r\n2018-09-02 20:52:39.665369: I tensorflow/core/common_runtime/gpu/gpu_device.cc:985] 0:   N \r\n2018-09-02 20:52:39.665731: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1098] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 9887 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1080 Ti, pci bus id: 0000:01:00.0, compute capability: 6.1)\r\n2018-09-02 20:52:39.756397: I tensorflow/core/common_runtime/process_util.cc:69] Creating new thread pool with default inter op setting: 2. Tune using inter_op_parallelism_threads for best performance.\r\nSegmentation fault (core dumped)\r\n```\r\n\r\nlog of run on CPU\r\n```\r\nCUDA_VISIBLE_DEVICES=\"\" python3 bug.py \r\nDS1 SCOPE =============\r\ngraph: <tensorflow.python.framework.ops.Graph object at 0x7ffaa61de390>\r\nscope: scope_0/scope_1\r\n name: scope_0/scope_1/x:0\r\n  var: <tf.Variable 'scope_0/scope_1/x:0' shape=() dtype=float32>\r\ninitial scope: \r\nDS1 SCOPE =============\r\ngraph: <tensorflow.python.framework.ops.Graph object at 0x7ffaa61de390>\r\nscope: scope_0/scope_1\r\n name: scope_0/scope_1/x_1:0\r\n  var: <tf.Variable 'scope_0/scope_1/x_1:0' shape=() dtype=float32>\r\n=============\r\n<tf.Variable 'scope_0/scope_1/x_1:0' shape=() dtype=float32>\r\nTensor(\"arg0:0\", shape=(), dtype=int32)\r\n2018-09-02 20:54:41.271567: E tensorflow/stream_executor/cuda/cuda_driver.cc:300] failed call to cuInit: CUDA_ERROR_NO_DEVICE: no CUDA-capable device is detected\r\n2018-09-02 20:54:41.271604: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:163] retrieving CUDA diagnostic information for host: 3bed2f328777\r\n2018-09-02 20:54:41.271614: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:170] hostname: 3bed2f328777\r\n2018-09-02 20:54:41.271655: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:194] libcuda reported version is: 396.26.0\r\n2018-09-02 20:54:41.271686: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:198] kernel reported version is: 396.26.0\r\n2018-09-02 20:54:41.271694: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:305] kernel version seems to match DSO: 396.26.0\r\n2018-09-02 20:54:41.271962: I tensorflow/core/common_runtime/process_util.cc:69] Creating new thread pool with default inter op setting: 2. Tune using inter_op_parallelism_threads for best performance.\r\nSegmentation fault (core dumped)\r\n```"},"comment":{"url":"https://api.github.com/repos/tensorflow/tensorflow/issues/comments/418029323","html_url":"https://github.com/tensorflow/tensorflow/issues/22013#issuecomment-418029323","issue_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/22013","id":418029323,"node_id":"MDEyOklzc3VlQ29tbWVudDQxODAyOTMyMw==","user":{"login":"mpekalski","id":2975068,"node_id":"MDQ6VXNlcjI5NzUwNjg=","avatar_url":"https://avatars1.githubusercontent.com/u/2975068?v=4","gravatar_id":"","url":"https://api.github.com/users/mpekalski","html_url":"https://github.com/mpekalski","followers_url":"https://api.github.com/users/mpekalski/followers","following_url":"https://api.github.com/users/mpekalski/following{/other_user}","gists_url":"https://api.github.com/users/mpekalski/gists{/gist_id}","starred_url":"https://api.github.com/users/mpekalski/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mpekalski/subscriptions","organizations_url":"https://api.github.com/users/mpekalski/orgs","repos_url":"https://api.github.com/users/mpekalski/repos","events_url":"https://api.github.com/users/mpekalski/events{/privacy}","received_events_url":"https://api.github.com/users/mpekalski/received_events","type":"User","site_admin":false},"created_at":"2018-09-03T07:44:02Z","updated_at":"2018-09-03T07:44:02Z","author_association":"NONE","body":"When executing line by line everything works fine until it starts the session."}}
{"actor":{"display_login":"mpekalski"}}