{"action":"created","issue":{"url":"https://api.github.com/repos/tensorflow/tensorflow/issues/16311","repository_url":"https://api.github.com/repos/tensorflow/tensorflow","labels_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/16311/labels{/name}","comments_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/16311/comments","events_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/16311/events","html_url":"https://github.com/tensorflow/tensorflow/issues/16311","id":290699695,"number":16311,"title":"Segmentation fault in _pywrap_tensorflow_internal.so","user":{"login":"mpkuse","id":10494231,"avatar_url":"https://avatars3.githubusercontent.com/u/10494231?v=4","gravatar_id":"","url":"https://api.github.com/users/mpkuse","html_url":"https://github.com/mpkuse","followers_url":"https://api.github.com/users/mpkuse/followers","following_url":"https://api.github.com/users/mpkuse/following{/other_user}","gists_url":"https://api.github.com/users/mpkuse/gists{/gist_id}","starred_url":"https://api.github.com/users/mpkuse/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mpkuse/subscriptions","organizations_url":"https://api.github.com/users/mpkuse/orgs","repos_url":"https://api.github.com/users/mpkuse/repos","events_url":"https://api.github.com/users/mpkuse/events{/privacy}","received_events_url":"https://api.github.com/users/mpkuse/received_events","type":"User","site_admin":false},"labels":[{"id":404586594,"url":"https://api.github.com/repos/tensorflow/tensorflow/labels/stat:awaiting%20tensorflower","name":"stat:awaiting tensorflower","color":"b60205","default":false}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":5,"created_at":"2018-01-23T03:46:06Z","updated_at":"2018-02-06T04:43:20Z","closed_at":null,"author_association":"NONE","body":"\r\n### System information\r\n- **Have I written custom code (as opposed to using a stock example script provided in TensorFlow)**: Yes\r\n- **OS Platform and Distribution (e.g., Linux Ubuntu 16.04)**: Ubuntu 16.04.3 LTS (Xenial)\r\n- **TensorFlow installed from (source or binary)**: Binary\r\n- **TensorFlow version (use command below)**: ('v1.4.0-19-ga52c8d9', '1.4.1')\r\n- **Python version**: 2.7.12\r\n- **Bazel version (if compiling from source)**:\r\n- **GCC/Compiler version (if compiling from source)**:\r\n- **CUDA/cuDNN version**: Cuda8, cudnn6\r\n- **GPU model and memory**: Titan Xp (with driver 387.26)\r\n- **Exact command to reproduce**:\r\n\r\nYou can collect some of this information using our environment capture script:\r\n\r\nhttps://github.com/tensorflow/tensorflow/tree/master/tools/tf_env_collect.sh\r\n```\r\n== cat /etc/issue ===============================================\r\nLinux ubuntu 4.4.0-87-generic #110-Ubuntu SMP Tue Jul 18 12:55:35 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux\r\nVERSION=\"16.04.3 LTS (Xenial Xerus)\"\r\nVERSION_ID=\"16.04\"\r\nVERSION_CODENAME=xenial\r\n\r\n== are we in docker =============================================\r\nNo\r\n\r\n== compiler =====================================================\r\nc++ (Ubuntu 5.4.0-6ubuntu1~16.04.5) 5.4.0 20160609\r\nCopyright (C) 2015 Free Software Foundation, Inc.\r\nThis is free software; see the source for copying conditions.  There is NO\r\nwarranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.\r\n\r\n\r\n== uname -a =====================================================\r\nLinux ubuntu 4.4.0-87-generic #110-Ubuntu SMP Tue Jul 18 12:55:35 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux\r\n\r\n== check pips ===================================================\r\nnumpy (1.14.0)\r\nprotobuf (3.5.1)\r\ntensorflow-gpu (1.4.1)\r\ntensorflow-tensorboard (0.4.0)\r\n\r\n== check for virtualenv =========================================\r\nFalse\r\n\r\n== tensorflow import ============================================\r\ntf.VERSION = 1.4.1\r\ntf.GIT_VERSION = v1.4.0-19-ga52c8d9\r\ntf.COMPILER_VERSION = v1.4.0-19-ga52c8d9\r\nSanity check: array([1], dtype=int32)\r\n\r\n== env ==========================================================\r\nLD_LIBRARY_PATH :/usr/local/cuda/lib64/\r\nDYLD_LIBRARY_PATH is unset\r\n\r\n== nvidia-smi ===================================================\r\nTue Jan 23 11:09:12 2018       \r\n+-----------------------------------------------------------------------------+\r\n| NVIDIA-SMI 387.26                 Driver Version: 387.26                    |\r\n|-------------------------------+----------------------+----------------------+\r\n| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |\r\n| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |\r\n|===============================+======================+======================|\r\n|   0  TITAN Xp            Off  | 00000000:04:00.0 Off |                  N/A |\r\n| 40%   66C    P2   182W / 250W |  11763MiB / 12189MiB |     73%      Default |\r\n+-------------------------------+----------------------+----------------------+\r\n|   1  TITAN Xp            Off  | 00000000:05:00.0 Off |                  N/A |\r\n| 23%   30C    P8     8W / 250W |  11591MiB / 12189MiB |      0%      Default |\r\n+-------------------------------+----------------------+----------------------+\r\n|   2  TITAN Xp            Off  | 00000000:06:00.0 Off |                  N/A |\r\n| 28%   48C    P0    62W / 250W |      0MiB / 12189MiB |      0%      Default |\r\n+-------------------------------+----------------------+----------------------+\r\n|   3  TITAN Xp            Off  | 00000000:07:00.0 Off |                  N/A |\r\n| 26%   46C    P0    63W / 250W |      0MiB / 12189MiB |      0%      Default |\r\n+-------------------------------+----------------------+----------------------+\r\n|   4  TITAN Xp            Off  | 00000000:08:00.0 Off |                  N/A |\r\n| 26%   46C    P0    63W / 250W |      0MiB / 12189MiB |      0%      Default |\r\n+-------------------------------+----------------------+----------------------+\r\n|   5  TITAN Xp            Off  | 00000000:0C:00.0 Off |                  N/A |\r\n| 23%   43C    P0    62W / 250W |      0MiB / 12189MiB |      0%      Default |\r\n+-------------------------------+----------------------+----------------------+\r\n|   6  TITAN Xp            Off  | 00000000:0E:00.0 Off |                  N/A |\r\n| 25%   44C    P0    62W / 250W |      0MiB / 12189MiB |      0%      Default |\r\n+-------------------------------+----------------------+----------------------+\r\n|   7  TITAN Xp            Off  | 00000000:0F:00.0 Off |                  N/A |\r\n| 42%   69C    P2   167W / 250W |  11833MiB / 12189MiB |     31%      Default |\r\n+-------------------------------+----------------------+----------------------+\r\n                                                                               \r\n+-----------------------------------------------------------------------------+\r\n| Processes:                                                       GPU Memory |\r\n|  GPU       PID   Type   Process name                             Usage      |\r\n|=============================================================================|\r\n|    0     19682      C   /home/peiliang/tensorflow/bin/python       11751MiB |\r\n|    1     19682      C   /home/peiliang/tensorflow/bin/python       11579MiB |\r\n|    7     27581      C   python                                     11823MiB |\r\n+-----------------------------------------------------------------------------+\r\n\r\n== cuda libs  ===================================================\r\n/usr/local/cuda-8.0/lib64/libcudart.so.8.0.61\r\n/usr/local/cuda-8.0/lib64/libcudart_static.a\r\n/usr/local/cuda-8.0/doc/man/man7/libcudart.so.7\r\n/usr/local/cuda-8.0/doc/man/man7/libcudart.7\r\n/usr/local/cuda-9.1/lib64/libcudart.so.9.1.85\r\n/usr/local/cuda-9.1/lib64/libcudart_static.a\r\n/usr/local/cuda-9.1/doc/man/man7/libcudart.so.7\r\n/usr/local/cuda-9.1/doc/man/man7/libcudart.7\r\n```\r\n\r\nYou can obtain the TensorFlow version with\r\n\r\npython -c \"import tensorflow as tf; print(tf.GIT_VERSION, tf.VERSION)\"\r\n\r\n### Describe the problem\r\nMy custom learning code works perfectly on my older workstation with 2 GPU cards. But am having issue with our new workstation which has 8 GPU cards. I get a Segmentation fault. \r\n\r\n\r\n### Source code / logs\r\n\r\nThe entire source code is: https://github.com/mpkuse/cartwheel_train/tree/config-files\r\n\r\nThe main-script is `train_netvlad.py`. Currently my learning data is private, \r\nif you really need it to test, I can provide the data as well (~100 GB). \r\n\r\nMy code basically builds a network with tf.slim. I have a custom operations to build a layer. Have a custom loss function. Can be found in `CartWheelFlow.py/ class VGGDescriptor`. It uses tf.while. \r\nData is managed by `class TimeMachineRender`\r\n\r\nstack trace for the crash. \r\n\r\n```\r\n$ gdb --args python train_netvlad.py -t tfsuper.logs/test \r\n(gdb) run\r\n.\r\n.\r\n.\r\nThread 1 \"python\" received signal SIGSEGV, Segmentation fault.\r\n0x00007ffef7f61a2c in std::__detail::_Map_base<std::string, std::pair<std::string const, unsigned long>, std::allocator<std::pair<std::string const, unsigned long> >, std::__detail::_Select1st, std::equal_to<std::string>, std::hash<std::string>, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<true, false, true>, true>::operator[](std::string const&) ()\r\n   from /usr/local/lib/python2.7/dist-packages/tensorflow/python/_pywrap_tensorflow_internal.so\r\n(gdb) where\r\n#0  0x00007ffef7f61a2c in std::__detail::_Map_base<std::string, std::pair<std::string const, unsigned long>, std::allocator<std::pair<std::string const, unsigned long> >, std::__detail::_Select1st, std::equal_to<std::string>, std::hash<std::string>, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<true, false, true>, true>::operator[](std::string const&) ()\r\n   from /usr/local/lib/python2.7/dist-packages/tensorflow/python/_pywrap_tensorflow_internal.so\r\n#1  0x00007ffef7f6c4d9 in tensorflow::DirectSession::Run(tensorflow::RunOptions const&, std::vector<std::pair<std::string, tensorflow::Tensor>, std::allocator<std::pair<std::string, tensorflow::Tensor> > > const&, std::vector<std::string, std::allocator<std::string> > const&, std::vector<std::string, std::allocator<std::string> > const&, std::vector<tensorflow::Tensor, std::allocator<tensorflow::Tensor> >*, tensorflow::RunMetadata*) ()\r\n   from /usr/local/lib/python2.7/dist-packages/tensorflow/python/_pywrap_tensorflow_internal.so\r\n#2  0x00007ffef5ed94ea in TF_Run_Helper(tensorflow::Session*, char const*, TF_Buffer const*, std::vector<std::pair<std::string, tensorflow::Tensor>, std::allocator<std::pair<std::string, tensorfl---Type <return> to continue, or q <return> to quit---\r\now::Tensor> > > const&, std::vector<std::string, std::allocator<std::string> > const&, TF_Tensor**, std::vector<std::string, std::allocator<std::string> > const&, TF_Buffer*, TF_Status*) ()\r\n   from /usr/local/lib/python2.7/dist-packages/tensorflow/python/_pywrap_tensorflow_internal.so\r\n#3  0x00007ffef5ed9824 in TF_Run ()\r\n   from /usr/local/lib/python2.7/dist-packages/tensorflow/python/_pywrap_tensorflow_internal.so\r\n#4  0x00007ffef5bf701a in tensorflow::TF_Run_wrapper_helper(TF_DeprecatedSession*, char const*, TF_Buffer const*, _object*, tensorflow::gtl::InlinedVector<char const*, 8> const&, tensorflow::gtl::InlinedVector<char const*, 8> const&, TF_Status*, tensorflow::gtl::InlinedVector<_object*, 8>*, TF_Buffer*) ()\r\n   from /usr/local/lib/python2.7/dist-packages/tensorflow/python/_pywrap_tensorflow_internal.so\r\n#5  0x00007ffef5bf7411 in tensorflow::TF_Run_wrapper(TF_DeprecatedSession*, TF_Buffer const*, _object*, tensorflow::gtl::InlinedVector<char const*, 8> const&, tensorflow::gtl::InlinedVector<char const*, 8> const&, TF_Status*, tensorflow::gtl::InlinedVector<_object*, 8>*, TF_Buffer*) ()\r\n   from /usr/local/lib/python2.7/dist-packages/tensorflow/python/---Type <return> to continue, or q <return> to quit---\r\n_pywrap_tensorflow_internal.so\r\n#6  0x00007ffef5bbb6f1 in _wrap_TF_Run ()\r\n   from /usr/local/lib/python2.7/dist-packages/tensorflow/python/_pywrap_tensorflow_internal.so\r\n#7  0x00000000004c45fa in PyEval_EvalFrameEx ()\r\n#8  0x00000000004c2705 in PyEval_EvalCodeEx ()\r\n#9  0x00000000004de69e in ?? ()\r\n#10 0x00000000004b0c93 in PyObject_Call ()\r\n#11 0x00000000004c6ef6 in PyEval_EvalFrameEx ()\r\n#12 0x00000000004c2705 in PyEval_EvalCodeEx ()\r\n#13 0x00000000004ca7df in PyEval_EvalFrameEx ()\r\n#14 0x00000000004c2705 in PyEval_EvalCodeEx ()\r\n#15 0x00000000004ca7df in PyEval_EvalFrameEx ()\r\n#16 0x00000000004c2705 in PyEval_EvalCodeEx ()\r\n#17 0x00000000004ca7df in PyEval_EvalFrameEx ()\r\n#18 0x00000000004c2705 in PyEval_EvalCodeEx ()\r\n#19 0x00000000004ca088 in PyEval_EvalFrameEx ()\r\n#20 0x00000000004c2705 in PyEval_EvalCodeEx ()\r\n#21 0x00000000004c24a9 in PyEval_EvalCode ()\r\n#22 0x00000000004f19ef in ?? ()\r\n#23 0x00000000004ec372 in PyRun_FileExFlags ()\r\n#24 0x00000000004eaaf1 in PyRun_SimpleFileExFlags ()\r\n#25 0x000000000049e208 in Py_Main ()\r\n#26 0x00007ffff7810830 in __libc_start_main (main=0x49db30 <main>, argc=4, \r\n    argv=0x7fffffffe558, init=<optimized out>, fini=<optimized out>, \r\n    rtld_fini=<optimized out>, stack_end=0x7fffffffe548) at ../csu/libc-start.c:291\r\n#27 0x000000000049da59 in _start ()\r\n```\r\n"},"comment":{"url":"https://api.github.com/repos/tensorflow/tensorflow/issues/comments/363309269","html_url":"https://github.com/tensorflow/tensorflow/issues/16311#issuecomment-363309269","issue_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/16311","id":363309269,"user":{"login":"mpkuse","id":10494231,"avatar_url":"https://avatars3.githubusercontent.com/u/10494231?v=4","gravatar_id":"","url":"https://api.github.com/users/mpkuse","html_url":"https://github.com/mpkuse","followers_url":"https://api.github.com/users/mpkuse/followers","following_url":"https://api.github.com/users/mpkuse/following{/other_user}","gists_url":"https://api.github.com/users/mpkuse/gists{/gist_id}","starred_url":"https://api.github.com/users/mpkuse/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mpkuse/subscriptions","organizations_url":"https://api.github.com/users/mpkuse/orgs","repos_url":"https://api.github.com/users/mpkuse/repos","events_url":"https://api.github.com/users/mpkuse/events{/privacy}","received_events_url":"https://api.github.com/users/mpkuse/received_events","type":"User","site_admin":false},"created_at":"2018-02-06T04:43:20Z","updated_at":"2018-02-06T04:43:20Z","author_association":"NONE","body":"Any updates on this?"}}
{"actor":{"display_login":"mpkuse"}}