{"action":"opened","issue":{"url":"https://api.github.com/repos/tensorflow/tensorflow/issues/20600","repository_url":"https://api.github.com/repos/tensorflow/tensorflow","labels_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/20600/labels{/name}","comments_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/20600/comments","events_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/20600/events","html_url":"https://github.com/tensorflow/tensorflow/issues/20600","id":339067305,"node_id":"MDU6SXNzdWUzMzkwNjczMDU=","number":20600,"title":"Tensorflow 1.8 graph destructor hanging indefinitely","user":{"login":"alanhdu","id":1914111,"node_id":"MDQ6VXNlcjE5MTQxMTE=","avatar_url":"https://avatars1.githubusercontent.com/u/1914111?v=4","gravatar_id":"","url":"https://api.github.com/users/alanhdu","html_url":"https://github.com/alanhdu","followers_url":"https://api.github.com/users/alanhdu/followers","following_url":"https://api.github.com/users/alanhdu/following{/other_user}","gists_url":"https://api.github.com/users/alanhdu/gists{/gist_id}","starred_url":"https://api.github.com/users/alanhdu/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/alanhdu/subscriptions","organizations_url":"https://api.github.com/users/alanhdu/orgs","repos_url":"https://api.github.com/users/alanhdu/repos","events_url":"https://api.github.com/users/alanhdu/events{/privacy}","received_events_url":"https://api.github.com/users/alanhdu/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2018-07-06T20:48:23Z","updated_at":"2018-07-06T20:48:23Z","closed_at":null,"author_association":"CONTRIBUTOR","body":"I'm sorry to dump this on y'all on a weekend, but we're having strange problems with our CI when trying to upgrade to Tensorflow 1.8.\r\n\r\n### System information\r\n- **Have I written custom code (as opposed to using a stock example script provided in TensorFlow)**:\r\n- **OS Platform and Distribution (e.g., Linux Ubuntu 16.04)**: Debian Stretch\r\n- **TensorFlow installed from (source or binary)**: Binary (CPU wheel)\r\n- **TensorFlow version (use command below)**: `v1.8.0-0-g93bc2e2072`\r\n- **Python version**: `3.6.6`\r\n\r\n### Describe the problem\r\nWhen we run our test suite, it will intermittently hang (we've let it run overnight without any progress). We've traced this problem to TensorFlow 1.8 -- when we set `TF_C_API_GRAPH_CONSTRUCTION=0`, our test suite passes without problems and when we use Tensorflow 1.6 and 1.7, we don't seem to have any problems.\r\n\r\nUnfortunately, I have not been able to create a minified, reproducible test case (it's also frustrating because I can't replicate this locally, despite using the same Docker image that our CI build is using). I have, however, been able to use [Pyflame](https://github.com/uber/pyflame) and gdb to grab stack traces (see https://gist.github.com/alanhdu/9ce45f9061a8e48ba2b0728489cb4ee7).\r\n\r\nI think the relevant lines are:\r\n```\r\n/usr/local/lib/python3.6/site-packages/tensorflow/python/framework/c_api_util.py:__del__:50\r\n/usr/local/lib/python3.6/inspect.py:getmodule:732\r\n/usr/local/lib/python3.6/inspect.py:findsource:780\r\n/usr/local/lib/python3.6/inspect.py:getframeinfo:1445\r\n/usr/local/lib/python3.6/inspect.py:getouterframes:1471\r\n/usr/local/lib/python3.6/inspect.py:stack:1494\r\n/usr/local/lib/python3.6/site-packages/coverage/debug.py:short_stack:143\r\n/usr/local/lib/python3.6/site-packages/coverage/collector.py:__init__:107\r\n/usr/local/lib/python3.6/site-packages/coverage/control.py:_init:266\r\n/usr/local/lib/python3.6/site-packages/coverage/control.py:load:675\r\n/usr/local/lib/python3.6/site-packages/pytest_cov/embed.py:init:67\r\n/usr/local/lib/python3.6/site-packages/pytest_cov/embed.py:multiprocessing_start:23\r\n/usr/local/lib/python3.6/multiprocessing/util.py:_run_after_forkers:132\r\n/usr/local/lib/python3.6/multiprocessing/process.py:_bootstrap:251\r\n/usr/local/lib/python3.6/multiprocessing/popen_fork.py:_launch:73\r\n/usr/local/lib/python3.6/multiprocessing/popen_fork.py:__init__:19\r\n/usr/local/lib/python3.6/multiprocessing/context.py:_Popen:277\r\n/usr/local/lib/python3.6/multiprocessing/process.py:start:105\r\n/usr/local/lib/python3.6/multiprocessing/pool.py:_repopulate_pool:239\r\n/usr/local/lib/python3.6/multiprocessing/pool.py:__init__:174\r\n/usr/local/lib/python3.6/multiprocessing/context.py:Pool:119\r\n```\r\n\r\nwhich seems to be hanging at\r\n\r\n```\r\n#0  __pthread_cond_destroy (cond=0x56312aa153d0) at pthread_cond_destroy.c:76\r\n#1  0x00007f6eef4deebb in tensorflow::thread::ThreadPool::Impl::~Impl() () from /usr/local/lib/python3.6/site-packages/tensorflow/python/../libtensorflow_framework.so\r\n#2  0x00007f6eef4df89a in tensorflow::thread::ThreadPool::~ThreadPool() () from /usr/local/lib/python3.6/site-packages/tensorflow/python/../libtensorflow_framework.so\r\n#3  0x00007f6eef87d665 in tensorflow::SingleThreadedCpuDevice::~SingleThreadedCpuDevice() ()\r\n   from /usr/local/lib/python3.6/site-packages/tensorflow/python/../libtensorflow_framework.so\r\n#4  0x00007f6eef87d85a in tensorflow::GraphRunner::~GraphRunner() () from /usr/local/lib/python3.6/site-packages/tensorflow/python/../libtensorflow_framework.so\r\n#5  0x00007f6ef13e3aef in TF_DeleteGraph () from /usr/local/lib/python3.6/site-packages/tensorflow/python/_pywrap_tensorflow_internal.so\r\n#6  0x00007f6ef1038784 in _wrap_TF_DeleteGraph () from /usr/local/lib/python3.6/site-packages/tensorflow/python/_pywrap_tensorflow_internal.so\r\n#7  0x00007f6f4a6c78e9 in _PyCFunction_FastCallDict (func_obj=0x7f6ef4de69d8, args=0x7f6e84006730, nargs=<optimized out>, kwargs=kwargs@entry=0x0)\r\nat Objects/methodobject.c:234\r\n```\r\n\r\nDo y'all happen to know what could have caused this error? Even tips on how to debug this would be super helpful.\r\n"}}
{"actor":{"display_login":"alanhdu"}}