{"action":"reopened","issue":{"url":"https://api.github.com/repos/istio/istio/issues/5694","repository_url":"https://api.github.com/repos/istio/istio","labels_url":"https://api.github.com/repos/istio/istio/issues/5694/labels{/name}","comments_url":"https://api.github.com/repos/istio/istio/issues/5694/comments","events_url":"https://api.github.com/repos/istio/istio/issues/5694/events","html_url":"https://github.com/istio/istio/issues/5694","id":324228813,"number":5694,"title":"envoy v2 crashes during TestIngressGateway503DuringRuleChange ","user":{"login":"ostromart","id":30089849,"avatar_url":"https://avatars3.githubusercontent.com/u/30089849?v=4","gravatar_id":"","url":"https://api.github.com/users/ostromart","html_url":"https://github.com/ostromart","followers_url":"https://api.github.com/users/ostromart/followers","following_url":"https://api.github.com/users/ostromart/following{/other_user}","gists_url":"https://api.github.com/users/ostromart/gists{/gist_id}","starred_url":"https://api.github.com/users/ostromart/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ostromart/subscriptions","organizations_url":"https://api.github.com/users/ostromart/orgs","repos_url":"https://api.github.com/users/ostromart/repos","events_url":"https://api.github.com/users/ostromart/events{/privacy}","received_events_url":"https://api.github.com/users/ostromart/received_events","type":"User","site_admin":false},"labels":[{"id":735781806,"url":"https://api.github.com/repos/istio/istio/labels/area/networking","name":"area/networking","color":"0e8a16","default":false}],"state":"open","locked":false,"assignee":{"login":"qiwzhang","id":11790727,"avatar_url":"https://avatars0.githubusercontent.com/u/11790727?v=4","gravatar_id":"","url":"https://api.github.com/users/qiwzhang","html_url":"https://github.com/qiwzhang","followers_url":"https://api.github.com/users/qiwzhang/followers","following_url":"https://api.github.com/users/qiwzhang/following{/other_user}","gists_url":"https://api.github.com/users/qiwzhang/gists{/gist_id}","starred_url":"https://api.github.com/users/qiwzhang/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/qiwzhang/subscriptions","organizations_url":"https://api.github.com/users/qiwzhang/orgs","repos_url":"https://api.github.com/users/qiwzhang/repos","events_url":"https://api.github.com/users/qiwzhang/events{/privacy}","received_events_url":"https://api.github.com/users/qiwzhang/received_events","type":"User","site_admin":false},"assignees":[{"login":"qiwzhang","id":11790727,"avatar_url":"https://avatars0.githubusercontent.com/u/11790727?v=4","gravatar_id":"","url":"https://api.github.com/users/qiwzhang","html_url":"https://github.com/qiwzhang","followers_url":"https://api.github.com/users/qiwzhang/followers","following_url":"https://api.github.com/users/qiwzhang/following{/other_user}","gists_url":"https://api.github.com/users/qiwzhang/gists{/gist_id}","starred_url":"https://api.github.com/users/qiwzhang/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/qiwzhang/subscriptions","organizations_url":"https://api.github.com/users/qiwzhang/orgs","repos_url":"https://api.github.com/users/qiwzhang/repos","events_url":"https://api.github.com/users/qiwzhang/events{/privacy}","received_events_url":"https://api.github.com/users/qiwzhang/received_events","type":"User","site_admin":false}],"milestone":null,"comments":5,"created_at":"2018-05-18T00:18:32Z","updated_at":"2018-05-29T20:05:32Z","closed_at":null,"author_association":"CONTRIBUTOR","body":"Steps to reproduce:\r\nre-enable TestIngressGateway503DuringRuleChange (currently skipped) and run it. \r\nObservations:\r\nEnvoy crashes about 12 sec into the test which altogether lasts 25s. This shows up as a window of \"connection refused\" of ~ 2 sec during the test. \r\nI strongly suspect that this is due to some mixer filter config code, since disabling the mixer filter config the test runs fine. The last config received by envoy about 200ms before the crash (this is received on a different thread from the crash, so config seems to be parsed fine and it seems like the data plane crashes shortly after):\r\n\r\n[503_crash.txt](https://github.com/istio/istio/files/2015065/503_crash.txt)\r\n\r\nThe regular image doesn't provide much useful trace info:\r\n\r\n[2018-05-17 21:29:42.041][34][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] Caught Segmentation fault, suspect faulting address 0x0\r\n[2018-05-17 21:29:42.066][34][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:90] Backtrace obj</usr/local/bin/envoy> thr<34> (use tools/stack_decode.py):\r\n[2018-05-17 21:29:42.094][34][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:107] thr<34> #0 0xc34078\r\n[2018-05-17 21:29:42.109][34][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:99] thr<34> obj</lib/x86_64-linux-gnu/libpthread.so.0>\r\n[2018-05-17 21:29:42.122][34][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:107] thr<34> #1 0x7f1d20c0838f\r\n[2018-05-17 21:29:42.131][34][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:110] end backtrace thread 34\r\n\r\n*stack_decode output*\r\n\r\n[2018-05-17 21:29:42.041][34][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] Caught Segmentation fault, suspect faulting address 0x0\r\n[2018-05-17 21:29:42.066][34][critical] Backtrace (most recent call first) from thread 34:\r\n #1 __gnu_cxx::__atomic_add_single(int*, int) at atomicity.h:74\r\n #2 \r\n #3 ?? ??:0\r\n #4 \r\n \r\nHowever repeating the test with a full envoy debug image doesn't lead to a crash, though the test still fails. "}}
{"actor":{"display_login":"ostromart"}}