{"action":"created","issue":{"url":"https://api.github.com/repos/istio/istio/issues/5864","repository_url":"https://api.github.com/repos/istio/istio","labels_url":"https://api.github.com/repos/istio/istio/issues/5864/labels{/name}","comments_url":"https://api.github.com/repos/istio/istio/issues/5864/comments","events_url":"https://api.github.com/repos/istio/istio/issues/5864/events","html_url":"https://github.com/istio/istio/issues/5864","id":326770607,"number":5864,"title":"Istio ingress segfaults on upgrade from 0.7 to 0.8-20180525-23-50","user":{"login":"mandarjog","id":800375,"avatar_url":"https://avatars2.githubusercontent.com/u/800375?v=4","gravatar_id":"","url":"https://api.github.com/users/mandarjog","html_url":"https://github.com/mandarjog","followers_url":"https://api.github.com/users/mandarjog/followers","following_url":"https://api.github.com/users/mandarjog/following{/other_user}","gists_url":"https://api.github.com/users/mandarjog/gists{/gist_id}","starred_url":"https://api.github.com/users/mandarjog/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mandarjog/subscriptions","organizations_url":"https://api.github.com/users/mandarjog/orgs","repos_url":"https://api.github.com/users/mandarjog/repos","events_url":"https://api.github.com/users/mandarjog/events{/privacy}","received_events_url":"https://api.github.com/users/mandarjog/received_events","type":"User","site_admin":false},"labels":[{"id":816498729,"url":"https://api.github.com/repos/istio/istio/labels/kind/blocking%20release","name":"kind/blocking release","color":"fbca04","default":false}],"state":"open","locked":false,"assignee":{"login":"costinm","id":84202,"avatar_url":"https://avatars1.githubusercontent.com/u/84202?v=4","gravatar_id":"","url":"https://api.github.com/users/costinm","html_url":"https://github.com/costinm","followers_url":"https://api.github.com/users/costinm/followers","following_url":"https://api.github.com/users/costinm/following{/other_user}","gists_url":"https://api.github.com/users/costinm/gists{/gist_id}","starred_url":"https://api.github.com/users/costinm/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/costinm/subscriptions","organizations_url":"https://api.github.com/users/costinm/orgs","repos_url":"https://api.github.com/users/costinm/repos","events_url":"https://api.github.com/users/costinm/events{/privacy}","received_events_url":"https://api.github.com/users/costinm/received_events","type":"User","site_admin":false},"assignees":[{"login":"costinm","id":84202,"avatar_url":"https://avatars1.githubusercontent.com/u/84202?v=4","gravatar_id":"","url":"https://api.github.com/users/costinm","html_url":"https://github.com/costinm","followers_url":"https://api.github.com/users/costinm/followers","following_url":"https://api.github.com/users/costinm/following{/other_user}","gists_url":"https://api.github.com/users/costinm/gists{/gist_id}","starred_url":"https://api.github.com/users/costinm/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/costinm/subscriptions","organizations_url":"https://api.github.com/users/costinm/orgs","repos_url":"https://api.github.com/users/costinm/repos","events_url":"https://api.github.com/users/costinm/events{/privacy}","received_events_url":"https://api.github.com/users/costinm/received_events","type":"User","site_admin":false},{"login":"mandarjog","id":800375,"avatar_url":"https://avatars2.githubusercontent.com/u/800375?v=4","gravatar_id":"","url":"https://api.github.com/users/mandarjog","html_url":"https://github.com/mandarjog","followers_url":"https://api.github.com/users/mandarjog/followers","following_url":"https://api.github.com/users/mandarjog/following{/other_user}","gists_url":"https://api.github.com/users/mandarjog/gists{/gist_id}","starred_url":"https://api.github.com/users/mandarjog/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mandarjog/subscriptions","organizations_url":"https://api.github.com/users/mandarjog/orgs","repos_url":"https://api.github.com/users/mandarjog/repos","events_url":"https://api.github.com/users/mandarjog/events{/privacy}","received_events_url":"https://api.github.com/users/mandarjog/received_events","type":"User","site_admin":false},{"login":"qiwzhang","id":11790727,"avatar_url":"https://avatars0.githubusercontent.com/u/11790727?v=4","gravatar_id":"","url":"https://api.github.com/users/qiwzhang","html_url":"https://github.com/qiwzhang","followers_url":"https://api.github.com/users/qiwzhang/followers","following_url":"https://api.github.com/users/qiwzhang/following{/other_user}","gists_url":"https://api.github.com/users/qiwzhang/gists{/gist_id}","starred_url":"https://api.github.com/users/qiwzhang/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/qiwzhang/subscriptions","organizations_url":"https://api.github.com/users/qiwzhang/orgs","repos_url":"https://api.github.com/users/qiwzhang/repos","events_url":"https://api.github.com/users/qiwzhang/events{/privacy}","received_events_url":"https://api.github.com/users/qiwzhang/received_events","type":"User","site_admin":false}],"milestone":null,"comments":18,"created_at":"2018-05-26T22:09:14Z","updated_at":"2018-05-29T18:46:26Z","closed_at":null,"author_association":"CONTRIBUTOR","body":"Regpatrol for build 0.8-20180525-23-50 failed with a segfault.\r\n\r\nhttps://ibmcloud-perf.istio.io/regpatrol/release-0.8-20180525-23-50/\r\nNo_mixer and Full_istio are both failed performance runs with many 504s.\r\n\r\nIssues observed\r\n1. istio-telemetry logs have accesslogs that report a lot of traffic to `istio-policy`, but `istio-policy` has no record of it. You can check and correlate  `istio-policy-6db8dccc8b-8p4jt.log, istio-telemetry-67bbd74f6d-7lv68.log`\r\n\r\n2. segfaults:  I suspect that envoy could not handle the fact CDS delivered mixer_check_server and mixer_report_server clusters were removed temporarily. Perhaps mixer client should be resilient.\r\n\r\n```\r\n[2018-05-26 12:22:38.797][119][info][upstream] external/envoy/source/server/lds_subscription.cc:70] lds: fetch failure: network error\r\n[2018-05-26 12:22:40.829][119][info][upstream] external/envoy/source/server/lds_subscription.cc:70] lds: fetch failure: network error\r\n[2018-05-26 12:22:43.616][119][info][upstream] external/envoy/source/server/lds_subscription.cc:70] lds: fetch failure: network error\r\n[2018-05-26 12:22:46.181][119][info][upstream] external/envoy/source/server/lds_subscription.cc:70] lds: fetch failure: network error\r\n[2018-05-26 12:22:48.945][119][info][upstream] external/envoy/source/server/lds_subscription.cc:70] lds: fetch failure: network error\r\n[2018-05-26 12:22:51.276][119][warning][upstream] external/envoy/source/common/config/grpc_mux_impl.cc:217] gRPC config stream closed: 1,\r\n[2018-05-26 12:22:51.276][119][warning][upstream] external/envoy/source/common/config/grpc_mux_impl.cc:217] gRPC config stream closed: 1,\r\n[2018-05-26 12:22:51.276][119][warning][upstream] external/envoy/source/common/config/grpc_mux_impl.cc:217] gRPC config stream closed: 1,\r\n[2018-05-26 12:22:51.276][119][warning][upstream] external/envoy/source/common/config/grpc_mux_impl.cc:217] gRPC config stream closed: 1,\r\n[2018-05-26 12:22:51.276][119][warning][upstream] external/envoy/source/common/config/grpc_mux_impl.cc:217] gRPC config stream closed: 1,\r\n[2018-05-26 12:22:51.633][119][info][upstream] external/envoy/source/server/lds_subscription.cc:70] lds: fetch failure: network error\r\n[2018-05-26 12:22:54.008][119][info][upstream] external/envoy/source/server/lds_subscription.cc:70] lds: fetch failure: network error\r\n[2018-05-26 12:22:56.872][119][info][upstream] external/envoy/source/server/lds_subscription.cc:70] lds: fetch failure: network error\r\n[2018-05-26 12:22:59.001][119][info][upstream] external/envoy/source/server/lds_subscription.cc:70] lds: fetch failure: network error\r\n[2018-05-26 12:23:00.809][119][info][upstream] external/envoy/source/server/lds_subscription.cc:70] lds: fetch failure: network error\r\n[2018-05-26 12:23:02.973][119][info][upstream] external/envoy/source/server/lds_subscription.cc:70] lds: fetch failure: network error\r\n[2018-05-26 12:23:05.732][119][info][upstream] external/envoy/source/server/lds_subscription.cc:70] lds: fetch failure: network error\r\n[2018-05-26 12:23:06.284][119][warning][upstream] external/envoy/source/common/config/grpc_mux_impl.cc:217] gRPC config stream closed: 1,\r\n[2018-05-26 12:23:06.285][119][warning][upstream] external/envoy/source/common/config/grpc_mux_impl.cc:217] gRPC config stream closed: 1,\r\n[2018-05-26 12:23:06.285][119][warning][upstream] external/envoy/source/common/config/grpc_mux_impl.cc:217] gRPC config stream closed: 1,\r\n[2018-05-26 12:23:06.285][119][warning][upstream] external/envoy/source/common/config/grpc_mux_impl.cc:217] gRPC config stream closed: 1,\r\n[2018-05-26 12:23:06.285][119][warning][upstream] external/envoy/source/common/config/grpc_mux_impl.cc:217] gRPC config stream closed: 1,\r\n[2018-05-26 12:23:07.744][119][info][upstream] external/envoy/source/server/lds_subscription.cc:70] lds: fetch failure: network error\r\n[2018-05-26 12:23:09.649][119][info][upstream] external/envoy/source/server/lds_subscription.cc:70] lds: fetch failure: network error\r\n[2018-05-26 12:23:12.169][119][info][upstream] external/envoy/source/server/lds_subscription.cc:70] lds: fetch failure: network error\r\n[2018-05-26 12:23:14.185][119][info][upstream] external/envoy/source/server/lds_subscription.cc:70] lds: fetch failure: network error\r\n[2018-05-26 12:23:16.536][119][info][upstream] external/envoy/source/server/lds_subscription.cc:70] lds: fetch failure: network error\r\n[2018-05-26 12:23:18.861][119][info][upstream] external/envoy/source/server/lds_subscription.cc:70] lds: fetch failure: network error\r\n[2018-05-26 12:23:21.057][119][info][upstream] external/envoy/source/server/lds_subscription.cc:70] lds: fetch failure: network error\r\n[2018-05-26 12:23:21.293][119][warning][upstream] external/envoy/source/common/config/grpc_mux_impl.cc:217] gRPC config stream closed: 1,\r\n[2018-05-26 12:23:21.293][119][warning][upstream] external/envoy/source/common/config/grpc_mux_impl.cc:217] gRPC config stream closed: 1,\r\n[2018-05-26 12:23:21.293][119][warning][upstream] external/envoy/source/common/config/grpc_mux_impl.cc:217] gRPC config stream closed: 1,\r\n[2018-05-26 12:23:21.293][119][warning][upstream] external/envoy/source/common/config/grpc_mux_impl.cc:217] gRPC config stream closed: 1,\r\n[2018-05-26 12:23:21.293][119][warning][upstream] external/envoy/source/common/config/grpc_mux_impl.cc:217] gRPC config stream closed: 1,\r\n[2018-05-26 12:23:23.649][119][info][upstream] external/envoy/source/server/lds_subscription.cc:70] lds: fetch failure: network error\r\n[2018-05-26 12:23:25.453][119][info][upstream] external/envoy/source/server/lds_subscription.cc:70] lds: fetch failure: network error\r\n[2018-05-26 12:23:27.813][119][info][upstream] external/envoy/source/server/lds_subscription.cc:70] lds: fetch failure: network error\r\n[2018-05-26 12:23:29.845][119][info][upstream] external/envoy/source/server/lds_subscription.cc:70] lds: fetch failure: network error\r\n[2018-05-26 12:23:32.813][119][info][upstream] external/envoy/source/server/lds_subscription.cc:70] lds: fetch failure: network error\r\n[2018-05-26 12:23:33.391][119][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:437] removing cluster mixer_check_server\r\n[2018-05-26 12:23:33.392][119][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:437] removing cluster mixer_report_server\r\n[2018-05-26 12:23:34.121][119][info][upstream] external/envoy/source/server/lds_api.cc:61] lds: add/update listener 'http_0.0.0.0_80'\r\n\r\n[2018-05-26 12:23:35.830][119][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] Caught Segmentation fault, suspect faulting address 0x0\r\n[2018-05-26 12:23:35.831][119][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:90] Backtrace obj</usr/local/bin/envoy> thr<0> (use tools/stack_decode.py):\r\n[2018-05-26 12:23:35.831][119][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:107] thr<0> #0 0x994558\r\n[2018-05-26 12:23:35.831][119][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:107] thr<0> #1 0x995dd1\r\n[2018-05-26 12:23:35.831][119][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:107] thr<0> #2 0x99655d\r\n[2018-05-26 12:23:35.831][119][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:107] thr<0> #3 0x996dbd\r\n[2018-05-26 12:23:35.831][119][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:107] thr<0> #4 0x9979ea\r\n[2018-05-26 12:23:35.831][119][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:107] thr<0> #5 0x7430ad\r\n[2018-05-26 12:23:35.831][119][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:107] thr<0> #6 0x748054\r\n[2018-05-26 12:23:35.831][119][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:107] thr<0> #7 0x84a791\r\n[2018-05-26 12:23:35.831][119][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:107] thr<0> #8 0x821425\r\n[2018-05-26 12:23:35.831][119][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:107] thr<0> #9 0x8247d0\r\n[2018-05-26 12:23:35.831][119][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:107] thr<0> #10 0x8263cd\r\n[2018-05-26 12:23:35.831][119][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:107] thr<0> #11 0x8cb65b\r\n[2018-05-26 12:23:35.831][119][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:107] thr<0> #12 0x8cb03d\r\n[2018-05-26 12:23:35.831][119][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:107] thr<0> #13 0x8d044a\r\n[2018-05-26 12:23:35.831][119][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:107] thr<0> #14 0x8cc780\r\n[2018-05-26 12:23:35.831][119][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:107] thr<0> #15 0x8ce90a\r\n[2018-05-26 12:23:35.831][119][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:107] thr<0> #16 0x860f1e\r\n[2018-05-26 12:23:35.831][119][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:107] thr<0> #17 0x86109c\r\n[2018-05-26 12:23:35.831][119][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:107] thr<0> #18 0x733af6\r\n[2018-05-26 12:23:35.831][119][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:107] thr<0> #19 0x7320de\r\n[2018-05-26 12:23:35.831][119][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:107] thr<0> #20 0x7328fd\r\n[2018-05-26 12:23:35.831][119][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:107] thr<0> #21 0x72c287\r\n[2018-05-26 12:23:35.831][119][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:107] thr<0> #22 0xa14431\r\n[2018-05-26 12:23:35.831][119][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:107] thr<0> #23 0xa14b8e\r\n[2018-05-26 12:23:35.831][119][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:107] thr<0> #24 0x70976d\r\n[2018-05-26 12:23:35.831][119][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:107] thr<0> #25 0x567790\r\n[2018-05-26 12:23:35.831][119][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:107] thr<0> #26 0x421068\r\n[2018-05-26 12:23:35.831][119][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:99] thr<0> obj</lib/x86_64-linux-gnu/libc.so.6>\r\n[2018-05-26 12:23:35.831][119][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:107] thr<0> #27 0x2b1adddd582f\r\n[2018-05-26 12:23:35.831][119][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:99] thr<0> obj</usr/local/bin/envoy>\r\n[2018-05-26 12:23:35.831][119][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:107] thr<0> #28 0x46a828\r\n[2018-05-26 12:23:35.831][119][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:110] end backtrace thread 0\r\n2018-05-26T12:23:35.840806Z     warn    Epoch 0 terminated with an error: signal: segmentation fault\r\n2018-05-26T12:23:35.840843Z     warn    Aborted all epochs\r\n```\r\n\r\n\r\n\r\n \r\n"},"comment":{"url":"https://api.github.com/repos/istio/istio/issues/comments/392893366","html_url":"https://github.com/istio/istio/issues/5864#issuecomment-392893366","issue_url":"https://api.github.com/repos/istio/istio/issues/5864","id":392893366,"user":{"login":"costinm","id":84202,"avatar_url":"https://avatars1.githubusercontent.com/u/84202?v=4","gravatar_id":"","url":"https://api.github.com/users/costinm","html_url":"https://github.com/costinm","followers_url":"https://api.github.com/users/costinm/followers","following_url":"https://api.github.com/users/costinm/following{/other_user}","gists_url":"https://api.github.com/users/costinm/gists{/gist_id}","starred_url":"https://api.github.com/users/costinm/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/costinm/subscriptions","organizations_url":"https://api.github.com/users/costinm/orgs","repos_url":"https://api.github.com/users/costinm/repos","events_url":"https://api.github.com/users/costinm/events{/privacy}","received_events_url":"https://api.github.com/users/costinm/received_events","type":"User","site_admin":false},"created_at":"2018-05-29T18:46:26Z","updated_at":"2018-05-29T18:46:26Z","author_association":"CONTRIBUTOR","body":"If they don't want to move: original ingress is still using v1 and should\nbehave identically with a 0.7.1, i.e. envoy has the same\nbootstrap and gets same config as it did in 0.7.1\n\n\nOn Tue, May 29, 2018 at 10:18 AM, mandarjog <notifications@github.com>\nwrote:\n\n> Using a fresh - v2 install, regpatrol is able to get new results. I am\n> removing \"release blocking\".\n>\n> This is still an issue of documenting how to upgrade when using 0.7\n> ingress to 0.8.\n> We should consider both cases, when user wants to move from ingress\n> resource to gateway and when the user does not want to move.\n>\n> —\n> You are receiving this because you were assigned.\n> Reply to this email directly, view it on GitHub\n> <https://github.com/istio/istio/issues/5864#issuecomment-392862306>, or mute\n> the thread\n> <https://github.com/notifications/unsubscribe-auth/AAFI6oV9pJ0mUj-kjrcOfuYacmi0s99mks5t3YLvgaJpZM4UPEXk>\n> .\n>\n"}}
{"actor":{"display_login":"costinm"}}