{"action":"created","issue":{"url":"https://api.github.com/repos/istio/proxy/issues/1789","repository_url":"https://api.github.com/repos/istio/proxy","labels_url":"https://api.github.com/repos/istio/proxy/issues/1789/labels{/name}","comments_url":"https://api.github.com/repos/istio/proxy/issues/1789/comments","events_url":"https://api.github.com/repos/istio/proxy/issues/1789/events","html_url":"https://github.com/istio/proxy/issues/1789","id":327517367,"number":1789,"title":"Frequent segmentation faults","user":{"login":"mattatcha","id":770790,"avatar_url":"https://avatars1.githubusercontent.com/u/770790?v=4","gravatar_id":"","url":"https://api.github.com/users/mattatcha","html_url":"https://github.com/mattatcha","followers_url":"https://api.github.com/users/mattatcha/followers","following_url":"https://api.github.com/users/mattatcha/following{/other_user}","gists_url":"https://api.github.com/users/mattatcha/gists{/gist_id}","starred_url":"https://api.github.com/users/mattatcha/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mattatcha/subscriptions","organizations_url":"https://api.github.com/users/mattatcha/orgs","repos_url":"https://api.github.com/users/mattatcha/repos","events_url":"https://api.github.com/users/mattatcha/events{/privacy}","received_events_url":"https://api.github.com/users/mattatcha/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2018-05-29T23:05:23Z","updated_at":"2018-05-29T23:13:22Z","closed_at":null,"author_association":"NONE","body":"Seems that segfaults only happen while proxy is handling http traffic.\r\n \r\nRunning istio/istio@d91d44ead27d9b4018ab5997ef7ec0f0a4621d9d on kubernetes.\r\n\r\nRaw backtrace\r\n```\r\n[libprotobuf INFO src/istio/mixerclient/check_cache.cc:155] Add a new Referenced for check cache: Absence-keys: Exact-keys: context.protocol, destination.service, destination.uid, source.ip, source.uid,\r\n[libprotobuf INFO src/istio/mixerclient/check_cache.cc:155] Add a new Referenced for check cache: Absence-keys: Exact-keys: context.protocol, destination.service, destination.uid, source.ip, source.uid,\r\n[2018-05-29 18:12:00.841][38][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] Caught Segmentation fault, suspect faulting address 0x0\r\n[2018-05-29 18:12:00.858][38][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:90] Backtrace obj</usr/local/bin/envoy> thr<38> (use tools/stack_decode.py):\r\n[2018-05-29 18:12:00.858][38][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:107] thr<38> #0 0x4db088\r\n[2018-05-29 18:12:00.858][38][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:107] thr<38> #1 0x4e69d2\r\n[2018-05-29 18:12:00.858][38][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:107] thr<38> #2 0x4e6d21\r\n[2018-05-29 18:12:00.858][38][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:107] thr<38> #3 0x4e6d87\r\n[2018-05-29 18:12:00.858][38][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:107] thr<38> #4 0x4e6e00\r\n[2018-05-29 18:12:00.858][38][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:107] thr<38> #5 0x4e5912\r\n[2018-05-29 18:12:00.858][38][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:107] thr<38> #6 0x4e59e0\r\n[2018-05-29 18:12:00.858][38][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:107] thr<38> #7 0x4aa1d8\r\n[2018-05-29 18:12:00.858][38][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:107] thr<38> #8 0x4aa210\r\n[2018-05-29 18:12:00.858][38][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:107] thr<38> #9 0x716859\r\n[2018-05-29 18:12:00.858][38][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:107] thr<38> #10 0x73e4fb\r\n[2018-05-29 18:12:00.858][38][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:107] thr<38> #11 0xa2d0b7\r\n[2018-05-29 18:12:00.858][38][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:107] thr<38> #12 0xa2d61e\r\n[2018-05-29 18:12:00.858][38][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:107] thr<38> #13 0x73ae75\r\n[2018-05-29 18:12:00.858][38][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:107] thr<38> #14 0xa75fdd\r\n[2018-05-29 18:12:00.858][38][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:99] thr<38> obj</lib/x86_64-linux-gnu/libpthread.so.0>\r\n[2018-05-29 18:12:00.858][38][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:107] thr<38> #15 0x7fe2fbc066b9\r\n[2018-05-29 18:12:00.858][38][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:99] thr<38> obj</lib/x86_64-linux-gnu/libc.so.6>\r\n[2018-05-29 18:12:00.858][38][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:107] thr<38> #16 0x7fe2fb63341c\r\n[2018-05-29 18:12:00.858][38][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:110] end backtrace thread 38\r\n2018-05-29T18:12:01.658404Z     warn    Epoch 0 terminated with an error: signal: segmentation fault (core dumped)\r\n2018-05-29T18:12:01.659389Z     warn    Aborted all epochs\r\n2018-05-29T18:12:01.659657Z     info    Epoch 0: set retry delay to 200ms, budget to 9\r\n2018-05-29T18:12:01.862365Z     info    Reconciling configuration (budget 9)\r\n2018-05-29T18:12:01.862486Z     info    Epoch 0 starting\r\n2018-05-29T18:12:01.863151Z     info    Envoy command: [-c /etc/istio/proxy/envoy-rev0.json --restart-epoch 0 --drain-time-s 45 --parent-shutdown-time-s 60 --service-cluster webhook --service-node sidecar~100.64.252.34~web-66f64f9cbc-flnrl.webhook~webhook.svc.cluster.local --max-obj-name-len 189 -l warn --v2-config-only]\r\n[2018-05-29 18:12:01.931][42][info][main] external/envoy/source/server/server.cc:181] initializing epoch 0 (hot restart version=9.200.16384.256.options=capacity=16384, num_slots=8209 hash=228984379728933363)\r\n[2018-05-29 18:12:01.931][42][info][main] external/envoy/source/server/server.cc:183] statically linked extensions:\r\n[2018-05-29 18:12:01.931][42][info][main] external/envoy/source/server/server.cc:185]   access_loggers: envoy.file_access_log,envoy.http_grpc_access_log\r\n[2018-05-29 18:12:01.931][42][info][main] external/envoy/source/server/server.cc:188]   filters.http: envoy.buffer,envoy.cors,envoy.ext_authz,envoy.fault,envoy.grpc_http1_bridge,envoy.grpc_json_transcoder,envoy.grpc_web,envoy.gzip,envoy.health_check,envoy.http_dynamo_filter,envoy.ip_tagging,envoy.lua,envoy.rate_limit,envoy.router,envoy.squash,istio_authn,jwt-auth,mixer\r\n[2018-05-29 18:12:01.931][42][info][main] external/envoy/source/server/server.cc:191]   filters.listener: envoy.listener.original_dst,envoy.listener.proxy_protocol,envoy.listener.tls_inspector\r\n[2018-05-29 18:12:01.931][42][info][main] external/envoy/source/server/server.cc:194]   filters.network: envoy.client_ssl_auth,envoy.echo,envoy.ext_authz,envoy.http_connection_manager,envoy.mongo_proxy,envoy.ratelimit,envoy.redis_proxy,envoy.tcp_proxy,mixer\r\n[2018-05-29 18:12:01.931][42][info][main] external/envoy/source/server/server.cc:196]   stat_sinks: envoy.dog_statsd,envoy.metrics_service,envoy.statsd\r\n[2018-05-29 18:12:01.931][42][info][main] external/envoy/source/server/server.cc:198]   tracers: envoy.dynamic.ot,envoy.lightstep,envoy.zipkin\r\n[2018-05-29 18:12:01.931][42][info][main] external/envoy/source/server/server.cc:201]   transport_sockets.downstream: alts,envoy.transport_sockets.capture,raw_buffer,tls\r\n[2018-05-29 18:12:01.931][42][info][main] external/envoy/source/server/server.cc:204]   transport_sockets.upstream: alts,envoy.transport_sockets.capture,raw_buffer,tls\r\n[2018-05-29 18:12:01.999][42][warning][upstream] external/envoy/source/common/config/grpc_mux_impl.cc:235] gRPC config stream closed: 14, no healthy upstream\r\n[2018-05-29 18:12:01.999][42][warning][upstream] external/envoy/source/common/config/grpc_mux_impl.cc:39] Unable to establish new stream\r\n[2018-05-29 18:12:01.999][42][info][config] external/envoy/source/server/configuration_impl.cc:53] loading 0 listener(s)\r\n```\r\n\r\nDecoded\r\n```\r\n[libprotobuf INFO src/istio/mixerclient/check_cache.cc:155] Add a new Referenced for check cache: Absence-keys: Exact-keys: context.protocol, destination.service, destination.uid, source.ip, source.uid,\r\n[2018-05-29 18:12:00.841][38][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] Caught Segmentation fault, suspect faulting address 0x0\r\n[2018-05-29 18:12:00.858][38][critical] Backtrace (most recent call first) from thread 38:\r\n  #1 std::_Function_handler<std::function<void ()> (istio::mixer::v1::ReportRequest const&, istio::mixer::v1::ReportResponse*, std::function<void (google::protobuf::util::Status const&)>), Envoy::Utils::GrpcTransport<istio::mixer::v1::ReportRequest, istio::mixer::v1::ReportResponse>::GetFunc(Envoy::Grpc::AsyncClientFactory&, Envoy::Tracing::Span&)::{lambda(istio::mixer::v1::ReportRequest const&, istio::mixer::v1::ReportResponse*, std::function<void (google::protobuf::util::Status const&)>)#1}>::_M_invoke(std::_Any_data const&, istio::mixer::v1::ReportRequest const&, istio::mixer::v1::ReportResponse*&&, std::function<void (google::protobuf::util::Status const&)>&&) at ??:?\r\n  #2 istio::mixerclient::ReportBatch::FlushWithLock() at ??:?\r\n  #3 istio::mixerclient::ReportBatch::Flush() at ??:?\r\n  #4 istio::mixerclient::ReportBatch::~ReportBatch() at ??:?\r\n  #5 istio::mixerclient::ReportBatch::~ReportBatch() at ??:?\r\n  #6 istio::mixerclient::MixerClientImpl::~MixerClientImpl() at ??:?\r\n  #7 istio::mixerclient::MixerClientImpl::~MixerClientImpl() at ??:?\r\n  #8 istio::control::http::ControllerImpl::~ControllerImpl() at ??:?\r\n  #9 istio::control::http::ControllerImpl::~ControllerImpl() at ??:?\r\n  #10 std::_Function_handler<void (), Envoy::ThreadLocal::InstanceImpl::removeSlot(Envoy::ThreadLocal::InstanceImpl::SlotImpl&)::{lambda()#1}>::_M_invoke(std::_Any_data const&) at thread_local_impl.cc:?\r\n  #11 Envoy::Event::DispatcherImpl::runPostCallbacks() at ??:?\r\n  #12 event_process_active_single_queue.isra.29 at event.c:?\r\n  #13 event_base_loop at ??:?\r\n  #14 Envoy::Server::WorkerImpl::threadRoutine(Envoy::Server::GuardDog&) at ??:?\r\n  #15 Envoy::Thread::Thread::Thread(std::function<void ()>)::{lambda(void*)#1}::_FUN(void*) at thread.cc:?\r\n  #16\r\n  #17 ?? ??:0\r\n  #18\r\n  #19 ?? ??:0\r\n  #20\r\n2018-05-29T18:12:01.658404Z     warn    Epoch 0 terminated with an error: signal: segmentation fault (core dumped)\r\n```"},"comment":{"url":"https://api.github.com/repos/istio/proxy/issues/comments/392977638","html_url":"https://github.com/istio/proxy/issues/1789#issuecomment-392977638","issue_url":"https://api.github.com/repos/istio/proxy/issues/1789","id":392977638,"user":{"login":"mattatcha","id":770790,"avatar_url":"https://avatars1.githubusercontent.com/u/770790?v=4","gravatar_id":"","url":"https://api.github.com/users/mattatcha","html_url":"https://github.com/mattatcha","followers_url":"https://api.github.com/users/mattatcha/followers","following_url":"https://api.github.com/users/mattatcha/following{/other_user}","gists_url":"https://api.github.com/users/mattatcha/gists{/gist_id}","starred_url":"https://api.github.com/users/mattatcha/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mattatcha/subscriptions","organizations_url":"https://api.github.com/users/mattatcha/orgs","repos_url":"https://api.github.com/users/mattatcha/repos","events_url":"https://api.github.com/users/mattatcha/events{/privacy}","received_events_url":"https://api.github.com/users/mattatcha/received_events","type":"User","site_admin":false},"created_at":"2018-05-29T23:13:22Z","updated_at":"2018-05-29T23:13:22Z","author_association":"NONE","body":"It also seems like multiple istio-proxys segfault at the same time causing bursts of 503s identical to https://groups.google.com/forum/#!topic/istio-users/WWZf9ehaBdI.\r\n\r\n"}}
{"actor":{"display_login":"mattatcha"}}