{"action":"created","issue":{"url":"https://api.github.com/repos/istio/istio/issues/7817","repository_url":"https://api.github.com/repos/istio/istio","labels_url":"https://api.github.com/repos/istio/istio/issues/7817/labels{/name}","comments_url":"https://api.github.com/repos/istio/istio/issues/7817/comments","events_url":"https://api.github.com/repos/istio/istio/issues/7817/events","html_url":"https://github.com/istio/istio/issues/7817","id":349642564,"node_id":"MDU6SXNzdWUzNDk2NDI1NjQ=","number":7817,"title":"istio-proxy extremely slow start time (consistently 60s+)","user":{"login":"Stono","id":1486729,"node_id":"MDQ6VXNlcjE0ODY3Mjk=","avatar_url":"https://avatars2.githubusercontent.com/u/1486729?v=4","gravatar_id":"","url":"https://api.github.com/users/Stono","html_url":"https://github.com/Stono","followers_url":"https://api.github.com/users/Stono/followers","following_url":"https://api.github.com/users/Stono/following{/other_user}","gists_url":"https://api.github.com/users/Stono/gists{/gist_id}","starred_url":"https://api.github.com/users/Stono/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Stono/subscriptions","organizations_url":"https://api.github.com/users/Stono/orgs","repos_url":"https://api.github.com/users/Stono/repos","events_url":"https://api.github.com/users/Stono/events{/privacy}","received_events_url":"https://api.github.com/users/Stono/received_events","type":"User","site_admin":false},"labels":[{"id":735781806,"node_id":"MDU6TGFiZWw3MzU3ODE4MDY=","url":"https://api.github.com/repos/istio/istio/labels/area/networking","name":"area/networking","color":"0e8a16","default":false},{"id":822212191,"node_id":"MDU6TGFiZWw4MjIyMTIxOTE=","url":"https://api.github.com/repos/istio/istio/labels/kind/customer%20issue","name":"kind/customer issue","color":"fbca04","default":false}],"state":"open","locked":false,"assignee":{"login":"costinm","id":84202,"node_id":"MDQ6VXNlcjg0MjAy","avatar_url":"https://avatars1.githubusercontent.com/u/84202?v=4","gravatar_id":"","url":"https://api.github.com/users/costinm","html_url":"https://github.com/costinm","followers_url":"https://api.github.com/users/costinm/followers","following_url":"https://api.github.com/users/costinm/following{/other_user}","gists_url":"https://api.github.com/users/costinm/gists{/gist_id}","starred_url":"https://api.github.com/users/costinm/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/costinm/subscriptions","organizations_url":"https://api.github.com/users/costinm/orgs","repos_url":"https://api.github.com/users/costinm/repos","events_url":"https://api.github.com/users/costinm/events{/privacy}","received_events_url":"https://api.github.com/users/costinm/received_events","type":"User","site_admin":false},"assignees":[{"login":"costinm","id":84202,"node_id":"MDQ6VXNlcjg0MjAy","avatar_url":"https://avatars1.githubusercontent.com/u/84202?v=4","gravatar_id":"","url":"https://api.github.com/users/costinm","html_url":"https://github.com/costinm","followers_url":"https://api.github.com/users/costinm/followers","following_url":"https://api.github.com/users/costinm/following{/other_user}","gists_url":"https://api.github.com/users/costinm/gists{/gist_id}","starred_url":"https://api.github.com/users/costinm/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/costinm/subscriptions","organizations_url":"https://api.github.com/users/costinm/orgs","repos_url":"https://api.github.com/users/costinm/repos","events_url":"https://api.github.com/users/costinm/events{/privacy}","received_events_url":"https://api.github.com/users/costinm/received_events","type":"User","site_admin":false},{"login":"PiotrSikora","id":190297,"node_id":"MDQ6VXNlcjE5MDI5Nw==","avatar_url":"https://avatars3.githubusercontent.com/u/190297?v=4","gravatar_id":"","url":"https://api.github.com/users/PiotrSikora","html_url":"https://github.com/PiotrSikora","followers_url":"https://api.github.com/users/PiotrSikora/followers","following_url":"https://api.github.com/users/PiotrSikora/following{/other_user}","gists_url":"https://api.github.com/users/PiotrSikora/gists{/gist_id}","starred_url":"https://api.github.com/users/PiotrSikora/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/PiotrSikora/subscriptions","organizations_url":"https://api.github.com/users/PiotrSikora/orgs","repos_url":"https://api.github.com/users/PiotrSikora/repos","events_url":"https://api.github.com/users/PiotrSikora/events{/privacy}","received_events_url":"https://api.github.com/users/PiotrSikora/received_events","type":"User","site_admin":false}],"milestone":null,"comments":3,"created_at":"2018-08-10T20:09:45Z","updated_at":"2018-08-12T11:49:59Z","closed_at":null,"author_association":"CONTRIBUTOR","body":"**Describe the bug**\r\nI am seeing applications have extremely slow startup times due to istio-proxy taking absolutely ages to come online.   This is extremely troublesome, as it:\r\n\r\n  - Causes issues for applications trying to talk outbound after booting\r\n  - Significantly slows down deployments\r\n  - Significantly impacts scaling up, especially via HPA\r\n\r\nFor example, this applications failed to connect to redis for around 90 seconds:\r\n```\r\n❯ kn logs oauth2-proxy-768b67f6f4-2tjjt master\r\n2018-08-10T19:45:41.298Z at:oauth2-proxy:server:web starting server on port 4180\r\n2018-08-10T19:45:41.390Z at:oauth2-proxy:server:metrics starting metrics server on port 9090\r\n2018-08-10T19:45:41.395Z at:oauth2-proxy:server:metrics metrics server started\r\n1 'Redis connection to oauth-redis:6379 failed - connect ECONNREFUSED 10.192.8.230:6379'\r\n2 'Redis connection to oauth-redis:6379 failed - connect ECONNREFUSED 10.192.8.230:6379'\r\n3 'Redis connection to oauth-redis:6379 failed - connect ECONNREFUSED 10.192.8.230:6379'\r\n4 'Redis connection to oauth-redis:6379 failed - connect ECONNREFUSED 10.192.8.230:6379'\r\n5 'Redis connection to oauth-redis:6379 failed - connect ECONNREFUSED 10.192.8.230:6379'\r\n6 'Redis connection to oauth-redis:6379 failed - connect ECONNREFUSED 10.192.8.230:6379'\r\n7 'Redis connection to oauth-redis:6379 failed - connect ECONNREFUSED 10.192.8.230:6379'\r\n8 'Redis connection to oauth-redis:6379 failed - connect ECONNREFUSED 10.192.8.230:6379'\r\n9 'Redis connection to oauth-redis:6379 failed - connect ECONNREFUSED 10.192.8.230:6379'\r\n10 'Redis connection to oauth-redis:6379 failed - connect ECONNREFUSED 10.192.8.230:6379'\r\n11 'Redis connection to oauth-redis:6379 failed - connect ECONNREFUSED 10.192.8.230:6379'\r\n2018-08-10T19:47:19.350Z at:oauth2-proxy:clients:azure Discovered issuer, generating oauth client\r\n```\r\n\r\nLooking at the istio-pilot logs, you can see how long it took to start up, around 70 seconds to get to `all dependencies initialized. starting workers`.\r\n\r\n```\r\n2018-08-10T19:45:38.539521Z\tinfo\tMonitored certs: []envoy.CertSource{envoy.CertSource{Directory:\"/etc/certs/\", Files:[]string{\"cert-chain.pem\", \"key.pem\", \"root-cert.pem\"}}}\r\n2018-08-10T19:45:38.539934Z\tinfo\tStarting proxy agent\r\n2018-08-10T19:45:38.540004Z\tinfo\tReceived new config, resetting budget\r\n2018-08-10T19:45:38.540022Z\tinfo\tReconciling configuration (budget 10)\r\n2018-08-10T19:45:38.540040Z\tinfo\tEpoch 0 starting\r\n2018-08-10T19:45:38.540676Z\tinfo\tEnvoy command: [-c /etc/istio/proxy/envoy-rev0.json --restart-epoch 0 --drain-time-s 45 --parent-shutdown-time-s 60 --service-cluster oauth2-proxy --service-node sidecar~10.194.2.82~oauth2-proxy-768b67f6f4-2tjjt.ingress-nginx~ingress-nginx.svc.cluster.local --max-obj-name-len 189 -l warn --v2-config-only]\r\n[2018-08-10 19:45:38.557][23][info][main] external/envoy/source/server/server.cc:183] initializing epoch 0 (hot restart version=10.200.16384.256.options=capacity=16384, num_slots=8209 hash=228984379728933363 size=4882536)\r\n[2018-08-10 19:45:38.557][23][info][main] external/envoy/source/server/server.cc:185] statically linked extensions:\r\n[2018-08-10 19:45:38.557][23][info][main] external/envoy/source/server/server.cc:187]   access_loggers: envoy.file_access_log,envoy.http_grpc_access_log\r\n[2018-08-10 19:45:38.557][23][info][main] external/envoy/source/server/server.cc:190]   filters.http: envoy.buffer,envoy.cors,envoy.ext_authz,envoy.fault,envoy.filters.http.header_to_metadata,envoy.filters.http.jwt_authn,envoy.filters.http.rbac,envoy.grpc_http1_bridge,envoy.grpc_json_transcoder,envoy.grpc_web,envoy.gzip,envoy.health_check,envoy.http_dynamo_filter,envoy.ip_tagging,envoy.lua,envoy.rate_limit,envoy.router,envoy.squash,istio_authn,jwt-auth,mixer\r\n[2018-08-10 19:45:38.557][23][info][main] external/envoy/source/server/server.cc:193]   filters.listener: envoy.listener.original_dst,envoy.listener.proxy_protocol,envoy.listener.tls_inspector\r\n[2018-08-10 19:45:38.558][23][info][main] external/envoy/source/server/server.cc:196]   filters.network: envoy.client_ssl_auth,envoy.echo,envoy.ext_authz,envoy.filters.network.thrift_proxy,envoy.http_connection_manager,envoy.mongo_proxy,envoy.ratelimit,envoy.redis_proxy,envoy.tcp_proxy,mixer\r\n[2018-08-10 19:45:38.558][23][info][main] external/envoy/source/server/server.cc:198]   stat_sinks: envoy.dog_statsd,envoy.metrics_service,envoy.stat_sinks.hystrix,envoy.statsd\r\n[2018-08-10 19:45:38.558][23][info][main] external/envoy/source/server/server.cc:200]   tracers: envoy.dynamic.ot,envoy.lightstep,envoy.zipkin\r\n[2018-08-10 19:45:38.558][23][info][main] external/envoy/source/server/server.cc:203]   transport_sockets.downstream: alts,envoy.transport_sockets.capture,raw_buffer,tls\r\n[2018-08-10 19:45:38.558][23][info][main] external/envoy/source/server/server.cc:206]   transport_sockets.upstream: alts,envoy.transport_sockets.capture,raw_buffer,tls\r\n[2018-08-10 19:45:38.560][23][info][config] external/envoy/source/server/configuration_impl.cc:50] loading 0 static secret(s)\r\n[2018-08-10 19:45:38.565][23][warning][upstream] external/envoy/source/common/config/grpc_mux_impl.cc:240] gRPC config stream closed: 14, no healthy upstream\r\n[2018-08-10 19:45:38.565][23][warning][upstream] external/envoy/source/common/config/grpc_mux_impl.cc:41] Unable to establish new stream\r\n[2018-08-10 19:45:38.565][23][info][config] external/envoy/source/server/configuration_impl.cc:60] loading 0 listener(s)\r\n[2018-08-10 19:45:38.565][23][info][config] external/envoy/source/server/configuration_impl.cc:94] loading tracing configuration\r\n[2018-08-10 19:45:38.565][23][info][config] external/envoy/source/server/configuration_impl.cc:103]   loading tracing driver: envoy.zipkin\r\n[2018-08-10 19:45:38.565][23][info][config] external/envoy/source/server/configuration_impl.cc:116] loading stats sink configuration\r\n[2018-08-10 19:45:38.565][23][info][main] external/envoy/source/server/server.cc:410] starting main dispatch loop\r\n[2018-08-10 19:45:38.599][23][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:129] cm init: initializing cds\r\n[2018-08-10 19:45:50.218][23][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:388] add/update cluster outbound|53||kube-dns.kube-system.svc.cluster.local during init\r\n[2018-08-10 19:45:50.219][23][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:388] add/update cluster outbound|9094||alertmanager.data-platform.svc.cluster.local during init\r\n[2018-08-10 19:45:50.219][23][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:388] add/update cluster outbound|6379||oauth-redis.ingress-nginx.svc.cluster.local during init\r\n[2018-08-10 19:45:50.221][23][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:388] add/update cluster outbound|2181||zookeeper-headless.search-solr.svc.cluster.local during init\r\n[2018-08-10 19:45:50.222][23][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:388] add/update cluster outbound|2888||zookeeper-headless.search-solr.svc.cluster.local during init\r\n[2018-08-10 19:45:50.223][23][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:388] add/update cluster outbound|3888||zookeeper-headless.search-solr.svc.cluster.local during init\r\n[2018-08-10 19:45:50.224][23][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:388] add/update cluster outbound|80||app.conference-application.svc.cluster.local during init\r\n[2018-08-10 19:45:50.226][23][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:388] add/update cluster outbound|9080||app.conference-application.svc.cluster.local during init\r\n[2018-08-10 19:45:50.227][23][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:388] add/update cluster outbound|80||ingress-nginx-internal.ingress-nginx.svc.cluster.local during init\r\n[2018-08-10 19:45:50.228][23][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:388] add/update cluster outbound|443||ingress-nginx-internal.ingress-nginx.svc.cluster.local during init\r\n[2018-08-10 19:45:50.229][23][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:388] add/update cluster outbound|80||cloudwatch-exporter.data-platform.svc.cluster.local during init\r\n[2018-08-10 19:45:50.230][23][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:388] add/update cluster outbound|80||kubeops.core-system.svc.cluster.local during init\r\n[2018-08-10 19:45:50.231][23][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:388] add/update cluster outbound|44134||tiller-deploy.kube-system.svc.cluster.local during init\r\n[2018-08-10 19:45:50.231][23][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:388] add/update cluster outbound|3000||grafana.istio-system.svc.cluster.local during init\r\n[2018-08-10 19:45:50.232][23][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:388] add/update cluster outbound|80||default-http-backend.ingress-nginx.svc.cluster.local during init\r\n[2018-08-10 19:45:50.232][23][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:388] add/update cluster outbound|5775||jaeger-agent.istio-system.svc.cluster.local during init\r\n[2018-08-10 19:45:50.233][23][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:388] add/update cluster outbound|6831||jaeger-agent.istio-system.svc.cluster.local during init\r\n[2018-08-10 19:45:50.233][23][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:388] add/update cluster outbound|6832||jaeger-agent.istio-system.svc.cluster.local during init\r\n[2018-08-10 19:45:50.234][23][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:388] add/update cluster outbound|80||app.mercury-admin-tool.svc.cluster.local during init\r\n[2018-08-10 19:45:50.235][23][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:388] add/update cluster outbound|80||oauth2-proxy.ingress-nginx.svc.cluster.local during init\r\n[2018-08-10 19:45:50.236][23][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:388] add/update cluster outbound|80||app.supply-and-demand-service.svc.cluster.local during init\r\n[2018-08-10 19:45:50.238][23][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:388] add/update cluster outbound|9080||app.supply-and-demand-service.svc.cluster.local during init\r\n[2018-08-10 19:45:50.238][23][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:388] add/update cluster outbound|9093||prometheus-alertmanager-frontend.istio-system.svc.cluster.local during init\r\n[2018-08-10 19:45:50.238][23][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:388] add/update cluster outbound|9090||prometheus.istio-system.svc.cluster.local during init\r\n[2018-08-10 19:45:50.239][23][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:388] add/update cluster outbound|8080||solr-headless.search-solr.svc.cluster.local during init\r\n[2018-08-10 19:45:50.240][23][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:388] add/update cluster outbound|80||app.vehicle-metric-service.svc.cluster.local during init\r\n[2018-08-10 19:45:50.242][23][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:388] add/update cluster outbound|8060||istio-citadel.istio-system.svc.cluster.local during init\r\n[2018-08-10 19:45:50.243][23][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:388] add/update cluster outbound|9093||istio-citadel.istio-system.svc.cluster.local during init\r\n[2018-08-10 19:45:50.244][23][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:388] add/update cluster outbound|9102||istio-statsd-prom-bridge.istio-system.svc.cluster.local during\r\n[2018-08-10 19:45:50.245][23][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:388] add/update cluster outbound|9125||istio-statsd-prom-bridge.istio-system.svc.cluster.local during init\r\n[2018-08-10 19:45:50.247][23][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:388] add/update cluster outbound|443||metrics-server.kube-system.svc.cluster.local during init\r\n[2018-08-10 19:45:50.248][23][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:388] add/update cluster outbound|80||cloudwatch-billing-exporter.data-platform.svc.cluster.local during init\r\n[2018-08-10 19:45:50.249][23][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:388] add/update cluster outbound|443||istio-sidecar-injector.istio-system.svc.cluster.local during init\r\n[2018-08-10 19:45:50.249][23][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:388] add/update cluster outbound|80||prometheus-pushgateway.istio-system.svc.cluster.local during init\r\n[2018-08-10 19:45:50.249][23][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:388] add/update cluster outbound|80||weave-scope.core-system.svc.cluster.local during init\r\n[2018-08-10 19:45:50.251][23][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:388] add/update cluster outbound|80||app.stock-roi-tool.svc.cluster.local during init\r\n[2018-08-10 19:45:50.251][23][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:388] add/update cluster outbound|80||solr.search-solr.svc.cluster.local during init\r\n[2018-08-10 19:45:50.252][23][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:388] add/update cluster outbound|80||ingress-nginx-external.ingress-nginx.svc.cluster.local during init\r\n[2018-08-10 19:45:50.253][23][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:388] add/update cluster outbound|443||ingress-nginx-external.ingress-nginx.svc.cluster.local during init\r\n[2018-08-10 19:45:50.254][23][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:388] add/update cluster outbound|80||heapster.kube-system.svc.cluster.local during init\r\n[2018-08-10 19:45:50.256][23][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:388] add/update cluster outbound|80||app.platform-dashboard.svc.cluster.local during init\r\n[2018-08-10 19:45:50.258][23][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:388] add/update cluster outbound|80||app.recruitment-website.svc.cluster.local during init\r\n[2018-08-10 19:45:50.259][23][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:388] add/update cluster outbound|9080||app.recruitment-website.svc.cluster.local during init\r\n[2018-08-10 19:45:50.259][23][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:388] add/update cluster outbound|80||zookeeper.search-solr.svc.cluster.local during init\r\n[2018-08-10 19:45:50.259][23][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:388] add/update cluster outbound|3000||grafana.data-platform.svc.cluster.local during init\r\n[2018-08-10 19:45:50.260][23][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:388] add/update cluster outbound|9091||istio-policy.istio-system.svc.cluster.local during init\r\n[2018-08-10 19:45:50.261][23][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:388] add/update cluster outbound|15004||istio-policy.istio-system.svc.cluster.local during init\r\n[2018-08-10 19:45:50.262][23][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:388] add/update cluster outbound|9093||istio-policy.istio-system.svc.cluster.local during init\r\n[2018-08-10 19:45:50.263][23][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:388] add/update cluster outbound|80||app.rate-of-sale-model-service.svc.cluster.local during init\r\n[2018-08-10 19:45:50.264][23][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:388] add/update cluster outbound|9080||app.rate-of-sale-model-service.svc.cluster.local during init\r\n[2018-08-10 19:45:50.264][23][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:388] add/update cluster outbound|9090||prometheus.data-platform.svc.cluster.local during init\r\n[2018-08-10 19:45:50.265][23][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:388] add/update cluster outbound|80||app.build-status.svc.cluster.local during init\r\n[2018-08-10 19:45:50.267][23][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:388] add/update cluster outbound|9080||app.build-status.svc.cluster.local during init\r\n[2018-08-10 19:45:50.267][23][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:388] add/update cluster outbound|9093||alertmanager-frontend.data-platform.svc.cluster.local during init\r\n[2018-08-10 19:45:50.268][23][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:388] add/update cluster outbound|15010||istio-pilot.istio-system.svc.cluster.local during init\r\n[2018-08-10 19:45:50.269][23][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:388] add/update cluster outbound|15011||istio-pilot.istio-system.svc.cluster.local during init\r\n[2018-08-10 19:45:50.270][23][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:388] add/update cluster outbound|8080||istio-pilot.istio-system.svc.cluster.local during init\r\n[2018-08-10 19:45:50.272][23][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:388] add/update cluster outbound|9093||istio-pilot.istio-system.svc.cluster.local during init\r\n[2018-08-10 19:45:50.273][23][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:388] add/update cluster outbound|80||app.consumer-platform.svc.cluster.local during init\r\n[2018-08-10 19:45:50.273][23][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:388] add/update cluster outbound|8000||zoonavigator.search-solr.svc.cluster.local during init\r\n[2018-08-10 19:45:50.274][23][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:388] add/update cluster outbound|80||app.carzone-network-report.svc.cluster.local during init\r\n[2018-08-10 19:45:50.274][23][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:388] add/update cluster outbound|80||tracing.istio-system.svc.cluster.local during init\r\n[2018-08-10 19:45:50.275][23][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:388] add/update cluster outbound|9411||zipkin.istio-system.svc.cluster.local during init\r\n[2018-08-10 19:45:50.290][23][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:388] add/update cluster outbound|80||app.gradle-cache.svc.cluster.local during init\r\n[2018-08-10 19:45:50.291][23][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:388] add/update cluster outbound|9091||istio-telemetry.istio-system.svc.cluster.local during init\r\n[2018-08-10 19:45:50.292][23][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:388] add/update cluster outbound|15004||istio-telemetry.istio-system.svc.cluster.local during init\r\n[2018-08-10 19:45:50.292][23][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:388] add/update cluster outbound|9093||istio-telemetry.istio-system.svc.cluster.local during init\r\n[2018-08-10 19:45:50.292][23][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:388] add/update cluster outbound|42422||istio-telemetry.istio-system.svc.cluster.local during init\r\n[2018-08-10 19:45:50.293][23][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:388] add/update cluster outbound|9600||logstash.core-system.svc.cluster.local during init\r\n[2018-08-10 19:45:50.295][23][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:388] add/update cluster outbound|4206||logstash.core-system.svc.cluster.local during init\r\n[2018-08-10 19:45:50.296][23][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:388] add/update cluster outbound|4207||logstash.core-system.svc.cluster.local during init\r\n[2018-08-10 19:45:50.298][23][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:388] add/update cluster outbound|4210||logstash.core-system.svc.cluster.local during init\r\n[2018-08-10 19:45:50.301][23][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:388] add/update cluster outbound|80||app.metrics-triage.svc.cluster.local during init\r\n[2018-08-10 19:45:50.302][23][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:388] add/update cluster outbound|9080||app.metrics-triage.svc.cluster.local during init\r\n[2018-08-10 19:45:50.303][23][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:388] add/update cluster outbound|80||app.platform-redirector.svc.cluster.local during init\r\n[2018-08-10 19:45:50.305][23][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:388] add/update cluster outbound|9000||zoonavigator-api.search-solr.svc.cluster.local during init\r\n[2018-08-10 19:45:50.306][23][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:388] add/update cluster outbound|80||app.brand-performance.svc.cluster.local during init\r\n[2018-08-10 19:45:50.307][23][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:388] add/update cluster outbound|5473||calico-typha.kube-system.svc.cluster.local during init\r\n[2018-08-10 19:45:50.309][23][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:388] add/update cluster outbound|80||app.vehicle-valuations-service.svc.cluster.local during init\r\n[2018-08-10 19:45:50.310][23][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:388] add/update cluster outbound|80||app.goal-collector-tool.svc.cluster.local during init\r\n[2018-08-10 19:45:50.310][23][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:388] add/update cluster outbound|443||kubernetes.default.svc.cluster.local during init\r\n[2018-08-10 19:45:50.311][23][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:388] add/update cluster outbound|80||app.vds-dvla-importer.svc.cluster.local during init\r\n[2018-08-10 19:45:50.313][23][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:388] add/update cluster outbound|8080||kube-state-metrics.istio-system.svc.cluster.local during init\r\n[2018-08-10 19:45:50.314][23][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:388] add/update cluster outbound|9094||prometheus-alertmanager.istio-system.svc.cluster.local during init\r\n[2018-08-10 19:45:50.314][23][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:388] add/update cluster outbound|80||www.google-analytics.com during init\r\n[2018-08-10 19:45:50.314][23][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:388] add/update cluster outbound|443||www.google-analytics.com during init\r\n[2018-08-10 19:45:50.315][23][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:388] add/update cluster outbound|80||www.google.com during init\r\n[2018-08-10 19:45:50.315][23][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:388] add/update cluster outbound|443||www.google.com during init\r\n[2018-08-10 19:45:50.315][23][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:388] add/update cluster outbound|80||marketingplatform.google.com during init\r\n[2018-08-10 19:45:50.315][23][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:388] add/update cluster outbound|443||marketingplatform.google.com during init\r\n[2018-08-10 19:45:50.316][23][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:388] add/update cluster outbound|80||www.facebook.com during init\r\n[2018-08-10 19:45:50.316][23][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:388] add/update cluster outbound|443||www.facebook.com during init\r\n[2018-08-10 19:45:50.316][23][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:388] add/update cluster outbound|80||graph.facebook.com during init\r\n[2018-08-10 19:45:50.316][23][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:388] add/update cluster outbound|443||graph.facebook.com during init\r\n[2018-08-10 19:45:50.316][23][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:388] add/update cluster outbound|80||api-read.facebook.com during init\r\n[2018-08-10 19:45:50.317][23][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:388] add/update cluster outbound|443||api-read.facebook.com during init\r\n[2018-08-10 19:45:50.317][23][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:388] add/update cluster outbound|80||graph-video.facebook.com during init\r\n[2018-08-10 19:45:50.317][23][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:388] add/update cluster outbound|443||graph-video.facebook.com during init\r\n[2018-08-10 19:45:50.317][23][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:388] add/update cluster outbound|80||autotrader.avature.net during init\r\n[2018-08-10 19:45:50.317][23][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:388] add/update cluster outbound|443||autotrader.avature.net during init\r\n[2018-08-10 19:45:50.318][23][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:388] add/update cluster outbound|80||automotivemxin.com during init\r\n[2018-08-10 19:45:50.318][23][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:388] add/update cluster outbound|443||automotivemxin.com during init\r\n[2018-08-10 19:45:50.318][23][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:388] add/update cluster outbound|80||app.joindrover.com during init\r\n[2018-08-10 19:45:50.318][23][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:388] add/update cluster outbound|443||app.joindrover.com during init\r\n[2018-08-10 19:45:50.319][23][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:388] add/update cluster outbound|80||www.joindrover.com during init\r\n[2018-08-10 19:45:50.319][23][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:388] add/update cluster outbound|443||www.joindrover.com during init\r\n[2018-08-10 19:45:50.319][23][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:388] add/update cluster outbound|80||graph.microsoft.com during init\r\n[2018-08-10 19:45:50.320][23][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:388] add/update cluster outbound|443||graph.microsoft.com during init\r\n[2018-08-10 19:45:50.320][23][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:388] add/update cluster outbound|80||login.microsoftonline.com during init\r\n[2018-08-10 19:45:50.320][23][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:388] add/update cluster outbound|443||login.microsoftonline.com during init\r\n[2018-08-10 19:45:50.320][23][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:388] add/update cluster outbound|25||smtp.dummy.hosts during init\r\n[2018-08-10 19:45:50.320][23][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:388] add/update cluster outbound|465||smtp.dummy.hosts during init\r\n[2018-08-10 19:45:50.320][23][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:388] add/update cluster outbound|587||smtp.dummy.hosts during init\r\n[2018-08-10 19:45:50.320][23][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:388] add/update cluster outbound|80||api.twitter.com during init\r\n[2018-08-10 19:45:50.321][23][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:388] add/update cluster outbound|443||api.twitter.com during init\r\n[2018-08-10 19:45:50.321][23][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:388] add/update cluster outbound|80||stream.twitter.com during init\r\n[2018-08-10 19:45:50.321][23][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:388] add/update cluster outbound|443||stream.twitter.com during init\r\n[2018-08-10 19:45:50.321][23][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:388] add/update cluster outbound|80||userstream.twitter.com during init\r\n[2018-08-10 19:45:50.321][23][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:388] add/update cluster outbound|443||userstream.twitter.com during init\r\n[2018-08-10 19:45:50.322][23][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:388] add/update cluster outbound|80||sitestream.twitter.com during init\r\n[2018-08-10 19:45:50.322][23][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:388] add/update cluster outbound|443||sitestream.twitter.com during init\r\n[2018-08-10 19:45:50.322][23][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:388] add/update cluster outbound|80||upload.twitter.com during init\r\n[2018-08-10 19:45:50.322][23][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:388] add/update cluster outbound|443||upload.twitter.com during init\r\n[2018-08-10 19:45:50.322][23][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:388] add/update cluster inbound|80||oauth2-proxy.ingress-nginx.svc.cluster.local during init\r\n[2018-08-10 19:45:50.322][23][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:388] add/update cluster BlackHoleCluster during init\r\n[2018-08-10 19:45:50.330][23][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:110] cm init: initializing secondary clusters\r\n[2018-08-10 19:45:50.609][23][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:133] cm init: all clusters initialized\r\n[2018-08-10 19:45:50.609][23][info][main] external/envoy/source/server/server.cc:390] all clusters initialized. initializing init manager\r\n[2018-08-10 19:45:50.710][23][warning][upstream] external/envoy/source/common/config/grpc_mux_impl.cc:65] Too many sendDiscoveryRequest calls for type.googleapis.com/envoy.api.v2.ClusterLoadAssignment\r\n[2018-08-10 19:45:58.176][23][info][upstream] external/envoy/source/server/lds_api.cc:73] lds: add/update listener '10.194.2.82_4180'\r\n[2018-08-10 19:45:58.230][23][info][upstream] external/envoy/source/server/lds_api.cc:73] lds: add/update listener '10.192.11.73_443'\r\n[2018-08-10 19:45:58.284][23][info][upstream] external/envoy/source/server/lds_api.cc:73] lds: add/update listener '10.192.7.6_80'\r\n[2018-08-10 19:45:58.337][23][info][upstream] external/envoy/source/server/lds_api.cc:73] lds: add/update listener '10.192.5.191_443'\r\n[2018-08-10 19:45:58.390][23][info][upstream] external/envoy/source/server/lds_api.cc:73] lds: add/update listener '10.192.12.64_9102'\r\n[2018-08-10 19:45:58.443][23][info][upstream] external/envoy/source/server/lds_api.cc:73] lds: add/update listener '10.192.11.245_80'\r\n[2018-08-10 19:45:58.499][23][info][upstream] external/envoy/source/server/lds_api.cc:73] lds: add/update listener '0.0.0.0_587'\r\n[2018-08-10 19:45:58.552][23][info][upstream] external/envoy/source/server/lds_api.cc:73] lds: add/update listener '0.0.0.0_3888'\r\n[2018-08-10 19:45:58.606][23][info][upstream] external/envoy/source/server/lds_api.cc:73] lds: add/update listener '0.0.0.0_25'\r\n[2018-08-10 19:45:58.663][23][warning][config] external/envoy/source/server/listener_manager_impl.cc:257] adding listener '0.0.0.0:443': filter chain match rules require TLS Inspector listener filter, but it isn't configured, trying to inject it (this might fail if Envoy is compiled without it)\r\n[2018-08-10 19:45:58.663][23][info][upstream] external/envoy/source/server/lds_api.cc:73] lds: add/update listener '0.0.0.0_443'\r\n[2018-08-10 19:45:58.718][23][info][upstream] external/envoy/source/server/lds_api.cc:73] lds: add/update listener '10.192.0.26_80'\r\n[2018-08-10 19:45:58.771][23][info][upstream] external/envoy/source/server/lds_api.cc:73] lds: add/update listener '10.192.5.111_80'\r\n[2018-08-10 19:45:58.827][23][info][upstream] external/envoy/source/server/lds_api.cc:73] lds: add/update listener '10.192.6.10_8080'\r\n[2018-08-10 19:45:58.880][23][info][upstream] external/envoy/source/server/lds_api.cc:73] lds: add/update listener '10.192.8.232_443'\r\n[2018-08-10 19:45:58.935][23][info][upstream] external/envoy/source/server/lds_api.cc:73] lds: add/update listener '10.192.10.21_15011'\r\n[2018-08-10 19:45:58.988][23][info][upstream] external/envoy/source/server/lds_api.cc:73] lds: add/update listener '10.192.3.11_9600'\r\n[2018-08-10 19:45:59.039][23][info][upstream] external/envoy/source/server/lds_api.cc:73] lds: add/update listener '10.192.9.73_44134'\r\n[2018-08-10 19:45:59.092][23][info][upstream] external/envoy/source/server/lds_api.cc:73] lds: add/update listener '10.192.3.11_4210'\r\n[2018-08-10 19:45:59.144][23][info][upstream] external/envoy/source/server/lds_api.cc:73] lds: add/update listener '0.0.0.0_2888'\r\n[2018-08-10 19:45:59.196][23][info][upstream] external/envoy/source/server/lds_api.cc:73] lds: add/update listener '10.192.0.1_443'\r\n[2018-08-10 19:45:59.249][23][info][upstream] external/envoy/source/server/lds_api.cc:73] lds: add/update listener '10.192.8.41_5473'\r\n[2018-08-10 19:45:59.304][23][info][upstream] external/envoy/source/server/lds_api.cc:73] lds: add/update listener '10.192.3.129_443'\r\n[2018-08-10 19:45:59.356][23][info][upstream] external/envoy/source/server/lds_api.cc:73] lds: add/update listener '0.0.0.0_465'\r\n[2018-08-10 19:45:59.411][23][info][upstream] external/envoy/source/server/lds_api.cc:73] lds: add/update listener '10.192.0.10_53'\r\n[2018-08-10 19:45:59.463][23][info][upstream] external/envoy/source/server/lds_api.cc:73] lds: add/update listener '10.192.8.89_42422'\r\n[2018-08-10 19:45:59.517][23][info][upstream] external/envoy/source/server/lds_api.cc:73] lds: add/update listener '10.192.8.230_6379'\r\n[2018-08-10 19:45:59.571][23][info][upstream] external/envoy/source/server/lds_api.cc:73] lds: add/update listener '0.0.0.0_8060'\r\n[2018-08-10 19:45:59.627][23][info][upstream] external/envoy/source/server/lds_api.cc:73] lds: add/update listener '0.0.0.0_2181'\r\n[2018-08-10 19:45:59.681][23][info][upstream] external/envoy/source/server/lds_api.cc:73] lds: add/update listener '0.0.0.0_9093'\r\n[2018-08-10 19:45:59.737][23][info][upstream] external/envoy/source/server/lds_api.cc:73] lds: add/update listener '0.0.0.0_9091'\r\n[2018-08-10 19:45:59.790][23][info][upstream] external/envoy/source/server/lds_api.cc:73] lds: add/update listener '0.0.0.0_15004'\r\n[2018-08-10 19:45:59.844][23][info][upstream] external/envoy/source/server/lds_api.cc:73] lds: add/update listener '0.0.0.0_8000'\r\n[2018-08-10 19:45:59.897][23][info][upstream] external/envoy/source/server/lds_api.cc:73] lds: add/update listener '0.0.0.0_9411'\r\n[2018-08-10 19:45:59.950][23][info][upstream] external/envoy/source/server/lds_api.cc:73] lds: add/update listener '0.0.0.0_9094'\r\n[2018-08-10 19:46:00.003][23][info][upstream] external/envoy/source/server/lds_api.cc:73] lds: add/update listener '0.0.0.0_8080'\r\n[2018-08-10 19:46:00.058][23][info][upstream] external/envoy/source/server/lds_api.cc:73] lds: add/update listener '0.0.0.0_3000'\r\n[2018-08-10 19:46:00.110][23][info][upstream] external/envoy/source/server/lds_api.cc:73] lds: add/update listener '0.0.0.0_9090'\r\n[2018-08-10 19:46:00.163][23][info][upstream] external/envoy/source/server/lds_api.cc:73] lds: add/update listener '0.0.0.0_15010'\r\n[2018-08-10 19:46:00.216][23][info][upstream] external/envoy/source/server/lds_api.cc:73] lds: add/update listener '0.0.0.0_9000'\r\n[2018-08-10 19:46:00.269][23][info][upstream] external/envoy/source/server/lds_api.cc:73] lds: add/update listener '0.0.0.0_80'\r\n[2018-08-10 19:46:00.324][23][info][upstream] external/envoy/source/server/lds_api.cc:73] lds: add/update listener '0.0.0.0_9080'\r\n[2018-08-10 19:46:00.376][23][info][upstream] external/envoy/source/server/lds_api.cc:73] lds: add/update listener 'virtual'\r\n[2018-08-10 19:46:46.822][23][info][config] external/envoy/source/server/listener_manager_impl.cc:903] all dependencies initialized. starting workers\r\n[2018-08-10 19:47:33.014][23][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:394] add/update cluster outbound|80||echo-server-service.core-system.svc.cluster.local starting warming\r\n[2018-08-10 19:47:33.016][23][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:394] add/update cluster outbound|80||rspec-cert-testing.default.svc.cluster.local starting warming\r\n[2018-08-10 19:47:46.810][23][info][main] external/envoy/source/server/drain_manager_impl.cc:63] shutting down parent after drain\r\n[2018-08-10 19:48:15.611][23][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:401] warming cluster outbound|80||rspec-cert-testing.default.svc.cluster.local complete\r\n[2018-08-10 19:48:15.612][23][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:401] warming cluster outbound|80||echo-server-service.core-system.svc.cluster.local complete\r\n[2018-08-10 19:48:21.716][23][warning][config] external/envoy/source/server/listener_manager_impl.cc:257] adding listener '0.0.0.0:443': filter chain match rules require TLS Inspector listener filter, but it isn't configured, trying to inject it (this might fail if Envoy is compiled without it)\r\n[2018-08-10 19:48:21.717][23][info][upstream] external/envoy/source/server/lds_api.cc:73] lds: add/update listener '0.0.0.0_443'\r\n[2018-08-10 19:48:45.718][23][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:394] add/update cluster outbound|80||mtls-1.default.svc.cluster.local starting warming\r\n[2018-08-10 19:49:31.101][23][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:401] warming cluster outbound|80||mtls-1.default.svc.cluster.local complete\r\n[2018-08-10 19:49:39.288][23][warning][config] external/envoy/source/server/listener_manager_impl.cc:257] adding listener '0.0.0.0:443': filter chain match rules require TLS Inspector listener filter, but it isn't configured, trying to inject it (this might fail if Envoy is compiled without it)\r\n[2018-08-10 19:49:39.289][23][info][upstream] external/envoy/source/server/lds_api.cc:73] lds: add/update listener '0.0.0.0_443'\r\n[2018-08-10 19:52:19.005][23][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:446] removing cluster outbound|80||mtls-1.default.svc.cluster.local\r\n[2018-08-10 19:52:19.006][23][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:446] removing cluster outbound|80||rspec-cert-testing.default.svc.cluster.local\r\n[2018-08-10 19:52:19.006][23][info][upstream] external/envoy/source/common/upstream/cluster_manager_impl.cc:446] removing cluster outbound|80||echo-server-service.core-system.svc.cluster.local\r\n[2018-08-10 19:53:18.487][23][warning][config] external/envoy/source/server/listener_manager_impl.cc:257] adding listener '0.0.0.0:443': filter chain match rules require TLS Inspector listener filter, but it isn't configured, trying to inject it (this might fail if Envoy is compiled without it)\r\n[2018-08-10 19:53:18.489][23][info][upstream] external/envoy/source/server/lds_api.cc:73] lds: add/update listener '0.0.0.0_443'\r\n[libprotobuf INFO src/istio/mixerclient/check_cache.cc:160] Add a new Referenced for check cache: Absence-keys: source.ip, Exact-keys: context.protocol, context.reporter.kind, destination.namespace, destination.port, destination.service, destination.uid, source.uid,\r\n[2018-08-10 19:58:33.600][23][warning][config] external/envoy/source/server/listener_manager_impl.cc:257] adding listener '0.0.0.0:443': filter chain match rules require TLS Inspector listener filter, but it isn't configured, trying to inject it (this might fail if Envoy is compiled without it)\r\n[2018-08-10 19:58:33.601][23][info][upstream] external/envoy/source/server/lds_api.cc:73] lds: add/update listener '0.0.0.0_443'\r\n[2018-08-10 19:59:36.980][23][warning][config] external/envoy/source/server/listener_manager_impl.cc:257] adding listener '0.0.0.0:443': filter chain match rules require TLS Inspector listener filter, but it isn't configured, trying to inject it (this might fail if Envoy is compiled without it)\r\n[2018-08-10 19:59:36.980][23][info][upstream] external/envoy/source/server/lds_api.cc:73] lds: add/update listener '0.0.0.0_443'\r\n[libprotobuf INFO src/istio/mixerclient/check_cache.cc:160] Add a new Referenced for check cache: Absence-keys: source.ip, Exact-keys: context.protocol, context.reporter.kind, destination.namespace, destination.port, destination.service, destination.uid, source.uid,\r\n[2018-08-10 20:01:08.598][23][warning][config] external/envoy/source/server/listener_manager_impl.cc:257] adding listener '0.0.0.0:443': filter chain match rules require TLS Inspector listener filter, but it isn't configured, trying to inject it (this might fail if Envoy is compiled without it)\r\n[2018-08-10 20:01:08.598][23][info][upstream] external/envoy/source/server/lds_api.cc:73] lds: add/update listener '0.0.0.0_443'\r\n[2018-08-10 20:02:28.878][23][warning][config] external/envoy/source/server/listener_manager_impl.cc:257] adding listener '0.0.0.0:443': filter chain match rules require TLS Inspector listener filter, but it isn't configured, trying to inject it (this might fail if Envoy is compiled without it)\r\n[2018-08-10 20:02:28.878][23][info][upstream] external/envoy/source/server/lds_api.cc:73] lds: add/update listener '0.0.0.0_443'\r\n[2018-08-10 20:03:41.899][23][warning][config] external/envoy/source/server/listener_manager_impl.cc:257] adding listener '0.0.0.0:443': filter chain match rules require TLS Inspector listener filter, but it isn't configured, trying to inject it (this might fail if Envoy is compiled without it)\r\n[2018-08-10 20:03:41.900][23][info][upstream] external/envoy/source/server/lds_api.cc:73] lds: add/update listener '0.0.0.0_443'\r\n```\r\n\r\n`[2018-08-10 19:45:50.710][23][warning][upstream] external/envoy/source/common/config/grpc_mux_impl.cc:65] Too many sendDiscoveryRequest calls for type.googleapis.com/envoy.api.v2.ClusterLoadAssignment` - sticks out to me?\r\n\r\nHere are the cpu and RAM graphs for this pod + proxy:\r\n<img width=\"1544\" alt=\"screen shot 2018-08-10 at 21 11 48\" src=\"https://user-images.githubusercontent.com/1486729/43979262-0ef885fa-9ce2-11e8-8b98-1df800154ad6.png\">\r\n\r\nHere is a cpu and RAM graph from one of the pilot instances:\r\n<img width=\"1541\" alt=\"screen shot 2018-08-10 at 21 13 17\" src=\"https://user-images.githubusercontent.com/1486729/43979321-40474f10-9ce2-11e8-9149-ecffb35c9e0e.png\">\r\n\r\nThis happens consistently across the cluster.  I've noticed pretty consistently around 45 seconds between these lines:\r\n\r\n```\r\n[2018-08-10 19:46:00.376][23][info][upstream] external/envoy/source/server/lds_api.cc:73] lds: add/update listener 'virtual'\r\n[2018-08-10 19:46:46.822][23][info][config] external/envoy/source/server/listener_manager_impl.cc:903] all dependencies initialized. starting workers\r\n```\r\n\r\n**Cluster Info**\r\nistio 1.0 on gke 1.10.5\r\n\r\n - Services: 60\r\n - VirtualServices: 30\r\n - DestinationRules: 38\r\n - Pods: 204\r\n\r\nWe have 4x pilots running 2500m cores, 1 gig ram"},"comment":{"url":"https://api.github.com/repos/istio/istio/issues/comments/412337486","html_url":"https://github.com/istio/istio/issues/7817#issuecomment-412337486","issue_url":"https://api.github.com/repos/istio/istio/issues/7817","id":412337486,"node_id":"MDEyOklzc3VlQ29tbWVudDQxMjMzNzQ4Ng==","user":{"login":"Stono","id":1486729,"node_id":"MDQ6VXNlcjE0ODY3Mjk=","avatar_url":"https://avatars2.githubusercontent.com/u/1486729?v=4","gravatar_id":"","url":"https://api.github.com/users/Stono","html_url":"https://github.com/Stono","followers_url":"https://api.github.com/users/Stono/followers","following_url":"https://api.github.com/users/Stono/following{/other_user}","gists_url":"https://api.github.com/users/Stono/gists{/gist_id}","starred_url":"https://api.github.com/users/Stono/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Stono/subscriptions","organizations_url":"https://api.github.com/users/Stono/orgs","repos_url":"https://api.github.com/users/Stono/repos","events_url":"https://api.github.com/users/Stono/events{/privacy}","received_events_url":"https://api.github.com/users/Stono/received_events","type":"User","site_admin":false},"created_at":"2018-08-12T11:49:59Z","updated_at":"2018-08-12T11:49:59Z","author_association":"CONTRIBUTOR","body":"@costinm I was able to get this down to sub 15s by increasing the CPU limits on `istio-pilot` from 2500m to 4000m so there seems to be some sort of correlation between istio-pilot being cpu strained and the `istio-proxy` containers boot time.  Maybe some of the CPU limit work you're doing around https://github.com/istio/istio/issues/6962 will help this issue too?"}}
{"actor":{"display_login":"Stono"}}