{"action":"created","issue":{"url":"https://api.github.com/repos/istio/istio/issues/6443","repository_url":"https://api.github.com/repos/istio/istio","labels_url":"https://api.github.com/repos/istio/istio/issues/6443/labels{/name}","comments_url":"https://api.github.com/repos/istio/istio/issues/6443/comments","events_url":"https://api.github.com/repos/istio/istio/issues/6443/events","html_url":"https://github.com/istio/istio/issues/6443","id":334241990,"node_id":"MDU6SXNzdWUzMzQyNDE5OTA=","number":6443,"title":"Pilot segfaults when given invalid remote cluster credentials","user":{"login":"a-robinson","id":7085343,"node_id":"MDQ6VXNlcjcwODUzNDM=","avatar_url":"https://avatars3.githubusercontent.com/u/7085343?v=4","gravatar_id":"","url":"https://api.github.com/users/a-robinson","html_url":"https://github.com/a-robinson","followers_url":"https://api.github.com/users/a-robinson/followers","following_url":"https://api.github.com/users/a-robinson/following{/other_user}","gists_url":"https://api.github.com/users/a-robinson/gists{/gist_id}","starred_url":"https://api.github.com/users/a-robinson/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/a-robinson/subscriptions","organizations_url":"https://api.github.com/users/a-robinson/orgs","repos_url":"https://api.github.com/users/a-robinson/repos","events_url":"https://api.github.com/users/a-robinson/events{/privacy}","received_events_url":"https://api.github.com/users/a-robinson/received_events","type":"User","site_admin":false},"labels":[{"id":670442116,"node_id":"MDU6TGFiZWw2NzA0NDIxMTY=","url":"https://api.github.com/repos/istio/istio/labels/area/environments","name":"area/environments","color":"0e8a16","default":false}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":5,"created_at":"2018-06-20T20:42:47Z","updated_at":"2018-06-21T01:42:06Z","closed_at":null,"author_association":"NONE","body":"**Describe the bug**\r\nAfter installing Istio 0.8 on a fresh GKE version 1.9.7-gke.3 cluster, the `istio-pilot` pod's `discovery` container is repeatedly crashing:\r\n\r\n```\r\n$ kubectl logs -n istio-system istio-pilot-6c5c6b586c-6k2tq discovery\r\n2018-06-20T20:36:58.338514Z\twarn\tNeither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.\r\n2018-06-20T20:36:58.340485Z\tinfo\tclusters configuration (*clusterregistry.ClusterStore)(0xc420275e80)({\r\n rc: (map[clusterregistry.Metadata]*clusterregistry.RemoteCluster) {\r\n },\r\n storeLock: (sync.RWMutex) {\r\n  w: (sync.Mutex) {\r\n   state: (int32) 0,\r\n   sema: (uint32) 0\r\n  },\r\n  writerSem: (uint32) 0,\r\n  readerSem: (uint32) 0,\r\n  readerCount: (int32) 0,\r\n  readerWait: (int32) 0\r\n }\r\n})\r\n\r\n2018-06-20T20:36:58.342244Z\tinfo\tmesh configuration (*v1alpha1.MeshConfig)(0xc4204d1740)(mixer_check_server:\"istio-policy.istio-system.svc.cluster.local:15004\" mixer_report_server:\"istio-telemetry.istio-system.svc.cluster.local:15004\" proxy_listen_port:15001 connect_timeout:<seconds:1 > ingress_class:\"istio\" ingress_service:\"istio-ingress\" ingress_controller_mode:STRICT rds_refresh_delay:<seconds:10 > enable_tracing:true access_log_file:\"/dev/stdout\" default_config:<config_path:\"/etc/istio/proxy\" binary_path:\"/usr/local/bin/envoy\" service_cluster:\"istio-proxy\" drain_duration:<seconds:45 > parent_shutdown_duration:<seconds:60 > discovery_address:\"istio-pilot.istio-system:15007\" discovery_refresh_delay:<seconds:10 > zipkin_address:\"zipkin.istio-system:9411\" connect_timeout:<seconds:10 > statsd_udp_address:\"istio-statsd-prom-bridge.istio-system:9125\" proxy_admin_port:15000 > )\r\n\r\n2018-06-20T20:36:58.342454Z\tinfo\tversion root@48d5ddfd72da-docker.io/istio-0.8.0-6f9f420f0c7119ff4fa6a1966a6f6d89b1b4db84-Clean\r\n2018-06-20T20:36:58.342678Z\tinfo\tflags (*bootstrap.PilotArgs)(0xc420202180)({\r\n DiscoveryOptions: (v1.DiscoveryServiceOptions) {\r\n  Port: (int) 8080,\r\n  GrpcAddr: (string) (len=6) \":15010\",\r\n  SecureGrpcAddr: (string) (len=6) \":15012\",\r\n  MonitoringPort: (int) 9093,\r\n  EnableProfiling: (bool) true,\r\n  EnableCaching: (bool) true,\r\n  WebhookEndpoint: (string) \"\"\r\n },\r\n Namespace: (string) (len=12) \"istio-system\",\r\n Mesh: (bootstrap.MeshArgs) {\r\n  ConfigFile: (string) (len=22) \"/etc/istio/config/mesh\",\r\n  MixerAddress: (string) \"\",\r\n  RdsRefreshDelay: (*duration.Duration)(<nil>)\r\n },\r\n Config: (bootstrap.ConfigArgs) {\r\n  ClusterRegistriesConfigmap: (string) \"\",\r\n  ClusterRegistriesNamespace: (string) (len=12) \"istio-system\",\r\n  KubeConfig: (string) \"\",\r\n  CFConfig: (string) \"\",\r\n  ControllerOptions: (kube.ControllerOptions) {\r\n   WatchedNamespace: (string) \"\",\r\n   ResyncPeriod: (time.Duration) 1m0s,\r\n   DomainSuffix: (string) (len=13) \"cluster.local\"\r\n  },\r\n  FileDir: (string) \"\"\r\n },\r\n Service: (bootstrap.ServiceArgs) {\r\n  Registries: ([]string) (len=1 cap=1) {\r\n   (string) (len=10) \"Kubernetes\"\r\n  },\r\n  Consul: (bootstrap.ConsulArgs) {\r\n   Config: (string) \"\",\r\n   ServerURL: (string) \"\",\r\n   Interval: (time.Duration) 2s\r\n  },\r\n  Eureka: (bootstrap.EurekaArgs) {\r\n   ServerURL: (string) \"\",\r\n   Interval: (time.Duration) 2s\r\n  }\r\n },\r\n RDSv2: (bool) false,\r\n MeshConfig: (*v1alpha1.MeshConfig)(<nil>),\r\n CtrlZOptions: (*ctrlz.Options)(0xc42043d560)({\r\n  Port: (uint16) 9876,\r\n  Address: (string) (len=9) \"127.0.0.1\"\r\n })\r\n})\r\n\r\n2018-06-20T20:36:58.343067Z\twarn\tNeither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.\r\n2018-06-20T20:36:58.343610Z\twarn\tNeither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.\r\n2018-06-20T20:36:58.344145Z\twarn\tNeither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.\r\n2018-06-20T20:36:58.347284Z\tinfo\tregistering for apiVersion config.istio.io/v1alpha2\r\n2018-06-20T20:36:58.394859Z\tinfo\tregistering for apiVersion networking.istio.io/v1alpha3\r\n2018-06-20T20:36:58.405580Z\tinfo\tregistering for apiVersion authentication.istio.io/v1alpha1\r\n2018-06-20T20:36:58.407581Z\tinfo\tCRD controller watching namespaces \"\"\r\n2018-06-20T20:36:58.407915Z\tinfo\tAdding Kubernetes registry adapter\r\n2018-06-20T20:36:58.407926Z\tinfo\tPrimary Cluster name: Kubernetes\r\n2018-06-20T20:36:58.407947Z\tinfo\tService controller watching namespace \"\" for service, endpoint, nodes and pods, refresh 60000000000\r\n2018-06-20T20:36:58.407986Z\tinfo\tIngress controller watching namespaces \"\"\r\n2018-06-20T20:36:58.408127Z\tinfo\tingress status syncer publishService istio-system/istio-ingress\r\n2018-06-20T20:36:58.425783Z\tinfo\tSetting up event handlers\r\n2018-06-20T20:36:58.426071Z\tinfo\tDiscovery service started at http=[::]:8080 grpc=[::]:15010\r\n2018-06-20T20:36:58.426372Z\tinfo\tStarting Secrets controller\r\n2018-06-20T20:36:58.426450Z\tinfo\tWaiting for informer caches to sync\r\n2018-06-20T20:36:58.431050Z\tinfo\tControlZ available at 10.40.1.12:9876\r\n2018-06-20T20:36:58.434608Z\tinfo\tStarting GRPC secure on [::]:15012 with certs in /etc/certs/\r\n2018-06-20T20:36:58.442043Z\tinfo\tattempting to acquire leader lease...\r\n2018-06-20T20:36:58.452725Z\tinfo\tProcessing add: istio-system/istio-b\r\n2018-06-20T20:36:58.453100Z\tinfo\tWork item failed (waiting till full synchronization), repeating after delay 1s\r\n2018-06-20T20:36:58.503712Z\tinfo\tCleared discovery service cache\r\n2018-06-20T20:36:58.504056Z\tinfo\tads\tXDS: Registry event, pushing. Services: 19, VirtualServices: 0, ConnectedEndpoints: 0\r\n2018-06-20T20:36:58.504864Z\tinfo\tsuccessfully acquired lease istio-system/istio-ingress-controller-leader-istio\r\n2018-06-20T20:36:58.526605Z\tinfo\tAdding new cluster member: istio-b\r\n2018-06-20T20:36:58.528253Z\tinfo\tService controller watching namespace \"\" for service, endpoint, nodes and pods, refresh 60000000000\r\n2018-06-20T20:36:58.528561Z\tinfo\tNumber of clusters in the cluster store: 1\r\n2018-06-20T20:36:58.529070Z\terror\tObserved a panic: \"invalid memory address or nil pointer dereference\" (runtime error: invalid memory address or nil pointer dereference)\r\n/workspace/go/src/istio.io/istio/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:72\r\n/workspace/go/src/istio.io/istio/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:65\r\n/workspace/go/src/istio.io/istio/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:51\r\n/usr/local/go/src/runtime/asm_amd64.s:573\r\n/usr/local/go/src/runtime/panic.go:502\r\n/usr/local/go/src/runtime/panic.go:63\r\n/usr/local/go/src/runtime/signal_unix.go:388\r\n/workspace/go/src/istio.io/istio/pilot/pkg/serviceregistry/kube/controller.go:131\r\n/workspace/go/src/istio.io/istio/vendor/k8s.io/client-go/tools/pager/pager.go:39\r\n/workspace/go/src/istio.io/istio/vendor/k8s.io/client-go/tools/pager/pager.go:77\r\n/workspace/go/src/istio.io/istio/vendor/k8s.io/client-go/tools/cache/listwatch.go:97\r\n/workspace/go/src/istio.io/istio/vendor/k8s.io/client-go/tools/cache/reflector.go:249\r\n/workspace/go/src/istio.io/istio/vendor/k8s.io/client-go/tools/cache/reflector.go:204\r\n/workspace/go/src/istio.io/istio/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133\r\n/workspace/go/src/istio.io/istio/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134\r\n/workspace/go/src/istio.io/istio/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88\r\n/workspace/go/src/istio.io/istio/vendor/k8s.io/client-go/tools/cache/reflector.go:203\r\n/workspace/go/src/istio.io/istio/vendor/k8s.io/client-go/tools/cache/controller.go:122\r\n/workspace/go/src/istio.io/istio/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:54\r\n/workspace/go/src/istio.io/istio/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:71\r\n/usr/local/go/src/runtime/asm_amd64.s:2361\r\n\r\npanic: runtime error: invalid memory address or nil pointer dereference [recovered]\r\n\tpanic: runtime error: invalid memory address or nil pointer dereference\r\n[signal SIGSEGV: segmentation violation code=0x1 addr=0xd0 pc=0x112aba7]\r\n\r\ngoroutine 366 [running]:\r\nistio.io/istio/vendor/k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0)\r\n\t/workspace/go/src/istio.io/istio/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:58 +0x107\r\npanic(0x15de700, 0x237c0b0)\r\n\t/usr/local/go/src/runtime/panic.go:502 +0x229\r\nistio.io/istio/pilot/pkg/serviceregistry/kube.NewController.func5(0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x17e9327, ...)\r\n\t/workspace/go/src/istio.io/istio/pilot/pkg/serviceregistry/kube/controller.go:131 +0x37\r\nistio.io/istio/vendor/k8s.io/client-go/tools/pager.SimplePageFunc.func1(0x1963f80, 0xc420044050, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)\r\n\t/workspace/go/src/istio.io/istio/vendor/k8s.io/client-go/tools/pager/pager.go:39 +0x64\r\nistio.io/istio/vendor/k8s.io/client-go/tools/pager.(*ListPager).List(0xc42067f9f8, 0x1963f80, 0xc420044050, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)\r\n\t/workspace/go/src/istio.io/istio/vendor/k8s.io/client-go/tools/pager/pager.go:77 +0x105\r\nistio.io/istio/vendor/k8s.io/client-go/tools/cache.(*ListWatch).List(0xc4204fe200, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)\r\n\t/workspace/go/src/istio.io/istio/vendor/k8s.io/client-go/tools/cache/listwatch.go:97 +0x147\r\nistio.io/istio/vendor/k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch(0xc420a0fe00, 0xc42083eea0, 0x0, 0x0)\r\n\t/workspace/go/src/istio.io/istio/vendor/k8s.io/client-go/tools/cache/reflector.go:249 +0x208\r\nistio.io/istio/vendor/k8s.io/client-go/tools/cache.(*Reflector).Run.func1()\r\n\t/workspace/go/src/istio.io/istio/vendor/k8s.io/client-go/tools/cache/reflector.go:204 +0x33\r\nistio.io/istio/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1(0xc420911718)\r\n\t/workspace/go/src/istio.io/istio/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133 +0x54\r\nistio.io/istio/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc42067ff18, 0x3b9aca00, 0x0, 0x1, 0xc42083eea0)\r\n\t/workspace/go/src/istio.io/istio/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134 +0xbd\r\nistio.io/istio/vendor/k8s.io/apimachinery/pkg/util/wait.Until(0xc420911718, 0x3b9aca00, 0xc42083eea0)\r\n\t/workspace/go/src/istio.io/istio/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88 +0x4d\r\nistio.io/istio/vendor/k8s.io/client-go/tools/cache.(*Reflector).Run(0xc420a0fe00, 0xc42083eea0)\r\n\t/workspace/go/src/istio.io/istio/vendor/k8s.io/client-go/tools/cache/reflector.go:203 +0x157\r\nistio.io/istio/vendor/k8s.io/client-go/tools/cache.(*Reflector).Run-fm(0xc42083eea0)\r\n\t/workspace/go/src/istio.io/istio/vendor/k8s.io/client-go/tools/cache/controller.go:122 +0x34\r\nistio.io/istio/vendor/k8s.io/apimachinery/pkg/util/wait.(*Group).StartWithChannel.func1()\r\n\t/workspace/go/src/istio.io/istio/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:54 +0x31\r\nistio.io/istio/vendor/k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1(0xc42050f2e0, 0xc4204fe420)\r\n\t/workspace/go/src/istio.io/istio/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:71 +0x4f\r\ncreated by istio.io/istio/vendor/k8s.io/apimachinery/pkg/util/wait.(*Group).Start\r\n\t/workspace/go/src/istio.io/istio/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:69 +0x62\r\n```\r\n\r\n\"istio-b\" is what I named the remote cluster, as instructed by https://istio.io/docs/setup/kubernetes/multicluster-install/#gather-credential-files-from-remote.\r\n\r\n**Expected behavior**\r\nI expected the pilot pod to run without crashing.\r\n\r\n**Steps to reproduce the bug**\r\nFirst, create two GKE clusters:\r\n\r\n```\r\n$ gcloud container clusters create istio --cluster-version=1.9.7-gke.3 --zone=us-central1-a\r\n$ gcloud container clusters create istio --cluster-version=1.9.7-gke.3 --zone=us-central1-b\r\n```\r\n\r\nThen, follow the steps in https://istio.io/docs/setup/kubernetes/multicluster-install/, installing istio in the cluster in us-central1-a and using the cluster in us-central1-b as the remote. I believe I've followed the instructions exactly, although for what it's worth I was getting a different segfault when I accidentally created a secret for the primary cluster (the one in us-central1-a). If you're insterested in that one, I've copied the logs to https://gist.github.com/a-robinson/eaac77f072e5e0a5ef23940e3eb09239.\r\n\r\n**Version**\r\n```\r\n$ istioctl version\r\nVersion: 0.8.0\r\nGitRevision: 6f9f420f0c7119ff4fa6a1966a6f6d89b1b4db84\r\nUser: root@48d5ddfd72da\r\nHub: docker.io/istio\r\nGolangVersion: go1.10.1\r\nBuildStatus: Clean\r\n$ kubectl version\r\nClient Version: version.Info{Major:\"1\", Minor:\"9\", GitVersion:\"v1.9.7\", GitCommit:\"dd5e1a2978fd0b97d9b78e1564398aeea7e7fe92\", GitTreeState:\"clean\", BuildDate:\"2018-04-19T00:05:56Z\", GoVersion:\"go1.9.3\", Compiler:\"gc\", Platform:\"darwin/amd64\"}\r\nServer Version: version.Info{Major:\"1\", Minor:\"9+\", GitVersion:\"v1.9.7-gke.3\", GitCommit:\"9b5b719c5f295c99de68ffb5b63101b0e0175376\", GitTreeState:\"clean\", BuildDate:\"2018-05-31T18:32:23Z\", GoVersion:\"go1.9.3b4\", Compiler:\"gc\", Platform:\"linux/amd64\"}\r\n```\r\n\r\n**Is Istio Auth enabled or not?**\r\nI don't know. I ran `kubectl apply -f install/kubernetes/istio-demo.yaml` from the 0.8 release package, as instructed by https://istio.io/docs/setup/kubernetes/quick-start/#installation-steps, which is linked from the steps in https://istio.io/docs/setup/kubernetes/multicluster-install/.\r\n\r\n**Environment**\r\nGKE version 1.9.7-gke.3 on the default OS."},"comment":{"url":"https://api.github.com/repos/istio/istio/issues/comments/398949572","html_url":"https://github.com/istio/istio/issues/6443#issuecomment-398949572","issue_url":"https://api.github.com/repos/istio/istio/issues/6443","id":398949572,"node_id":"MDEyOklzc3VlQ29tbWVudDM5ODk0OTU3Mg==","user":{"login":"tiswanso","id":3629949,"node_id":"MDQ6VXNlcjM2Mjk5NDk=","avatar_url":"https://avatars0.githubusercontent.com/u/3629949?v=4","gravatar_id":"","url":"https://api.github.com/users/tiswanso","html_url":"https://github.com/tiswanso","followers_url":"https://api.github.com/users/tiswanso/followers","following_url":"https://api.github.com/users/tiswanso/following{/other_user}","gists_url":"https://api.github.com/users/tiswanso/gists{/gist_id}","starred_url":"https://api.github.com/users/tiswanso/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tiswanso/subscriptions","organizations_url":"https://api.github.com/users/tiswanso/orgs","repos_url":"https://api.github.com/users/tiswanso/repos","events_url":"https://api.github.com/users/tiswanso/events{/privacy}","received_events_url":"https://api.github.com/users/tiswanso/received_events","type":"User","site_admin":false},"created_at":"2018-06-21T01:42:06Z","updated_at":"2018-06-21T01:42:06Z","author_association":"CONTRIBUTOR","body":"still reading through this... 99% sure this is a case where the remote-cluster's secret name is different than the filename in secret data.  There's a straight dereference of a map with the secret name in the code without validating the key exists."}}
{"actor":{"display_login":"tiswanso"}}