title: Failed to verify healthcheck kubernetes + rancher
body: <p>Does any body know about the following error? I already running <code>kube-apiserver</code> on host <code>192.168.88.245]</code></p>
  
  <pre><code>Failed to bring up Control Plane: Failed to verify healthcheck: Failed to check https://localhost:6443/healthz for service [kube-apiserver] on host [192.168.88.245]:
  </code></pre>
  
  <p>RKE up commend <code>rke up --config ./rancher-cluser.yml</code></p>
  
  <p>Details logs </p>
  
  <pre><code>INFO[0256] [certificates] Successfully started [rke-bundle-cert] container on host [192.168.88.245]
  INFO[0257] [certificates] successfully saved certificate bundle [/opt/rke/etcd-snapshots//pki.bundle.tar.gz] on host [192.168.88.245]
  INFO[0259] [etcd] Successfully started [rke-log-linker] container on host [192.168.88.245]
  INFO[0260] [remove/rke-log-linker] Successfully removed container on host [192.168.88.245]
  INFO[0260] [etcd] Successfully started etcd plane..
  INFO[0260] [controlplane] Building up Controller Plane..
  INFO[0260] [remove/service-sidekick] Successfully removed container on host [192.168.88.249]
  INFO[0260] [remove/service-sidekick] Successfully removed container on host [192.168.88.243]
  INFO[0260] [healthcheck] Start Healthcheck on service [kube-apiserver] on host [192.168.88.249]
  INFO[0260] [healthcheck] Start Healthcheck on service [kube-apiserver] on host [192.168.88.243]
  INFO[0261] [controlplane] Successfully started [kube-apiserver] container on host [192.168.88.245]
  INFO[0261] [healthcheck] Start Healthcheck on service [kube-apiserver] on host [192.168.88.245]
  FATA[0471] [controlPlane] Failed to bring up Control Plane: Failed to verify healthcheck: Failed to check https://localhost:6443/healthz for service [kube-apiserver] on host [192.168.88.245]: Get https://localhost:6443/healthz: EOF, log: I0205 07:10:08.522166       1 plugins.go:161] Loaded 6 validating admission controller(s) successfully in the following order: LimitRanger,ServiceAccount,Priority,PersistentVolumeClaimResize,ValidatingAdmissionWebhook,ResourceQuota.
  [root@localhost ~]# Failed to bring up Control Plane: Failed to verify healthcheck: Failed to check https://localhost:6443/healthz for service [kube-apiserver] on host [192.168.88.245]: Get https://localhost:6443/healthz: EOF, log: I0205 07:10:08.522166       1 plugins.go:161] Loaded 6 validating admission controller(s) successfully in the following order: LimitRanger,ServiceAccount,Priority,PersistentVolumeClaimResize,ValidatingAdmissionWebhook,ResourceQuota.
  </code></pre>
  
  <p><code>kube-apiserver</code> info, which is running as docker container on host (192.168.88.245)</p>
  
  <pre><code>[
      {
          "Id": "bb669a98fd09df705c309038c1e81748a76e334aed821a60415fcea3732aba79",
          "Created": "2019-02-05T07:06:51.657829117Z",
          "Path": "/opt/rke-tools/entrypoint.sh",
          "Args": [
              "kube-apiserver",
              "--bind-address=0.0.0.0",
              "--tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem",
              "--etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem",
              "--requestheader-extra-headers-prefix=X-Remote-Extra-",
              "--requestheader-username-headers=X-Remote-User",
              "--etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem",
              "--requestheader-allowed-names=kube-apiserver-proxy-client",
              "--service-cluster-ip-range=10.43.0.0/16",
              "--enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota",
              "--kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname",
              "--cloud-provider=",
              "--etcd-prefix=/registry",
              "--insecure-bind-address=127.0.0.1",
              "--storage-backend=etcd3",
              "--requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem",
              "--etcd-certfile=/etc/kubernetes/ssl/kube-node.pem",
              "--proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem",
              "--tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305",
              "--secure-port=6443",
              "--allow-privileged=true",
              "--tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem",
              "--kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem",
              "--service-account-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem",
              "--client-ca-file=/etc/kubernetes/ssl/kube-ca.pem",
              "--kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem",
              "--etcd-servers=https://172.17.0.1:2379,https://172.17.0.1:2379,https://172.17.0.1:2379",
              "--proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem",
              "--authorization-mode=Node,RBAC",
              "--insecure-port=0",
              "--requestheader-group-headers=X-Remote-Group",
              "--service-node-port-range=30000-32767"
          ],
          "State": {
              "Status": "running",
              "Running": true,
              "Paused": false,
              "Restarting": false,
              "OOMKilled": false,
              "Dead": false,
              "Pid": 27063,
              "ExitCode": 0,
              "Error": "",
              "StartedAt": "2019-02-05T07:31:52.985537625Z",
              "FinishedAt": "2019-02-05T07:31:52.643734399Z"
          },
          "Image": "sha256:ad8f7f0613557e8f3cffce5b69859807590aa4f47e79ab72bc1fc5029952cb81",
          "ResolvConfPath": "/var/lib/docker/containers/bb669a98fd09df705c309038c1e81748a76e334aed821a60415fcea3732aba79/resolv.conf",
          "HostnamePath": "/var/lib/docker/containers/bb669a98fd09df705c309038c1e81748a76e334aed821a60415fcea3732aba79/hostname",
          "HostsPath": "/var/lib/docker/containers/bb669a98fd09df705c309038c1e81748a76e334aed821a60415fcea3732aba79/hosts",
          "LogPath": "/var/lib/docker/containers/bb669a98fd09df705c309038c1e81748a76e334aed821a60415fcea3732aba79/bb669a98fd09df705c309038c1e81748a76e334aed821a60415fcea3732aba79-json.log",
          "Name": "/kube-apiserver",
          "RestartCount": 69,
          "Driver": "devicemapper",
          "MountLabel": "",
          "ProcessLabel": "",
          "AppArmorProfile": "",
          "ExecIDs": null,
          "HostConfig": {
              "Binds": [
                  "/etc/kubernetes:/etc/kubernetes:z"
              ],
              "ContainerIDFile": "",
              "LogConfig": {
                  "Type": "json-file",
                  "Config": {}
              },
              "NetworkMode": "host",
              "PortBindings": {},
              "RestartPolicy": {
                  "Name": "always",
                  "MaximumRetryCount": 0
              },
              "AutoRemove": false,
              "VolumeDriver": "",
              "VolumesFrom": [
                  "service-sidekick"
              ],
              "CapAdd": null,
              "CapDrop": null,
              "Dns": null,
              "DnsOptions": null,
              "DnsSearch": null,
              "ExtraHosts": null,
              "GroupAdd": null,
              "IpcMode": "",
              "Cgroup": "",
              "Links": null,
              "OomScoreAdj": 0,
              "PidMode": "",
              "Privileged": false,
              "PublishAllPorts": false,
              "ReadonlyRootfs": false,
              "SecurityOpt": null,
              "UTSMode": "",
              "UsernsMode": "",
              "ShmSize": 67108864,
              "Runtime": "runc",
              "ConsoleSize": [
                  0,
                  0
              ],
              "Isolation": "",
              "CpuShares": 0,
              "Memory": 0,
              "NanoCpus": 0,
              "CgroupParent": "",
              "BlkioWeight": 0,
              "BlkioWeightDevice": null,
              "BlkioDeviceReadBps": null,
              "BlkioDeviceWriteBps": null,
              "BlkioDeviceReadIOps": null,
              "BlkioDeviceWriteIOps": null,
              "CpuPeriod": 0,
              "CpuQuota": 0,
              "CpuRealtimePeriod": 0,
              "CpuRealtimeRuntime": 0,
              "CpusetCpus": "",
              "CpusetMems": "",
              "Devices": null,
              "DiskQuota": 0,
              "KernelMemory": 0,
              "MemoryReservation": 0,
              "MemorySwap": 0,
              "MemorySwappiness": -1,
              "OomKillDisable": false,
              "PidsLimit": 0,
              "Ulimits": null,
              "CpuCount": 0,
              "CpuPercent": 0,
              "IOMaximumIOps": 0,
              "IOMaximumBandwidth": 0
          },
          "GraphDriver": {
              "Name": "devicemapper",
              "Data": {
                  "DeviceId": "61",
                  "DeviceName": "docker-253:0-7207168-a1bf4a074e482562707a2f224a46dead5c480d2f5346e26398e131b9326ea9bc",
                  "DeviceSize": "10737418240"
              }
          },
          "Mounts": [
              {
                  "Type": "volume",
                  "Name": "2bcf8575371aa24b7675208e53345f4f0876e87b9a39b379d80689a6fce55f6a",
                  "Source": "/var/lib/docker/volumes/2bcf8575371aa24b7675208e53345f4f0876e87b9a39b379d80689a6fce55f6a/_data",
                  "Destination": "/opt/rke-tools",
                  "Driver": "local",
                  "Mode": "",
                  "RW": true,
                  "Propagation": ""
              },
              {
                  "Type": "bind",
                  "Source": "/etc/kubernetes",
                  "Destination": "/etc/kubernetes",
                  "Mode": "z",
                  "RW": true,
                  "Propagation": "rprivate"
              }
          ],
          "Config": {
              "Hostname": "localhost.localdomain",
              "Domainname": "",
              "User": "",
              "AttachStdin": false,
              "AttachStdout": false,
              "AttachStderr": false,
              "Tty": false,
              "OpenStdin": false,
              "StdinOnce": false,
              "Env": [
                  "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
              ],
              "Cmd": null,
              "Image": "rancher/hyperkube:v1.11.6-rancher1",
              "Volumes": null,
              "WorkingDir": "",
              "Entrypoint": [
                  "/opt/rke-tools/entrypoint.sh",
                  "kube-apiserver",
                  "--bind-address=0.0.0.0",
                  "--tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem",
                  "--etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem",
                  "--requestheader-extra-headers-prefix=X-Remote-Extra-",
                  "--requestheader-username-headers=X-Remote-User",
                  "--etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem",
                  "--requestheader-allowed-names=kube-apiserver-proxy-client",
                  "--service-cluster-ip-range=10.43.0.0/16",
                  "--enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota",
                  "--kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname",
                  "--cloud-provider=",
                  "--etcd-prefix=/registry",
                  "--insecure-bind-address=127.0.0.1",
                  "--storage-backend=etcd3",
                  "--requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem",
                  "--etcd-certfile=/etc/kubernetes/ssl/kube-node.pem",
                  "--proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem",
                  "--tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305",
                  "--secure-port=6443",
                  "--allow-privileged=true",
                  "--tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem",
                  "--kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem",
                  "--service-account-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem",
                  "--client-ca-file=/etc/kubernetes/ssl/kube-ca.pem",
                  "--kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem",
                  "--etcd-servers=https://172.17.0.1:2379,https://172.17.0.1:2379,https://172.17.0.1:2379",
                  "--proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem",
                  "--authorization-mode=Node,RBAC",
                  "--insecure-port=0",
                  "--requestheader-group-headers=X-Remote-Group",
                  "--service-node-port-range=30000-32767"
              ],
              "OnBuild": null,
              "Labels": {
                  "io.rancher.rke.container.name": "kube-apiserver",
                  "org.label-schema.build-date": "2018-12-20T16:43:39Z",
                  "org.label-schema.schema-version": "1.0",
                  "org.label-schema.vcs-ref": "1ed08cea7de947e05cc025a727a58973b5c45949",
                  "org.label-schema.vcs-url": "https://github.com/rancher/hyperkube.git"
              }
          },
          "NetworkSettings": {
              "Bridge": "",
              "SandboxID": "1e0727d2b5e06ef1a1f4da0ad3fc91e619f8288d512adf41565d4e174a277105",
              "HairpinMode": false,
              "LinkLocalIPv6Address": "",
              "LinkLocalIPv6PrefixLen": 0,
              "Ports": {},
              "SandboxKey": "/var/run/docker/netns/default",
              "SecondaryIPAddresses": null,
              "SecondaryIPv6Addresses": null,
              "EndpointID": "",
              "Gateway": "",
              "GlobalIPv6Address": "",
              "GlobalIPv6PrefixLen": 0,
              "IPAddress": "",
              "IPPrefixLen": 0,
              "IPv6Gateway": "",
              "MacAddress": "",
              "Networks": {
                  "host": {
                      "IPAMConfig": null,
                      "Links": null,
                      "Aliases": null,
                      "NetworkID": "e7ef2fcec1a2befbc201a33a80bf3aaa8e6fcf125d104c648492351508ac94d8",
                      "EndpointID": "abb852ffe19b95b765410915d3fe4a9dd05bc3dd8b171396a455f0a07fc22d89",
                      "Gateway": "",
                      "IPAddress": "",
                      "IPPrefixLen": 0,
                      "IPv6Gateway": "",
                      "GlobalIPv6Address": "",
                      "GlobalIPv6PrefixLen": 0,
                      "MacAddress": ""
                  }
              }
          }
      }
  ]
  </code></pre>
  
  <p><strong>update</strong></p>
  
  <p>rancher-cluster.yml</p>
  
  <pre><code>nodes:
    - address: 192.168.88.249
      internal_address: 172.17.0.1
      user: dockeruser
      role: [controlplane,worker,etcd]
    - address: 192.168.88.243
      internal_address: 172.17.0.1
      user: dockeruser
      role: [controlplane,worker,etcd]
    - address: 192.168.88.245
      internal_address: 172.17.0.1
      user: dockeruser
      role: [controlplane,worker,etcd]
  
  services:
    etcd:
      snapshot: true
      creation: 6h
      retention: 24h
  </code></pre>