<p>I deployed apiserver on master node (core01) with following conf:</p>

<pre><code>core01&gt; /opt/bin/kube-apiserver \
  --insecure_bind_address=127.0.0.1 \
  --insecure_port=8080 \
  --kubelet_port=10250 \
  --etcd_servers=http://core01:2379,http://core02:2379,http://core03:2379 \
  --service-cluster-ip-range=10.1.0.0/16 \
  --allow_privileged=false \
  --logtostderr=true \
  --v=5 \
  --tls-cert-file="/var/run/kubernetes/apiserver_36kr.pem" \
  --tls-private-key-file="/var/run/kubernetes/apiserver_36kr.key" \
  --client-ca-file="/var/run/kubernetes/cacert.pem" \
  --kubelet-certificate-authority="/var/run/kubernetes/cacert.pem" \
  --kubelet-client-certificate="/var/run/kubernetes/kubelet_36kr.pem" \
  --kubelet-client-key="/var/run/kubernetes/kubelet_36kr.key"
</code></pre>

<p>On minion node (core02), I can call api from HTTPS:</p>

<pre><code>core02&gt; curl https://core01:6443/api/v1/nodes --cert /var/run/kubernetes/kubelet_36kr.pem --key /var/run/kubernetes/kubelet_36kr.key
&gt; GET /api/v1/nodes HTTP/1.1
&gt; Host: core01:6443
&gt; User-Agent: curl/7.42.1
&gt; Accept: */*
&gt; 
&lt; HTTP/1.1 200 OK
&lt; Content-Type: application/json
&lt; Date: Sat, 27 Jun 2015 15:33:50 GMT
&lt; Content-Length: 1577
&lt; 
{
  "kind": "NodeList",
  "apiVersion": "v1",
  "metadata": {
    "selfLink": "/api/v1/nodes",
    "resourceVersion": "510078"
  }, ....
</code></pre>

<p>However, I can not start kubelet on this minion. It always complain no credentials. </p>

<p>How can I make it work? Is there any doc on master &lt;-> minion communication authentication? Could you please give me the best practice?</p>

<hr>

<p>FYI, The command is following:</p>

<pre><code>core02&gt; /opt/bin/kubelet \
  --logtostderr=true \
  --v=0 \
  --api_servers=https://core01:6443 \
  --address=127.0.0.1 \
  --port=10250 \
  --allow-privileged=false \
  --tls-cert-file="/var/run/kubernetes/kubelet_36kr.pem" \
  --tls-private-key-file="/var/run/kubernetes/kubelet_36kr.key"
</code></pre>

<p>kubelet log is following:</p>

<pre><code>W0627 23:34:03.646311    3004 server.go:460] Could not load kubeconfig file /var/lib/kubelet/kubeconfig: stat /var/lib/kubelet/kubeconfig: no such file or directory. Trying auth path instead.
W0627 23:34:03.646520    3004 server.go:422] Could not load kubernetes auth path /var/lib/kubelet/kubernetes_auth: stat /var/lib/kubelet/kubernetes_auth: no such file or directory. Continuing with defaults.
I0627 23:34:03.646710    3004 manager.go:127] cAdvisor running in container: "/system.slice/sshd.service"
I0627 23:34:03.647292    3004 fs.go:93] Filesystem partitions: map[/dev/sda9:{mountpoint:/ major:0 minor:30} /dev/sda4:{mountpoint:/usr major:8 minor:4} /dev/sda6:{mountpoint:/usr/share/oem major:8 minor:6}]
I0627 23:34:03.648234    3004 manager.go:156] Machine: {NumCores:1 CpuFrequency:2399996 MemoryCapacity:1046294528 MachineID:29f94a4fad8b31668bd219ca511bdeb0 SystemUUID:4F4AF929-8BAD-6631-8BD2-19CA511BDEB0 BootID:fa1bea28-675e-4989-ad86-00797721a794 Filesystems:[{Device:/dev/sda9 Capacity:18987593728} {Device:/dev/sda4 Capacity:1031946240} {Device:/dev/sda6 Capacity:113229824}] DiskMap:map[8:0:{Name:sda Major:8 Minor:0 Size:21474836480 Scheduler:cfq} 8:16:{Name:sdb Major:8 Minor:16 Size:1073741824 Scheduler:cfq}] NetworkDevices:[{Name:eth0 MacAddress:52:54:71:f6:fc:b8 Speed:0 Mtu:1500} {Name:flannel0 MacAddress: Speed:10 Mtu:1472}] Topology:[{Id:0 Memory:1046294528 Cores:[{Id:0 Threads:[0] Caches:[{Size:32768 Type:Data Level:1} {Size:32768 Type:Instruction Level:1} {Size:4194304 Type:Unified Level:2}]}] Caches:[]}]}
I0627 23:34:03.649934    3004 manager.go:163] Version: {KernelVersion:4.0.5 ContainerOsVersion:CoreOS 695.2.0 DockerVersion:1.6.2 CadvisorVersion:0.15.1}
I0627 23:34:03.651758    3004 plugins.go:69] No cloud provider specified.
I0627 23:34:03.651855    3004 docker.go:289] Connecting to docker on unix:///var/run/docker.sock
I0627 23:34:03.652877    3004 server.go:659] Watching apiserver
E0627 23:34:03.748954    3004 reflector.go:136] Failed to list *api.Pod: the server has asked for the client to provide credentials (get pods)
E0627 23:34:03.750157    3004 reflector.go:136] Failed to list *api.Node: the server has asked for the client to provide credentials (get nodes)
E0627 23:34:03.751666    3004 reflector.go:136] Failed to list *api.Service: the server has asked for the client to provide credentials (get services)
I0627 23:34:03.758158    3004 plugins.go:56] Registering credential provider: .dockercfg
I0627 23:34:03.856215    3004 server.go:621] Started kubelet
E0627 23:34:03.858346    3004 kubelet.go:662] Image garbage collection failed: unable to find data for container /
I0627 23:34:03.869739    3004 kubelet.go:682] Running in container "/kubelet"
I0627 23:34:03.869755    3004 server.go:63] Starting to listen on 127.0.0.1:10250
E0627 23:34:03.899877    3004 event.go:185] Server rejected event '&amp;api.Event{TypeMeta:api.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:api.ObjectMeta{Name:"core02.13eba23275ceda25", GenerateName:"", Namespace:"default", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:util.Time{Time:time.Time{sec:0, nsec:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*util.Time)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil)}, InvolvedObject:api.ObjectReference{Kind:"Node", Namespace:"", Name:"core02", UID:"core02", APIVersion:"", ResourceVersion:"", FieldPath:""}, Reason:"starting", Message:"Starting kubelet.", Source:api.EventSource{Component:"kubelet", Host:"core02"}, FirstTimestamp:util.Time{Time:time.Time{sec:63571016043, nsec:856189989, loc:(*time.Location)(0x1ba6120)}}, LastTimestamp:util.Time{Time:time.Time{sec:63571016043, nsec:856189989, loc:(*time.Location)(0x1ba6120)}}, Count:1}': 'the server has asked for the client to provide credentials (post events)' (will not retry!)
I0627 23:34:04.021297    3004 factory.go:226] System is using systemd
I0627 23:34:04.021790    3004 factory.go:234] Registering Docker factory
I0627 23:34:04.022241    3004 factory.go:89] Registering Raw factory
I0627 23:34:04.144065    3004 manager.go:946] Started watching for new ooms in manager
I0627 23:34:04.144655    3004 oomparser.go:183] oomparser using systemd
I0627 23:34:04.145379    3004 manager.go:243] Starting recovery of all containers
I0627 23:34:04.293020    3004 manager.go:248] Recovery completed
I0627 23:34:04.343829    3004 status_manager.go:56] Starting to sync pod status with apiserver
I0627 23:34:04.343928    3004 kubelet.go:1683] Starting kubelet main sync loop.
E0627 23:34:04.457765    3004 event.go:185] Server rejected event '&amp;api.Event{TypeMeta:api.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:api.ObjectMeta{Name:"core02.13eba232995c8213", GenerateName:"", Namespace:"default", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:util.Time{Time:time.Time{sec:0, nsec:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*util.Time)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil)}, InvolvedObject:api.ObjectReference{Kind:"Node", Namespace:"", Name:"core02", UID:"core02", APIVersion:"", ResourceVersion:"", FieldPath:""}, Reason:"NodeReady", Message:"Node core02 status is now: NodeReady", Source:api.EventSource{Component:"kubelet", Host:"core02"}, FirstTimestamp:util.Time{Time:time.Time{sec:63571016044, nsec:452676115, loc:(*time.Location)(0x1ba6120)}}, LastTimestamp:util.Time{Time:time.Time{sec:63571016044, nsec:452676115, loc:(*time.Location)(0x1ba6120)}}, Count:1}': 'the server has asked for the client to provide credentials (post events)' (will not retry!)
E0627 23:34:04.659874    3004 event.go:185] Server rejected event '&amp;api.Event{TypeMeta:api.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:api.ObjectMeta{Name:"core02.13eba232a599cf8c", GenerateName:"", Namespace:"default", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:util.Time{Time:time.Time{sec:0, nsec:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*util.Time)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil)}, InvolvedObject:api.ObjectReference{Kind:"Node", Namespace:"", Name:"core02", UID:"core02", APIVersion:"", ResourceVersion:"", FieldPath:""}, Reason:"NodeReady", Message:"Node core02 status is now: NodeReady", Source:api.EventSource{Component:"kubelet", Host:"core02"}, FirstTimestamp:util.Time{Time:time.Time{sec:63571016044, nsec:658020236, loc:(*time.Location)(0x1ba6120)}}, LastTimestamp:util.Time{Time:time.Time{sec:63571016044, nsec:658020236, loc:(*time.Location)(0x1ba6120)}}, Count:1}': 'the server has asked for the client to provide credentials (post events)' (will not retry!)
</code></pre>

<p>The first two lines of the kubelet log file actually point to the underlying problem -- you aren't specifying any client credentials for the kubelet to connect to the master. </p>

<p>The <code>--tls-cert-file</code> and <code>--tls-private-key-file</code> arguments for the kubelet are used to configure the http server on the kubelet (if not specified, the kubelet <a href="https://github.com/GoogleCloudPlatform/kubernetes/blob/master/cmd/kubelet/app/server.go#L198">will generate a self-signed certificate for its https endpoint</a>). This certificate / key pair are not used as the client certificate presented to the master for authentication. </p>

<p>To specify credentials, there are two options: a kubeconfig file and a kubernetes_auth file. The later is deprecated, so I would recommend using a kubeconfig file. </p>

<p>Inside the kubeconfig file you need to specify either a bearer token or a client certificate that the kubelet should present to the apiserver. You can also specify the CA certificate for the apiserver (if you want the connection to be secure) or tell the kubelet to skip checking the certificate presented by the apiserver. Since you have certificates for the apiserver, I'd recommend adding the CA certificate to the kubeconfig file.</p>

<p>The kubeconfig file should look like:</p>

<pre><code>apiVersion: v1
kind: Config
users:
- name: kubelet
  user:
    client-certificate-data: &lt;base64-encoded-cert&gt;
    client-key-data: &lt;base64-encoded-key&gt;
clusters:
- name: local
  cluster:
    certificate-authority-data: &lt;base64-encoded-ca-cert&gt;
contexts:
- context:
    cluster: local
    user: kubelet
  name: service-account-context
current-context: service-account-context
</code></pre>

<p>To generate the base64 encoded client cert, you should be able to run something like <code>cat /var/run/kubernetes/kubelet_36kr.pem | base64</code>. If you don't have the CA certificate handy, you can replace the <code>certificate-authority-data: &lt;base64-encoded-ca-cert&gt;</code> with <code>insecure-skip-tls-verify: true</code>. </p>

<p>If you put this file at <code>/var/lib/kubelet/kubeconfig</code> it should get picked up automatically. Otherwise, you can use the <code>--kubeconfig</code> argument to specify a custom location. </p>
