<p>I'm trying to bring kubernetes api server up using etcd config (kubernetes uses go-etcd which has a method to read all parameters from a configuration file):</p>

<pre><code>{ 
  "cluster": {
    "machines": [ "https://my-public-hostname:2379" ] 
  }, 
  "config": { 
  "certFile": "/etc/ssl/etcd/client.pem", 
  "keyFile": "/etc/ssl/etcd/client.key.pem", 
  "caCertFiles": [ 
  "/etc/ssl/etcd/ca.pem" 
  ], 
    "timeout": 5, 
    "consistency": "WEAK" 
  } 
}
</code></pre>

<p>But fails in kube-apiserver because it cannot reach etcd successfully. I think this is because it tries to sync the cluster... but I don't know.</p>

<p>I have created a (etcd) cluster using internal ips for advertise and client addresses except for the listen-client-urls which is set to 0.0.0.0/0. Also, the whole cluster is behind a load balancer which is accessible through <code>my-public-hostname</code>.</p>

<p>Inside the container (because i'm using <code>hyperkube</code>), <code>etcdctl</code> won't work unless I set the '--no-sync' parameter. If i use etcdctl without that parameter it suspiciously fails like kube-apiserver does. But I wasn't able to check the piece of code in kubernetes which does the cluster syncrhonization...</p>

<p>Any ideas?</p>

<p>Thanks in advance.</p>

<p><strong>EDIT:</strong></p>

<p>It seems to be an error related to the current etcd client in kubernetes (<a href="https://github.com/coreos/go-etcd" rel="nofollow">https://github.com/coreos/go-etcd</a>), which is not the newest one (<a href="https://github.com/coreos/etcd/client" rel="nofollow">https://github.com/coreos/etcd/client</a>). I tested this empirically and "etcd/client" works but "go-etcd" doesn't, you can check this test here: <strike><a href="https://github.com/glerchundi/etcd-go-clients-test" rel="nofollow">https://github.com/glerchundi/etcd-go-clients-test</a></strike>. </p>

<p>It's worth noting that there is an ongoing work to migrate go-etcd to etcd/client in kubernetes: <a href="https://github.com/kubernetes/kubernetes/issues/11962" rel="nofollow">https://github.com/kubernetes/kubernetes/issues/11962</a>.</p>

<p>Can anyone from the Kubernetes team confirm this?</p>

<p><strong>APPENDIX 1</strong></p>

<p>I'm trying to run kubernetes in CoreOS and: <code>flannel</code> works, <code>locksmithd</code> works, <code>fleet</code> works (they access to etcd using the very same etcd client credentials) so it's probably something related to how kubernetes accesses to the etcd endpoint.</p>

<p><strong>APPENDIX 2</strong> (these commands are executed inside the hyperkube container, concretely this one: <code>gcr.io/google_containers/hyperkube:v1.0.6</code>)</p>

<p>etcdctl without --no-sync fails outputting this:</p>

<pre><code>root@98b2524464f1:/# etcdctl --cert-file="/etc/ssl/etcd/client.pem" --key-file="/etc/ssl/etcd/client.key.pem" --ca-file="/etc/ssl/etcd/ca.pem" --peers="http//my-public-hostname:2379" ls / 
Error: 501: All the given peers are not reachable (failed to propose on members [https://10.1.0.1:2379 https://10.1.0.0:2379 https://10.1.0.2:2379] twice [last error: Get https://10.1.0.0:2379/v2/keys/?quorum=false&amp;recursive=false&amp;sorted=false: dial tcp 10.1.0.0:2379: i/o timeout]) [0]
</code></pre>

<p>And kube-apiserver with this:</p>

<pre><code>root@98b2524464f1:/# /hyperkube \ 
apiserver \ 
--bind-address=0.0.0.0 \ 
--etcd_config=/etc/kubernetes/ssl/etcd.json \ 
--allow-privileged=true \ 
--service-cluster-ip-range=10.3.0.0/24 \ 
--secure_port=443 \ 
--advertise-address=10.0.0.2 \ 
--admission-control=NamespaceLifecycle,NamespaceExists,LimitRanger,SecurityContextDeny,ServiceAccount,ResourceQuota \ 
--tls-cert-file=/etc/kubernetes/ssl/apiserver.pem \ 
--tls-private-key-file=/etc/kubernetes/ssl/apiserver.key.pem \ 
--client-ca-file=/etc/kubernetes/ssl/ca.pem \ 
--service-account-key-file=/etc/kubernetes/ssl/apiserver.key.pem

F1002 09:47:29.348527 384 controller.go:80] Unable to perform initial IP allocation check: unable to refresh the service IP block: 501: All the given peers are not reachable (failed to propose on members [https://my-public-hostname:2379] twice [last error: Get https://my-public-hostname:2379/v2/keys/registry/ranges/serviceips?quorum=false&amp;recursive=false&amp;sorted=false: dial tcp: i/o timeout]) [0]
</code></pre>

<p><strong>APPENDIX 3</strong></p>

<pre><code>etcd #0:
  etcd2:
    name: etcd0
    initial-cluster-state: new
    initial-cluster: etcd0=http://10.1.0.0:2380,etcd1=http://10.1.0.1:2380,etcd2=http://10.1.0.2:2380
    data-dir: /var/lib/etcd2
    advertise-client-urls: https://10.1.0.0:2379
    initial-advertise-peer-urls: http://10.1.0.0:2380
    listen-client-urls: https://0.0.0.0:2379
    listen-peer-urls: http://10.1.0.0:2380
    client-cert-auth: true
    trusted-ca-file: /etc/ssl/etcd/certs/ca-chain.cert.pem
    cert-file: /etc/ssl/etcd/certs/etcd-server.cert.pem
    key-file: /etc/ssl/etcd/private/etcd-server.key.pem

etcd #1:
  etcd2:
    name: etcd1
    initial-cluster-state: new
    initial-cluster: etcd0=http://10.1.0.0:2380,etcd1=http://10.1.0.1:2380,etcd2=http://10.1.0.2:2380
    data-dir: /var/lib/etcd2
    advertise-client-urls: https://10.1.0.1:2379
    initial-advertise-peer-urls: http://10.1.0.1:2380
    listen-client-urls: https://0.0.0.0:2379
    listen-peer-urls: http://10.1.0.1:2380
    client-cert-auth: true
    trusted-ca-file: /etc/ssl/etcd/certs/ca-chain.cert.pem
    cert-file: /etc/ssl/etcd/certs/etcd-server.cert.pem
    key-file: /etc/ssl/etcd/private/etcd-server.key.pem

etcd #2:
  etcd2:
    name: etcd2
    initial-cluster-state: new
    initial-cluster: etcd0=http://10.1.0.0:2380,etcd1=http://10.1.0.1:2380,etcd2=http://10.1.0.2:2380
    data-dir: /var/lib/etcd2
    advertise-client-urls: https://10.1.0.2:2379
    initial-advertise-peer-urls: http://10.1.0.2:2380
    listen-client-urls: https://0.0.0.0:2379
    listen-peer-urls: http://10.1.0.2:2380
    client-cert-auth: true
    trusted-ca-file: /etc/ssl/etcd/certs/ca-chain.cert.pem
    cert-file: /etc/ssl/etcd/certs/etcd-server.cert.pem
    key-file: /etc/ssl/etcd/private/etcd-server.key.pem
</code></pre>

<p>Finally I find out what was causing this problem. Timeout was not defined correctly because <code>go-etcd</code> unmarshalls json timeout value into time.Duration which uses nanoseconds as a base unit. So that for a value of 1s, 1000000000 should be written.</p>

<p>Following the example above:</p>

<pre><code>{ 
  "cluster": {
    "machines": [ "https://my-public-hostname:2379" ] 
  }, 
  "config": { 
    "certFile": "/etc/ssl/etcd/client.pem", 
    "keyFile": "/etc/ssl/etcd/client.key.pem", 
    "caCertFiles": [ 
      "/etc/ssl/etcd/ca.pem" 
    ], 
    "timeout": 5000000000, 
    "consistency": "WEAK" 
  } 
}
</code></pre>
