<p>I am using the ansible scripts from <code>kargo</code> to build my cluster. I am unable to find where the data is stored in etcd3, despite looking over the verbose logs from the apiserver.</p>

<p>Here is what I see the hyperkube apiserver logs:</p>

<pre><code>$ docker logs k8s_kube-apiserver.fd19548d_kube-apiserver-kube-master-01_kube-system_2f6ad6b0bf81ca6a0e2b4d499a25fc89_aa25196e
[[ SNIP ]]
I0127 23:31:55.871267       1 storage_factory.go:242] storing { podtemplates} in v1, reading as __internal from { /registry [https://10.60.68.11:2379 https://10.60.68.39:2379 https://10.60.68.35:2379] /etc/ssl/etcd/ssl/node-kube-master-01-key.pem /etc/ssl/etcd/ssl/node-kube-master-01.pem /etc/ssl/etcd/ssl/ca.pem true 1000 &lt;nil&gt;}
I0127 23:31:55.875975       1 storage_factory.go:242] storing { events} in v1, reading as __internal from { /registry [https://10.60.68.11:2379 https://10.60.68.39:2379 https://10.60.68.35:2379] /etc/ssl/etcd/ssl/node-kube-master-01-key.pem /etc/ssl/etcd/ssl/node-kube-master-01.pem /etc/ssl/etcd/ssl/ca.pem true 1000 &lt;nil&gt;}
I0127 23:31:55.876169       1 reflector.go:234] Listing and watching *api.PodTemplate from k8s.io/kubernetes/pkg/storage/cacher.go:215
I0127 23:31:55.877950       1 compact.go:55] compactor already exists for endpoints [https://10.60.68.11:2379 https://10.60.68.39:2379 https://10.60.68.35:2379]
I0127 23:31:55.878148       1 storage_factory.go:242] storing { limitranges} in v1, reading as __internal from { /registry [https://10.60.68.11:2379 https://10.60.68.39:2379 https://10.60.68.35:2379] /etc/ssl/etcd/ssl/node-kube-master-01-key.pem /etc/ssl/etcd/ssl/node-kube-master-01.pem /etc/ssl/etcd/ssl/ca.pem true 1000 &lt;nil&gt;}
I0127 23:31:55.879372       1 compact.go:55] compactor already exists for endpoints [https://10.60.68.11:2379 https://10.60.68.39:2379 https://10.60.68.35:2379]
</code></pre>

<p>the <code>hyperkube apiserver</code> is started with these arguments:</p>

<pre><code>$ docker inspect k8s_kube-apiserver.b6395694_kube-apiserver-kube-master-01_kube-system_2f6ad6b0bf81ca6a0e2b4d499a25fc89_4338b366
[
    {
        "Id": "33c76fa64bbd5d5a656e329cf87ed3707077659c69dc281127751f594460242b",
        "Created": "2017-01-27T23:35:10.691147667Z",
        "Path": "/hyperkube",
        "Args": [
            "apiserver",
            "--advertise-address=10.60.68.23",
            "--etcd-servers=https://10.60.68.11:2379,https://10.60.68.39:2379,https://10.60.68.35:2379",
            "--etcd-quorum-read=true",
            "--etcd-cafile=/etc/ssl/etcd/ssl/ca.pem",
            "--etcd-certfile=/etc/ssl/etcd/ssl/node-kube-master-01.pem",
            "--etcd-keyfile=/etc/ssl/etcd/ssl/node-kube-master-01-key.pem",
            "--insecure-bind-address=127.0.0.1",
            "--apiserver-count=3",
            "--admission-control=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,ResourceQuota",
            "--service-cluster-ip-range=10.233.0.0/18",
            "--service-node-port-range=30000-32767",
            "--client-ca-file=/etc/kubernetes/ssl/ca.pem",
            "--basic-auth-file=/etc/kubernetes/users/known_users.csv",
            "--tls-cert-file=/etc/kubernetes/ssl/apiserver.pem",
            "--tls-private-key-file=/etc/kubernetes/ssl/apiserver-key.pem",
            "--token-auth-file=/etc/kubernetes/tokens/known_tokens.csv",
            "--service-account-key-file=/etc/kubernetes/ssl/apiserver-key.pem",
            "--secure-port=443",
            "--insecure-port=8080",
            "--v=4",
            "--allow-privileged=true",
            "--cloud-provider=openstack",
            "--cloud-config=/etc/kubernetes/cloud_config",
            "--anonymous-auth=False"
        ],
</code></pre>

<p>No where does it override the default etcd prefix of <code>/registry</code>. I have no idea where apiserver is storing data.</p>

<pre><code>$ docker exec -it etcd3 etcdctl --peers https://10.60.68.11:2379 ls /registry
Error:  100: Key not found (/registry) [163]
</code></pre>

<p>To get keys and values stored in etcd v3 by kubernetes:</p>

<pre><code>ETCDCTL_API=3 etcdctl --endpoints=http://localhost:2379 get --prefix /registry
</code></pre>

<p>To get 1 specified key and value from etcd v3, e.g.:</p>

<pre><code>ETCDCTL_API=3 etcdctl --endpoints=http://localhost:2379 get /registry/services/specs/default/kubernetes
</code></pre>

<p>Based on: <a href="https://github.com/coreos/etcd/blob/master/Documentation/dev-guide/interacting_v3.md" rel="nofollow noreferrer">https://github.com/coreos/etcd/blob/master/Documentation/dev-guide/interacting_v3.md</a></p>
