<p>I have a Kubernetes 1.5.3 cluster brought up on two nodes using kubeadm (details below). I've been trying to start up <a href="https://coreos.com/blog/the-prometheus-operator.html" rel="nofollow noreferrer">prometheus-operator</a> tonight. After I deleted a resource of kind=Prometheus, my cluster got in to a very strange state, where when I attempted to recreate the prometheus resource, no pod was ever created, though the TPR exists. Investigation lead me to:</p>

<p><code>
rrix@hypervisor01:~$ kubectl logs -n kube-system kube-apiserver-hypervisor01 | tail -10
W0226 03:57:34.641244       1 listers.go:69] can not retrieve list of objects using index : Index with name namespace does not exist
W0226 03:57:34.645073       1 listers.go:69] can not retrieve list of objects using index : Index with name namespace does not exist
W0226 03:58:04.642150       1 listers.go:69] can not retrieve list of objects using index : Index with name namespace does not exist
W0226 03:58:04.647953       1 listers.go:69] can not retrieve list of objects using index : Index with name namespace does not exist
W0226 03:58:34.642118       1 listers.go:69] can not retrieve list of objects using index : Index with name namespace does not exist
W0226 03:58:34.646427       1 listers.go:69] can not retrieve list of objects using index : Index with name namespace does not exist
W0226 03:59:04.642189       1 listers.go:69] can not retrieve list of objects using index : Index with name namespace does not exist
W0226 03:59:04.647978       1 listers.go:69] can not retrieve list of objects using index : Index with name namespace does not exist
W0226 03:59:34.646129       1 listers.go:69] can not retrieve list of objects using index : Index with name namespace does not exist
W0226 03:59:34.666355       1 listers.go:69] can not retrieve list of objects using index : Index with name namespace does not exist
</code></p>

<p>The workaround in this <a href="https://github.com/rancher/rancher/issues/7626" rel="nofollow noreferrer">rancher github issue</a> is to simply restart the apiserver, which would be fine, except this didn't resolve the issue. Otherwise, it points to high load on the etcd, which as you can see below, is hosted as a pod, on the host which currently has a load of less than 1. etcdctl commands run successfully, leading me to believe that the etcd itself is fine. For good measure I bumped the etcd docker container, and it still works fine. From here, however, I am lost and need some assistance getting my cluster back in working order. </p>

<p>Cluster overview:</p>

<p><code>
rrix@hypervisor01:~$ kubectl get no
NAME                           STATUS         AGE
hypervisor01                   Ready,master   2d
kubes01.pss9.kickass.systems   Ready          2d
rrix@hypervisor01:~$ kubectl get -n kube-system po
NAME                                   READY     STATUS    RESTARTS   AGE
dummy-2088944543-2p3zf                 1/1       Running   1          2d
etcd-hypervisor01                      1/1       Running   2          2d
kube-apiserver-hypervisor01            1/1       Running   3          2d
kube-controller-manager-hypervisor01   1/1       Running   1          2d
kube-discovery-1769846148-v8h50        1/1       Running   1          2d
kube-dns-2924299975-3s26d              4/4       Running   4          2d
kube-proxy-vpw73                       1/1       Running   1          2d
kube-proxy-zfh13                       1/1       Running   0          2d
kube-registry-proxy-6hhk9              1/1       Running   1          2d
kube-registry-proxy-nl1s1              1/1       Running   0          2d
kube-registry-v0-4d94t                 1/1       Running   0          2d
kube-scheduler-hypervisor01            1/1       Running   1          2d
rrix@hypervisor01:~$ kubectl get po
NAME                                  READY     STATUS    RESTARTS   AGE
kube-flannel-ds-589fw                 2/2       Running   3          2d
kube-flannel-ds-7f5sx                 2/2       Running   0          2d
kube-state-metrics-3229993571-20nbc   1/1       Running   0          1h
node-exporter-1cdlj                   1/1       Running   0          1h
node-exporter-jc54s                   1/1       Running   0          1h
prometheus-operator-996254120-0wzg6   1/1       Running   0          2h
</code></p>

<p><code>
E0226 16:20:47.861762       1 pet_set.go:272] Error syncing StatefulSet default/prometheus-prometheus-k8s, requeuing: Failed to create prometheus-prometheus-k8s-db-prometheus-prometheus-k8s-0: PersistentVolumeClaim "prometheus-prometheus-k8s-db-prometheus-prometheus-k8s-0" is invalid: spec.resources[storage]: Required value
I0226 16:20:47.862793       1 event.go:217] Event(api.ObjectReference{Kind:"StatefulSet", Namespace:"default", Name:"prometheus-prometheus-k8s", UID:"272d42fd-fbd1-11e6-9ae6-a0481cb808c8", APIVersion:"apps", ResourceVersion:"304212", FieldPath:""}): type: 'Warning' reason: 'FailedCreate' pvc: prometheus-prometheus-k8s-db-prometheus-prometheus-k8s-0, error: PersistentVolumeClaim "prometheus-prometheus-k8s-db-prometheus-prometheus-k8s-0" is invalid: spec.resources[storage]: Required value
E0226 16:20:47.865352       1 pet_set.go:272] Error syncing StatefulSet default/prometheus-prometheus-services, requeuing: Failed to create prometheus-prometheus-services-db-prometheus-prometheus-services-0: PersistentVolumeClaim "prometheus-prometheus-services-db-prometheus-prometheus-services-0" is invalid: spec.resources[storage]: Required value
I0226 16:20:47.865472       1 event.go:217] Event(api.ObjectReference{Kind:"StatefulSet", Namespace:"default", Name:"prometheus-prometheus-services", UID:"1899b833-fbd2-11e6-9ae6-a0481cb808c8", APIVersion:"apps", ResourceVersion:"304733", FieldPath:""}): type: 'Warning' reason: 'FailedCreate' pvc: prometheus-prometheus-services-db-prometheus-prometheus-services-0, error: PersistentVolumeClaim "prometheus-prometheus-services-db-prometheus-prometheus-services-0" is invalid: spec.resources[storage]: Required value
E0226 16:21:17.854692       1 pet_set.go:272] Error syncing StatefulSet default/prometheus-prometheus-k8s, requeuing: Failed to create prometheus-prometheus-k8s-db-prometheus-prometheus-k8s-0: PersistentVolumeClaim "prometheus-prometheus-k8s-db-prometheus-prometheus-k8s-0" is invalid: spec.resources[storage]: Required value
I0226 16:21:17.855043       1 event.go:217] Event(api.ObjectReference{Kind:"StatefulSet", Namespace:"default", Name:"prometheus-prometheus-k8s", UID:"272d42fd-fbd1-11e6-9ae6-a0481cb808c8", APIVersion:"apps", ResourceVersion:"304212", FieldPath:""}): type: 'Warning' reason: 'FailedCreate' pvc: prometheus-prometheus-k8s-db-prometheus-prometheus-k8s-0, error: PersistentVolumeClaim "prometheus-prometheus-k8s-db-prometheus-prometheus-k8s-0" is invalid: spec.resources[storage]: Required value
E0226 16:21:17.858436       1 pet_set.go:272] Error syncing StatefulSet default/prometheus-prometheus-services, requeuing: Failed to create prometheus-prometheus-services-db-prometheus-prometheus-services-0: PersistentVolumeClaim "prometheus-prometheus-services-db-prometheus-prometheus-services-0" is invalid: spec.resources[storage]: Required value
I0226 16:21:17.858561       1 event.go:217] Event(api.ObjectReference{Kind:"StatefulSet", Namespace:"default", Name:"prometheus-prometheus-services", UID:"1899b833-fbd2-11e6-9ae6-a0481cb808c8", APIVersion:"apps", ResourceVersion:"304733", FieldPath:""}): type: 'Warning' reason: 'FailedCreate' pvc: prometheus-prometheus-services-db-prometheus-prometheus-services-0, error: PersistentVolumeClaim "prometheus-prometheus-services-db-prometheus-prometheus-services-0" is invalid: spec.resources[storage]: Required value
</code></p>

<p>User error <em>facepalm</em>, didn't think to look in the manager logs.</p>
