<p>We have a Kubernetes 1.7.8 clusters deployed with Kops 1.7 in HA with three masters. The cluster has 10 nodes and around 400 pods.</p>

<p>The cluster has heapster, prometheus, and ELK (collecting logs for some pods).</p>

<p>We are seeing a very high activity in the masters, over 90% of CPU used by the api-server.</p>

<p>Checking prometheus numbers we can see that near 5000 requests to the kube-apiserver are WATCH verbs, the rest are less than 50 request (GET, LIST, PATCH, PUT).</p>

<p>Almost all requests are reported with client "Go-Http-client/2.0" (the default User Agent for the Go HTTP library).</p>

<p>Is this a normal situation?</p>

<p>How can we debug which are the pods sending these requests? (How can we add the source IP to the kube-apiserver logs?)</p>

<p>[kube-apiserver.manifest][1]</p>

<p>Thanks,
Charles</p>

<pre><code>[1]: https://pastebin.com/nGxSXuZb
</code></pre>

<p>Regarding the Kubernetes architecture this is a normal behavior because all kubernetes cluster components are calling the api-server to watch for changes. </p>

<p>That is why you have more than 5000 WATCH entries in your logs. Please take a look how the <a href="https://kubernetes.io/docs/admin/high-availability/building/" rel="nofollow noreferrer">kubernetes cluster is managed by kube api server</a> and how the <a href="https://kubernetes.io/docs/concepts/architecture/master-node-communication/" rel="nofollow noreferrer">master-node comunication is organized</a></p>
