<p>I'm working on memory monitoring using Prometheus (prometheus-operator Helm chart).
While investigating values I've noticed that memory usage (<code>container_memory_working_set_bytes </code>) is being scraped from two endpoints:</p>
<ul>
<li><code>/metrics/cadvisor</code></li>
<li><code>/metrics/resource/v1alpha1</code> (<code>/metrics/resource</code> from kubernetes 1.18)</li>
</ul>
<p>I've figured out how to disable one of the endpoints in the chart but I'd like to understand the purpose of both.<br/>
I understand that <code>/metrics/cadvisor</code> returns three values - pod's container (or more if a pod has multiple containers), some special container <code>POD</code> (is it some internal memory usage to run a POD service?) and a sum of all containers (then the result has empty label <code>container=&quot;&quot;</code>).<br/>
On the other hand <code>/metrics/resource/v1alpha1</code> returns only memory usage of a pod's containers (without <code>container=&quot;POD&quot;</code> and without sum of these <code>container=&quot;&quot;</code>)</p>
<p>Is <code>/metrics/resource/v1alpha1</code> then planned to replace <code>/metrics/cadvisor</code> as a single source of metrics?
Seeing that both endpoints (both are enabled by default in <code>prometheus-operator</code>) return the same metrics any <code>sum()</code> queries can return values 2 as big as a real memory usage.</p>
<p>Appreciate any clarification in this subject!</p>

<p>Answer is partial</p>
<blockquote>
<p>I understand that /metrics/cadvisor returns three values - pod's
container (or more if a pod has multiple containers), some special
container POD (is it some internal memory usage to run a POD service?)
and a sum of all containers (then the result has empty label
container=&quot;&quot;).</p>
</blockquote>
<p><code>container_name==&quot;POD&quot;</code> is the &quot;pause&quot; container for the pods. The <a href="https://groups.google.com/g/kubernetes-users/c/jVjv0QK4b_o" rel="noreferrer">pause container</a> is a container which holds the network namespace for the pod. Kubernetes creates pause containers to acquire the respective pod’s IP address and set up the network namespace for all other containers that join that pod. This container is a part of whole ecosystem and it starts  first in pods to configure PODs network in the first place prior to scheduling  another pods. After pod has been started  - there is nothing to do  for pause container.</p>
<p>Pause container code  for your reference: <a href="https://github.com/kubernetes/kubernetes/tree/master/build/pause" rel="noreferrer">https://github.com/kubernetes/kubernetes/tree/master/build/pause</a></p>
<p>Example of pause containers:</p>
<pre><code>docker ps |grep pause
k8s_POD_etcd-master-1_kube-system_ea5105896423fc919bf9bfc0ab339888_0
k8s_POD_kube-scheduler-master-1_kube-system_155707e0c19147c8dc5e997f089c0ad1_0
k8s_POD_kube-apiserver-master-1_kube-system_fe660a7e8840003352195a8c40a01ef8_0
k8s_POD_kube-controller-manager-master-1_kube-system_807045fe48b23a157f7fe1ef20001ba0_0
k8s_POD_kube-proxy-76g9l_kube-system_e2348a94-eb96-4630-86b2-1912a9ce3a0f_0
k8s_POD_kube-flannel-ds-amd64-76749_kube-system_bf441436-bca3-4b49-b6fb-9e031ef7513d_0
</code></pre>
<p><code>container_name!==&quot;POD&quot;</code>  It filters out metric streams for the pause container, not metadata generally. Most people, if they want to graph the containers in their pod, don't want to see resource usage for the pause container, as it doesn't do much. The name of the pause container is an implementation detail of some container runtimes, but doesn't apply to all, and isn't guaranteed to stick around.</p>
<p>Official (obsolete v1.14) <a href="https://v1-14.docs.kubernetes.io/docs/tasks/debug-application-cluster/resource-usage-monitoring/" rel="noreferrer">page</a> shows differences between cadvisor and metrics resource monitoring:</p>
<p>Kubelet</p>
<blockquote>
<p>The Kubelet acts as a bridge between the Kubernetes master and
the nodes. It manages the pods and containers running on a machine.
Kubelet translates each pod into its constituent containers and
fetches individual container usage statistics from the container
runtime, through the container runtime interface. For the legacy
docker integration, it fetches this information from cAdvisor. It then
exposes the aggregated pod resource usage statistics through the
kubelet resource metrics api. This api is served at
/metrics/resource/v1alpha1 on the kubelet’s authenticated and
read-only ports.</p>
</blockquote>
<p><a href="https://github.com/google/cadvisor" rel="noreferrer">cAdvisor</a></p>
<blockquote>
<p>cAdvisor is an open source container resource usage and
performance analysis agent. It is purpose-built for containers and
supports Docker containers natively. In Kubernetes, cAdvisor is
integrated into the Kubelet binary. cAdvisor auto-discovers all
containers in the machine and collects CPU, memory, filesystem, and
network usage statistics. cAdvisor also provides the overall machine
usage by analyzing the ‘root’ container on the machine.</p>
</blockquote>
<p>Also you should know that kubelet  exposes metrics in /metrics/cadvisor, /metrics/resource and /metrics/probes endpoints. Those 3 metrics do not have same lifecycle.</p>
<p>As per <a href="https://raw.githubusercontent.com/helm/charts/master/stable/prometheus-operator/values.yaml" rel="noreferrer">helm prometheus values yaml</a> - there are 3 options and you can disable what you dont need</p>
<pre><code>    ## Enable scraping /metrics/cadvisor from kubelet's service
    ##
    cAdvisor: true

    ## Enable scraping /metrics/probes from kubelet's service
    ##
    probes: true

    ## Enable scraping /metrics/resource from kubelet's service
    ##
    resource: true
    # From kubernetes 1.18, /metrics/resource/v1alpha1 renamed to /metrics/resource
    resourcePath: &quot;/metrics/resource/v1alpha1&quot; 
</code></pre>
<p>My opinion <code>/metrics/resource/</code> wont replace google's cadvisor. Just disable in your case what you dont need. It just depends on your needs. For example, I found an article <a href="https://rtfm.co.ua/en/kubernetes-monitoring-with-prometheus-exporters-a-service-discovery-and-its-roles/" rel="noreferrer">Kubernetes: monitoring with Prometheus – exporters, a Service Discovery, and its roles</a> where 4 diff tools being used to monitor everything.</p>
<ol>
<li><p>metrics-server – CPU, memory, file-descriptors, disks, etc of the cluster</p>
</li>
<li><p>cAdvisor – a Docker daemon metrics – containers monitoring</p>
</li>
<li><p>kube-state-metrics –  deployments, pods, nodes</p>
</li>
<li><p>node-exporter: EC2 instances metrics – CPU, memory, network</p>
</li>
</ol>
<p>In your case, to monitor memory i believe it will be enough 1 :)</p>
