<p>If I start up a fresh clean empty minikube and <code>helm install</code> the latest <code>stable/prometheus-operator</code> with strictly default settings I see four active Prometheus alarms.</p>

<p>In this super simplified scenario where I have a clean fresh minikube that is running absolutely nothing other than Prometheus, there should be no problems and no alarms. Are these alarms bogus or broken? Is something wrong with my setup or should I submit a bug report and disable these alarms for the time being?</p>

<p>Here are my basic setup steps:</p>

<pre><code>minikube delete
# Any lower memory/cpu settings will experience problems
minikube start --memory 10240 --cpus 4 --kubernetes-version v1.12.2
eval $(minikube docker-env)
helm init
helm repo update
# wait a minute for Helm Tiller to start up.
helm install --name my-prom stable/prometheus-operator
</code></pre>

<p>Wait several minutes for everything to start up, then run port forwarding on Prometheus server and on Grafana:</p>

<pre><code>kubectl port-forward service/my-prom-prometheus-operato-prometheus 9090:9090
kubectl port-forward service/my-prom-grafana 8080:80
</code></pre>

<p>Then go to <code>http://localhost:9090/alerts</code> and see:</p>

<pre><code>DeadMansSwitch (1 active)
KubeControllerManagerDown (1 active)
KubeSchedulerDown (1 active)
TargetDown (1 active)
</code></pre>

<p>Are these bogus? Is something genuinely wrong? Should I disable these?</p>

<p>Two of these alarms are missing metrics:</p>

<ul>
<li>KubeControllerManagerDown: <code>absent(up{job="kube-controller-manager"} == 1)</code></li>
<li>KubeSchedulerDown: <code>absent(up{job="kube-scheduler"} == 1)</code></li>
</ul>

<p>In <code>http://localhost:9090/config</code>, I don't see either job configured but I do see very closely related a jobs with <code>job_name</code> values of <code>default/my-prom-prometheus-operato-kube-controller-manager/0</code> and <code>default/my-prom-prometheus-operato-kube-scheduler/0</code>. This suggests that <code>job_name</code> values are supposed to match and there is a bug where they do not match. I also don't see any collected metrics for either job. Are slashes allowed in job names?</p>

<p>The other two alarms:</p>

<ul>
<li>DeadMansSwitch: The alarm expression is <code>vector(1)</code>. I have no idea what this is.</li>
<li>TargetDown: This alarm is being triggered over <code>up{job="kubelet"}</code> which has two metric values, one up with a value of 1.0 and one down with a value of 0.0. The up value is for <code>endpoint="http-metrics"</code> and the down valie is for <code>endpoint="cadvisor"</code>. Is that latter endpoint supposed to be up? Why wouldn't it be?</li>
</ul>

<p>I go to <code>http://localhost:9090/graph</code> and run <code>sum(up) by (job)</code> I see <code>1.0</code> values for all of:</p>

<pre><code>{job="node-exporter"}
{job="my-prom-prometheus-operato-prometheus"}
{job="my-prom-prometheus-operato-operator"}
{job="my-prom-prometheus-operato-alertmanager"}
{job="kubelet"}
{job="kube-state-metrics"}
{job="apiserver"}
</code></pre>

<p>fyi, <code>kubectl version</code> shows:</p>

<pre><code>Client Version: version.Info{Major:"1", Minor:"12", GitVersion:"v1.12.2", GitCommit:"17c77c7898218073f14c8d573582e8d2313dc740", GitTreeState:"clean", BuildDate:"2018-10-30T21:39:16Z", GoVersion:"go1.11.1", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"12", GitVersion:"v1.12.2", GitCommit:"17c77c7898218073f14c8d573582e8d2313dc740", GitTreeState:"clean", BuildDate:"2018-10-24T06:43:59Z", GoVersion:"go1.10.4", Compiler:"gc", Platform:"linux/amd64"}
</code></pre>

<p>The <a href="https://github.com/coreos/prometheus-operator/blob/773090e44e162e9508de0073efefc1152e6cf3d9/contrib/kube-prometheus/manifests/prometheus-rules.yaml#L870-L877" rel="noreferrer"><code>Watchdog</code></a> alert (formerly named as <code>DeadManSwitch</code>) is:</p>

<blockquote>
  <p>An alert meant to ensure that the entire alerting pipeline is functional.
            This alert is always firing, therefore it should always be firing in Alertmanager
            and always fire against a receiver.</p>
</blockquote>

<p>In Minikube, the <code>kube-controller-manager</code> and <code>kube-scheduler</code> listen by default on <em>127.0.0.1</em>, so Prometheus cannot scrape metrics from them. You need to start Minikube with these components listening on all interfaces:</p>

<pre><code>minikube start --kubernetes-version v1.12.2 \
--bootstrapper=kubeadm \
--extra-config=scheduler.address=0.0.0.0 \
--extra-config=controller-manager.address=0.0.0.0
</code></pre>

<p>Another cause of <code>TargetDown</code> is that the default service selectors created by Prometheus Operator helm chart don’t match the labels used by Minikube components. You need to match them by setting the <code>kubeControllerManager.selector</code> and <code>kubeScheduler.selector</code> helm parameters.</p>

<p>Take a look at this article: <a href="https://medium.com/@eduardobaitello/trying-prometheus-operator-with-helm-minikube-b617a2dccfa3" rel="noreferrer">Trying Prometheus Operator with Helm + Minikube</a>. It addresses all these problems, how to solve them and much more.</p>
