title: Proper way to configure AlertManager and Prometheus ServiceMonitors with kube-prometheus-stack
body: <p>I've been reading through the github docs for kube-prometheus-stack trying to figure out the correct way to configure ServiceMonitors and AlertManager.</p>
  <p>kube-prometheus-stack points back to the prometheus-operator docs for <a href="https://github.com/prometheus-operator/prometheus-operator/blob/master/Documentation/user-guides/getting-started.md" rel="nofollow noreferrer">Getting Started</a> and <a href="https://github.com/prometheus-operator/prometheus-operator/blob/master/Documentation/user-guides/alerting.md" rel="nofollow noreferrer">Alerting</a>.</p>
  <p>Here's where I could use some pointers:</p>
  <p><em><strong>Regarding ServiceMonitors:</strong></em></p>
  <ul>
  <li>Generally speaking, should there always be 1 <code>ServiceMonitor</code> object to 1 k8s <code>Service</code>?</li>
  </ul>
  <p>In my values-override.yaml override that I'm passing in with the helm install, I specified the below configuration attempting to have Prometheus pick up any <code>ServiceMonitor</code> object created:</p>
  <pre class="lang-yaml prettyprint-override"><code>prometheus:
    prometheusSpec:
      serviceMonitorSelectorNilUsesHelmValues: false
  </code></pre>
  <ul>
  <li>Once the <code>ServiceMonitor</code> is created does Prometheus pick up the change automatically?</li>
  </ul>
  <p><em><strong>Regarding Alerting:</strong></em></p>
  <ul>
  <li><p>Should my global configuration items (like <code>smtp_from</code>, <code>smtp_smarthost</code>, etc) be added in a local values-override.yaml file that I pass in during helm install? (<code>helm install -f values-overrides.yaml prometheus prometheus-community/kube-prometheus-stack</code>) -- or should this configuration go in as part of the <code>AlertmanagerConfig</code> object?</p>
  </li>
  <li><p>Speaking of <code>AlertmanagerConfig</code> - is the intention that there will only be ONE of these objects, or is it appropriate to create one for each alert route?</p>
  </li>
  </ul>
  <p>And finally at this point, my understanding is that the &quot;routes&quot; in the configuration behind AlertManager serve as the funnel through which any notifications will flow. So if any rule I create is triggered, AlertManager will find an appropriate matching route through which to send the the alert.  This, in turn, means that there is no coupling between rules and alerts because the AlertmanagerConfig can be changed at any time to alter how alerts are sent out.</p>
  <p>The learning curve for this seems rather steep... and I haven't even scratched the surface of scraping yet (pun intended). Any insight, confirmation, or general pointing-in-the-right-direction is GREATLY appreciated.</p>