title: Prometheus dns service discovery in docker swarm
body: <p>I'm searching for some monitor &amp; alert solutions for my services. I found following nice related works.</p>
  
  <ul>
  <li><a href="https://dzone.com/articles/swarmprom-prometheus-monitoring-for-docker-swarm" rel="noreferrer">Prometheus monitoring for docker swarm</a></li>
  <li><a href="https://chmod666.org/2017/08/monitoring-a-docker-swarm-cluster-with-prometheus" rel="noreferrer">Monitoring a docker swarm cluster with Prometheus</a></li>
  </ul>
  
  <p>Both works use dns service discovery to monitor multiple replicas of services. </p>
  
  <p>I've tried to replay these work, but I found I can only get single backend container ip.</p>
  
  <pre><code># dig A node-exporter
  
  ; &lt;&lt;&gt;&gt; DiG 9.10.4-P8 &lt;&lt;&gt;&gt; A node-exporter
  ;; global options: +cmd
  ;; Got answer:
  ;; -&gt;&gt;HEADER&lt;&lt;- opcode: QUERY, status: NOERROR, id: 18749
  ;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0
  
  ;; QUESTION SECTION:
  ;node-exporter.         IN  A
  
  ;; ANSWER SECTION:
  node-exporter.      600 IN  A   10.0.0.42
  
  ;; Query time: 0 msec
  ;; SERVER: 127.0.0.11#53(127.0.0.11)
  ;; WHEN: Mon Jan 29 02:57:51 UTC 2018
  ;; MSG SIZE  rcvd: 60
  </code></pre>
  
  <p>When I inspect the service, I found the endpoint mode of node-exporter is vip.</p>
  
  <pre><code>&gt; docker inspect 242pn4obqsly
  ...
  "Endpoint": {
  "Spec": {
      "Mode": "vip"
  },
  "VirtualIPs": [
      {
          "NetworkID": "61fn8hmgwg0n7rhg49ju2fdld",
          "Addr": "10.0.0.3/24"
      }
  ]
  ...
  </code></pre>
  
  <p>This means when contact with dns, prometheus can only get a single delegate service ip. Then the inner lbs strategy will route the income request to different backend instances. </p>
  
  <p>Then how does the related works succeeded? </p>
  
  <p>Thx!</p>