title: 502 Bad gateway when communicating between docker containers in the same network
body: <p>I am trying to setup grafana with prometheus, cadvisor and alarmmanager. I'm trying to do this via docker-compose. I have got everything working except for grafana to commmunicate with prometheus as a datasource. First my setup via <code>docker-compose.yml</code></p>
  <pre><code>version: '3.7'
  
  volumes:
  prometheus_data: {}
  grafana_data: {}
  
  networks:
  monitor-net:
  
  services:
  prometheus:
    image: prom/prometheus:v2.13.1
    volumes:
      - ./prometheus/:/etc/prometheus/
      - prometheus_data:/prometheus
    command:
      - '--config.file=/etc/prometheus/prometheus.yml'
      - '--storage.tsdb.path=/prometheus'
      - '--web.console.libraries=/usr/share/prometheus/console_libraries'
      - '--web.console.templates=/usr/share/prometheus/consoles'
    ports:
      - 9090:9090
    expose:
      - 9090
    networks:
      - monitor-net
    links:
      - cadvisor:cadvisor
      - node-exporter:node-exporter
    restart: always
  
  node-exporter:
    image: prom/node-exporter
    volumes:
      - /proc:/host/proc:ro
      - /sys:/host/sys:ro
      - /:/rootfs:ro
    command:
      - '--path.procfs=/host/proc'
      - '--path.sysfs=/host/sys'
      - --collector.filesystem.ignored-mount-points
      - &quot;^/(sys|proc|dev|host|etc|rootfs/var/lib/docker/containers|rootfs/var/lib/docker/overlay2|rootfs/run/docker/net&gt;    ports:
      - 9100:9100
    expose:
      - 9100
    networks:
      - monitor-net
    restart: always
  
  alertmanager:
    image: prom/alertmanager
    ports:
      - 9093:9093
    expose:
      - 9093
    volumes:
      - ./alertmanager/:/etc/alertmanager/
    networks:
      - monitor-net
    restart: always
    command:
      - '--config.file=/etc/alertmanager/config.yml'
      - '--storage.path=/alertmanager'
  
  cadvisor:
    image: google/cadvisor
    volumes:
      - /:/rootfs:ro
      - /var/run:/var/run:rw
      - /sys:/sys:ro
      - /var/lib/docker/:/var/lib/docker:ro
    ports:
      - 8080:8080
    expose:
      - 8080
    networks:
      - monitor-net
    restart: always
  
  grafana:
    image: grafana/grafana:6.7.2
    user: &quot;472&quot;
    ports:
      - 3000:3000
    expose:
      - 3000
    volumes:
      - grafana_data:/var/lib/grafana
      - ./grafana/provisioning/:/etc/grafana/provisioning/
    env_file:
      - ./grafana/config.monitoring
    networks:
      - monitor-net
    links:
      - prometheus:prometheus
    restart: always
  </code></pre>
  <p>I can access al services individually in the browser via a domain name coupled to my VPS. I can also see in portainer they really are all 6 in the same network. But when it try to access http://prometheus:9090 from my grafana application:</p>
  <p><a href="https://i.stack.imgur.com/XWOeT.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/XWOeT.png" alt="prometheus" /></a></p>
  <p>i get the following error:</p>
  <pre><code>t=2021-03-21T19:24:09+0000 lvl=eror msg=&quot;Data proxy error&quot; logger=data-proxy-log userId=1 orgId=1 uname=admin path=/api/datasources/proxy/21/api/v1/query remote_addr=172.26.0.1 referer=http://&lt;my-remote-host&gt;:3000/datasources/edit/21/ error=&quot;http: proxy error: dial tcp 172.26.0.5:9090: i/o timeout&quot;
  
  
  t=2021-03-21T19:24:09+0000 lvl=info msg=&quot;Request Completed&quot; logger=context userId=1 orgId=1 uname=admin method=GET path=/api/datasources/proxy/21/api/v1/query status=502 remote_addr=172.26.0.1 time_ms=30061 size=0 referer=http://&lt;my-remote-host&gt;:3000/datasources/edit/21/
  </code></pre>
  <p>Now i have successfully done similar setups of containers communicating with each other but i can't get this to work.. I can't open up the port in UFW(running ubuntu server) for obvious security reasons, but that shouldn't make a difference. I also got iptables disabled in the <code>docker-daemon.json</code></p>
  <p>I hope you guys might have a suggestion!</p>