title: Docker-ized Consul, Zookeeper and Kafka in Amazon-ECS
body: <p>I’m trying to deploy a dockerized Kafka (with Zookeeper) to AWS. I’ve combed through several resources, and got close. But when I ssh into my Kafka boxes in AWS, I can see that Kafka is exiting with -1. I can only assume that Kafka can’t reach Zookeeper, because I haven’t set up any name resolution. I’m looking at Consul as a solution for this (see <a href="http://docs.aws.amazon.com/AmazonECS/latest/developerguide/common_use_cases.html" rel="noreferrer">here</a>).</p>
  
  <p>This looks like a decent <a href="https://aws.amazon.com/blogs/compute/service-discovery-via-consul-with-amazon-ecs" rel="noreferrer">resource</a>. But I can’t even successfully run the official Consul <a href="https://hub.docker.com/_/consul" rel="noreferrer">docker image</a> (in either server or agent mode). I can get <a href="https://hub.docker.com/r/progrium/consul" rel="noreferrer">this one</a> to work (used <a href="https://github.com/camptocamp/docker-consul-demo" rel="noreferrer">here</a>). But I’d like to use the official image if possible. </p>
  
  <p>Has anyone gotten the official Consul image working with Kafka and Zookeeper? </p>
  
  <p><strong>A)</strong> To start, I think I’d like to get it working in docker-compose. This is my docker-compose.yml <a href="https://github.com/twashing/simple-kafka-onyx-commander/blob/f2cc68c16e461fc677a444b3e6642155fc8e2cba/docker-compose.yml#L4-L26" rel="noreferrer">file</a>, with me trying to have Kafka use Consul, to discover Zookeeper nodes. </p>
  
  <p>Below is the relevant code chunk, which you can run with <code>docker-compose up consul</code>. This doesn't show any errors. But <em>i)</em> I can't reach <em><a href="http://localhost:8500" rel="noreferrer">http://localhost:8500</a></em> . And <em>ii)</em> How do I then point Kafka to Zookeeper and Kafka clients to Kafka?</p>
  
  <pre><code>version: '2'
  services:
  
    consul:
      image: consul:0.8.3
      expose:
        - 8300
        - 8301
        - 8301/udp
        - 8302
        - 8302/udp
        - 8400
        - 8500
        - 8600
        - 8600/udp
      ports:
        - 8300:8300
        - 8301:8301
        - 8301/udp:8301/udp
        - 8302:8302
        - 8302/udp:8302/udp
        - 8400:8400
        - 8500:8500
        - 8600:8600
        - 8600/udp:8600/udp
      command: "agent -server -bind=0.0.0.0 -retry-join=0.0.0.0 -bootstrap-expect=1"
  </code></pre>
  
  <p><strong>B)</strong> Then I can try to use it in AWS (via <a href="https://github.com/twashing/simple-kafka-onyx-commander/blob/136176749f5a63c3adb9f2d47989fe6d5453d3d5/aws.tf" rel="noreferrer">Terraform</a>). Ideally, I’d like to have 10 Kafka nodes and 2 Zookeeper nodes. But that’s as a stretch goal. </p>
  
  <p><strong>EDIT</strong></p>
  
  <p>As @MattSchuchard pointed out, there is the option of Mesos, with Marathon and Chronos. </p>
  
  <p><strong>A)</strong> But I wanted to try out something that I thought would be more straightforward (i.e., Consul, Zookeeper, Kafka), before layering on another abstraction. Installing DCOS seems simplified with things like <a href="https://github.com/zutherb/terraform-dcos" rel="noreferrer">zutherb/terraform-dcos</a>. </p>
  
  <p><strong>B)</strong> But going that route would mean changing the deployment workflow from i) just deploying ECS instances, to ii) deploying a DCOS cluster to AWS, then deploying docker images to that. If I could know that it were easy to deploy Consul, Zookeeper, Kafka, etc, containers to Marathon, then that would be good. Far as I can tell though, I’d have to use the docs cli, losing the declarative property of Terraform for my actual apps and services. Because as it stands, there’s no official Terraform provider for interacting with Marathon (<a href="https://github.com/nicgrayson/terraform-provider-marathon" rel="noreferrer">nicgrayson/terraform-provider-marathon</a> notwithstanding). </p>
  
  <p><strong>C)</strong> So I’ll take a look at Mesosphere DCOS. But hopefully there’s just a small config error with what I have up now.</p>