title: Benchmarking Kafka - mediocre performance
body: <p>I'm benchmarking Kafka 0.8.1.1 by streaming 1k size messages on EC2 servers. </p>
  
  <p>I installed zookeeper on two m3.xlarge servers and have the following configuration:</p>
  
  <pre><code>dataDir=/var/zookeeper/                                                                    
  clientPort=2181
  initLimit=5
  syncLimit=2
  server.server1=zoo1:2888:3888                                
  server.server2=zoo2:2888:3888 
  </code></pre>
  
  <p>Second I installed Single Kafka Server on i2.2xlarge machine with 32Gb RAM and additional 6 SSD Drives where each disk partitioned as <code>/mnt/a , mnt/b, etc....</code>. On server I have one broker, single topic on port 9092 and 8 partitions with replication factor 1:</p>
  
  <pre><code>broker.id=1
  port=9092
  num.network.threads=4
  num.io.threads=8
  socket.send.buffer.bytes=1048576
  socket.receive.buffer.bytes=1048576
  socket.request.max.bytes=104857600
  log.dirs=/mnt/a/dfs-data/kafka-logs,/mnt/b/dfs-data/kafka-logs,/mnt/c/dfs-data/kafka-logs,/mnt/d/dfs-data/kafka-logs,/mnt/e/dfs-data/kafka-logs,/mnt/f/dfs-data/kafka-logs
  num.partitions=8
  log.retention.hours=168
  log.segment.bytes=536870912
  log.cleanup.interval.mins=1
  zookeeper.connect=172.31.26.252:2181,172.31.26.253:2181
  zookeeper.connection.timeout.ms=1000000
  kafka.metrics.polling.interval.secs=5
  kafka.metrics.reporters=kafka.metrics.KafkaCSVMetricsReporter
  kafka.csv.metrics.dir=/tmp/kafka_metrics
  kafka.csv.metrics.reporter.enabled=false
  replica.lag.max.messages=10000000
  </code></pre>
  
  <p>All my tests are done from another instance and latency between instances is less than 1 ms.
  I wrote producer/consumer java client using one thread producer and 8 threads consumer, when partition key is a random number from 0 till 7. 
  I serialized each message using Json by providing custom encoder.</p>
  
  <p>My consumer producer properties are the following:</p>
  
  <pre><code>metadata.broker.list = 172.31.47.136:9092
  topic = mytopic
  group.id = mytestgroup
  zookeeper.connect = 172.31.26.252:2181,172.31.26.253:2181
  serializer.class = com.vanilla.kafka.JsonEncoder
  key.serializer.class = kafka.serializer.StringEncoder
  producer.type=async
  queue.enqueue.timeout.ms = -1
  batch.num.messages=200
  compression.codec=0
  zookeeper.session.timeout.ms=400
  zookeeper.sync.time.ms=200
  auto.commit.interval.ms=1000
  number.messages = 100000
  </code></pre>
  
  <p>Now when I'm sending 100k messages, I'm getting 10k messages per second capacity and about 1 ms latency. </p>
  
  <p>that means that I have 10 Megabyte per second which equals to 80Mb/s, this is not bad, but I would expect better performance from those instances located in the same zone.</p>
  
  <p>Am I missing something in configuration?   </p>