title: Apache Kafka - Testing Fault Tolerance
body: <p><strong>Disclaimer</strong>: I am learning, and obviously I am doing something wrong. Any help is appreciated!  </p>
  
  <p><strong>Set up:</strong><br>
  I created an apache kafka cluster with 3 nodes using vagrant (each node has a separate zk and kafka process running). I want to test fault tolerance, so I created a replicated topic and when I 'describe' it, all 3 nodes are up:<br>
  <code>$KAFKA_HOME/bin/kafka-topics.sh --describe --zookeeper 192.168.33.21:2181 --topic replicated-topic-1
  Topic:replicated-topic-1        PartitionCount:1        ReplicationFactor:3     Configs:
          Topic: replicated-topic-1       Partition: 0    Leader: 1       Replicas: 2,3,1  Isr: 1,2,3</code></p>
  
  <p><strong>To test fault tolerance:</strong><br>
  On the leader (node 1 in this case) I want to kill the process that started kafka. So I find the pid on leader to kill it:<br>
  <code>ps -elf | grep server.properties
  4 S root      2975  2900  0  80   0 - 10738 ?      15:40 pts/0    00:00:00 sudo /usr/local/kafka/kafka_2.11-0.10.0.0//bin/kafka-server-start.sh /usr/local/kafka/kafka_2.11-0.10.0.0//config/server.properties
  0 S vagrant   3438  2900  0  80   0 -  2184 -      15:41 pts/0    00:00:00 grep server.properties</code>  </p>
  
  <p>I assume the pid to kill is 2975. I use the following command to kill it:<br>
  <code>sudo kill -9 2975
  [1]+  Killed                  sudo $KAFKA_HOME/bin/kafka-server-start.sh $KAFKA_HOME/config/server.properties</code></p>
  
  <p>All good so far. I assume the kafka process on the leader is killed. But the describe command says the leader is not down:  </p>
  
  <p><code>$KAFKA_HOME/bin/kafka-topics.sh --describe --zookeeper 192.168.33.21:2181 --topic replicated-topic-1
  Topic:replicated-topic-1        PartitionCount:1        ReplicationFactor:3     Configs:
          Topic: replicated-topic-1       Partition: 0    Leader: 1       Replicas: 2,3,1  Isr: 1,2,3</code>   </p>
  
  <p>Just to make sure I killed the kafka process, I try this command:
  <code>ps -elf | grep server.properties
  0 S vagrant   3654  2900  0  80   0 -  2183 -      15:45 pts/0    00:00:00 grep server.properties</code>  </p>
  
  <p>I guess the process did get killed, but all 3 nodes in the kafka cluster are still up. Just to try, I killed the other process that vagrant is running (pid: 3654):<br>
  <code>sudo kill -9 3657
  vagrant@debian-70rc1-x64-vbox4210:~$ ps -elf | grep server.properties
  0 S vagrant   3661  2900  0  80   0 -  2183 -      15:50 pts/0    00:00:00 grep server.properties</code><br>
  But apparently, it starts again with a different pid:<br>
  <code>ps -elf | grep server.properties
  0 S vagrant   3663  2900  0  80   0 -  2183 -      15:50 pts/0    00:00:00 grep server.properties</code>  </p>
  
  <p><strong>What am I missing. How can I kill the leader and test fault tolerance?</strong></p>