title: Kafka SASL/PLAIN Setup with SSL
body: <p>I'm trying to configure SASL/PLAIN with SSL in our Kafka environment. The SSL part is done, but I'm running into the below error when starting the Zookeeper.<br>
  Has anyone configured SASL/PLAIN with SSL for Zookeeper and broker ?</p>
  
  <p><strong>Server-Broker Config</strong></p>
  
  <pre><code>broker.id=19
  port=19001
  
  #host.name=localhost
  host.name=jcia9081
  
  #advertised.host.name=11.72.29.153
  
  num.network.threads=3
  num.io.threads=8
  
  socket.send.buffer.bytes=102400
  socket.receive.buffer.bytes=102400
  socket.request.max.bytes=104857600
  
  log.dirs=/appsdata/osp/kafka/datastore1/AIN_TEST_SSL/kafka,/appsdata/osp/kafka/datastore2/AIN_TEST_SSL/kafka,/appsdata/osp/kafka/datastore3/AIN_TEST_SSL/kafka
  
  num.partitions=1
  num.recovery.threads.per.data.dir=1
  
  log.retention.hours=96
  #log.retention.bytes=1073741824
  
  log.segment.bytes=1073741824
  log.retention.check.interval.ms=300000
  
  log.cleaner.enable=false
  
  replica.fetch.max.bytes=52428800
  
  #The maximum size of a message that the server can receive.
  message.max.bytes=52428800
  
  zookeeper.connect=jcia9081:19101,jcia9131:19101,jcia9434:19101
  zookeeper.connection.timeout.ms=6000
  #################################
  auto.create.topics.enable=false
  auto.leader.rebalance.enable=true
  delete.topic.enable=true
  controlled.shutdown.enable=true
  
  ### SSL properties
  security.inter.broker.protocol=SASL_SSL
  listeners=SSL://jcia9081:19001,SASL_SSL://jcia9081:19002
  ssl.keystore.location=/home/kafka/certs/jcia9081.federated.fds.jks
  ssl.keystore.password=xxxx
  ssl.key.password=xxxx
  ssl.truststore.location=/home/kafka/certs/test.truststore.jks
  ssl.truststore.password=xxx
  sasl.mechanism.inter.broker.protocol=PLAIN
  sasl.enabled.mechanisms=PLAIN
  #listeners=SSL://jcia9081:19001
  allow.everyone.if.no.acl.found=true
  </code></pre>
  
  <p><strong>Zookeeper Config</strong></p>
  
  <pre><code>dataDir=/appsdata/osp/kafka/datastore1/AIN_TEST_SSL/zookeeper
  
  clientPort=19101
  
  maxClientCnxns=0
  
  tickTime=2000
  
  initLimit=60
  syncLimit=10
  
  server.19=jcia9081:19121:19131
  server.20=jcia9131:19121:19131
  server.21=jcia9434:19121:19131
  </code></pre>
  
  <p><strong>JAAS Zookeeper Configuration File</strong></p>
  
  <pre><code>Server {
     org.apache.kafka.common.security.plain.PlainLoginModule required
     username="admin"
     password="admin-secret"
     user_admin="admin-secret"
  };
  </code></pre>
  
  <p><strong>JAAS Broker Configuration File</strong></p>
  
  <pre><code>    KafkaServer {
          org.apache.kafka.common.security.plain.PlainLoginModule required
          username="admin"
          password="admin-secret"
          user_admin="admin-secret";
          };
  Client {
          org.apache.kafka.common.security.plain.PlainLoginModule required
          username="admin"
          password="admin-secret";
  };
  </code></pre>
  
  <p><strong>Startup Error</strong></p>
  
  <pre><code>[2017-04-26 10:05:24,775] INFO Client environment:user.dir=/appsdata/osp/kafka/AIN_TEST_SSL/startup (org.apache.zookeeper.ZooKeeper)
  [2017-04-26 10:05:24,776] INFO Initiating client connection, connectString=jcia9081:19101 sessionTimeout=6000 watcher=org.I0Itec.zkclient.ZkClient@43195e57 (org.apache.zookeeper.ZooKeeper)
  [2017-04-26 10:05:24,776] INFO Starting ZkClient event thread. (org.I0Itec.zkclient.ZkEventThread)
  [2017-04-26 10:05:24,778] DEBUG zookeeper.disableAutoWatchReset is false (org.apache.zookeeper.ClientCnxn)
  [2017-04-26 10:05:24,789] DEBUG JAAS loginContext is: Client (org.apache.zookeeper.client.ZooKeeperSaslClient)
  [2017-04-26 10:05:24,795] DEBUG Awaiting connection to Zookeeper server (org.I0Itec.zkclient.ZkClient)
  [2017-04-26 10:05:24,795] INFO Waiting for keeper state SaslAuthenticated (org.I0Itec.zkclient.ZkClient)
  [2017-04-26 10:05:24,881] INFO successfully logged in. (org.apache.zookeeper.Login)
  [2017-04-26 10:05:24,881] INFO Client will use DIGEST-MD5 as SASL mechanism. (org.apache.zookeeper.client.ZooKeeperSaslClient)
  [2017-04-26 10:05:24,895] INFO Opening socket connection to server jcia9081.federated.fds/11.168.24.82:19101. Will attempt to SASL-authenticate using Login Context section 'Client' (org.apache.zookeeper.ClientCnxn)
  [2017-04-26 10:05:24,968] INFO Socket connection established to jcia9081.federated.fds/11.168.24.82:19101, initiating session (org.apache.zookeeper.ClientCnxn)
  [2017-04-26 10:05:24,970] DEBUG Session establishment request sent on jcia9081.federated.fds/11.168.24.82:19101 (org.apache.zookeeper.ClientCnxn)
  [2017-04-26 10:05:25,001] INFO Session establishment complete on server jcia9081.federated.fds/11.168.24.82:19101, sessionid = 0x15baa8300580003, negotiated timeout = 6000 (org.apache.zookeeper.ClientCnxn)
  [2017-04-26 10:05:25,002] DEBUG ClientCnxn:sendSaslPacket:length=0 (org.apache.zookeeper.client.ZooKeeperSaslClient)
  [2017-04-26 10:05:25,004] DEBUG Received event: WatchedEvent state:SyncConnected type:None path:null (org.I0Itec.zkclient.ZkClient)
  [2017-04-26 10:05:25,004] INFO zookeeper state changed (SyncConnected) (org.I0Itec.zkclient.ZkClient)
  [2017-04-26 10:05:25,004] DEBUG Leaving process event (org.I0Itec.zkclient.ZkClient)
  [2017-04-26 10:05:25,006] ERROR SASL authentication failed using login context 'Client'. (org.apache.zookeeper.client.ZooKeeperSaslClient)
  [2017-04-26 10:05:25,006] DEBUG Received event: WatchedEvent state:AuthFailed type:None path:null (org.I0Itec.zkclient.ZkClient)
  [2017-04-26 10:05:25,006] INFO zookeeper state changed (AuthFailed) (org.I0Itec.zkclient.ZkClient)
  [2017-04-26 10:05:25,006] DEBUG Leaving process event (org.I0Itec.zkclient.ZkClient)
  [2017-04-26 10:05:25,006] DEBUG Closing ZkClient... (org.I0Itec.zkclient.ZkClient)
  [2017-04-26 10:05:25,006] INFO Terminate ZkClient event thread. (org.I0Itec.zkclient.ZkEventThread)
  [2017-04-26 10:05:25,006] DEBUG Closing ZooKeeper connected to jcia9081:19101 (org.I0Itec.zkclient.ZkConnection)
  [2017-04-26 10:05:25,006] DEBUG Close called on already closed client (org.apache.zookeeper.ZooKeeper)
  [2017-04-26 10:05:25,006] DEBUG Closing ZkClient...done (org.I0Itec.zkclient.ZkClient)
  [2017-04-26 10:05:25,007] FATAL Fatal error during KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer)
  org.I0Itec.zkclient.exception.ZkAuthFailedException: Authentication failure
          at org.I0Itec.zkclient.ZkClient.waitForKeeperState(ZkClient.java:946)
          at org.I0Itec.zkclient.ZkClient.waitUntilConnected(ZkClient.java:923)
          at org.I0Itec.zkclient.ZkClient.connect(ZkClient.java:1230)
          at org.I0Itec.zkclient.ZkClient.&lt;init&gt;(ZkClient.java:156)
          at org.I0Itec.zkclient.ZkClient.&lt;init&gt;(ZkClient.java:130)
          at kafka.utils.ZkUtils$.createZkClientAndConnection(ZkUtils.scala:76)
          at kafka.utils.ZkUtils$.apply(ZkUtils.scala:58)
          at kafka.server.KafkaServer.initZk(KafkaServer.scala:327)
          at kafka.server.KafkaServer.startup(KafkaServer.scala:200)
          at kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:39)
          at kafka.Kafka$.main(Kafka.scala:67)
          at kafka.Kafka.main(Kafka.scala)
  [2017-04-26 10:05:25,009] INFO shutting down (kafka.server.KafkaServer)
  </code></pre>
  
  <p><strong>Broker Start Command</strong></p>
  
  <pre><code>    . /appsdata/osp/kafka/AIN_TEST_SSL/startup/setenv.sh
  export JMX_PORT=19011
  $KAFKA_HOME/bin/kafka-server-start.sh $KAFKA_CONFIG/server_AIN_TEST_SSL.properties &gt; $LOG_DIR/kafka-broker-startup.log 2&gt;&amp;1 &amp;
  </code></pre>
  
  <p><strong>setenv</strong></p>
  
  <pre><code>    export APP_HOME=/apps/osp
  
  export JAVA_HOME=$APP_HOME/jdk1.8.0_101
  export JAVA=$JAVA_HOME/bin/java
  
  export KAFKA_HOME=$APP_HOME/kafka/v2.11-0.10.1.1
  export KAFKA_CONFIG=/appsdata/osp/kafka/AIN_TEST_SSL/config
  
  export ZK_HOME=$KAFKA_HOME
  export ZK_CONFIG=$KAFKA_CONFIG
  
  export LOG_DIR=/applogs/osp/kafka/AIN_TEST_SSL
  export KAFKA_LOG4J_OPTS="-Dlog4j.configuration=file:$KAFKA_CONFIG/log4j.properties"
  </code></pre>
  
  <p><strong>In Termninal 1</strong></p>
  
  <pre><code>echo $KAFKA_OPTS
  -Djava.security.auth.login.config=/appsdata/osp/kafka/AIN_TEST_SSL/config/jaas_AIN_TEST_SSL_zookeeper.conf
  </code></pre>
  
  <p><strong>In Terminal 2</strong></p>
  
  <pre><code>echo $KAFKA_OPTS
  -Djava.security.auth.login.config=/appsdata/osp/kafka/AIN_TEST_SSL/config/jaas_AIN_TEST_SSL_server.conf
  </code></pre>