title: Load Balancing logs flowing more multiple services via logstash
body: <p>We have a ElasticSearch-Logstash-Kibana deployment in which multiple logstash-forwarders (from multiple service logs) pushes log to logstash which then sends it to Kafka and then one more logstash pulls those logs from Kafka and indexes them in ElasticSearch cluster.</p>
  
  <p>Can some one let me know:-</p>
  
  <ol>
  <li>What is the best way to handle traffic spikes in such deployment?</li>
  <li>Can we configure Kafka in such a way that a single
  logstash-forwarder (of a single service) does not hog the entire
  set-up? We want to ensure that all services can use the set-up
  fairly? </li>
  <li>How to ensure that one index does not grow huge compared to
  others? We have many daily indexes of several service logs and we
  want to ensure that during spike if one service index (or shards)
  become too large we want to block that service logs. Such that the
  cluster performance does not get effected?</li>
  </ol>