<p>We have a Rails app that we deploy in a Docker environment (currently ECS but moving to Kubernetes). We are adding Kafka to our architecture, using <a href="https://aws.amazon.com/msk/" rel="nofollow noreferrer">Amazon's managed Kafka service</a>. </p>

<p>What's the best way to add the Kafka consumer library to our Rails app? We are using <a href="https://github.com/appsignal/rdkafka-ruby" rel="nofollow noreferrer">rdkafka-ruby</a>, but should we:</p>

<ul>
<li>Add it to our Rails app container, or</li>
<li>Deploy it as a separate container?</li>
</ul>

<p>According to <a href="https://github.com/appsignal/rdkafka-ruby" rel="nofollow noreferrer">rdkafka</a> intro:</p>

<blockquote>
  <p>This gem only provides a high-level Kafka consumer</p>
</blockquote>

<p>As for service, it's recommended to use <code>kafka</code> service in separate instances. Since you are going to use <a href="https://aws.amazon.com/msk/" rel="nofollow noreferrer">Amazon MSK</a> as a <code>kafka</code> service, it seems ok to add the <code>rdkafka</code> gem into your app.</p>

<p>P.S. That's how Xing <a href="https://tech.xing.com/asynchronous-state-replication-9e504aeac0de" rel="nofollow noreferrer">used</a> <a href="https://github.com/appsignal/rdkafka-ruby" rel="nofollow noreferrer">rdkafka-ruby</a> for <a href="https://www.infoq.com/presentations/event-streams-kafka/" rel="nofollow noreferrer">migrate from transactions to streams</a></p>
