<p>Is it possible to have a centralized storage/volume that can be shared between two pods/instances of an application that exist in different worker nodes in Kubernetes?</p>

<p>So to explain my case:</p>

<ul>
<li><p>I have a Kubernetes cluster with 2 worker nodes. In each one of these I have 1 instance of app X running. This means I have 2 instances of app X running totally at the same time.</p></li>
<li><p>Both instances subscribe on the topic topicX, that has 2 partitions, and are part of a consumer group in Apache Kafka called groupX.</p></li>
</ul>

<p>As I understand it the message load will be split among the partitions, but also among the consumers in the consumer group. So far so good, right? </p>

<p>So to my problem:</p>

<ul>
<li><p>In my whole solution I have a hierarchy division with the unique constraint by country and ID. Each combination of country and ID has a pickle model (python Machine Learning Model), which is stored in a directory accessed by the application. For each combination of a country and ID I receive one message per minute.</p></li>
<li><p>At the moment I have 2 countries, so to be able to scale properly I wanted to split the load between two instances of app X, each one handling its own country. </p></li>
<li><p>The problem is that with Kafka the messages can be balanced between the different instances, and to access the pickle-files in each instance without know what country the message belongs to, I have to store the pickle-files in both instances.</p></li>
</ul>

<p>Is there a way to solve this? I would rather keep the setup as simple as possible so it is easy to scale and add a third, fourth and fifth country later.</p>

<p>Keep in mind that this is an overly simplified way of explaining the problem. The number of instances is much higher in reality etc.</p>

danielo
<p>Yes. It's possible if you look at <a href="https://kubernetes.io/docs/concepts/storage/persistent-volumes/#access-modes" rel="noreferrer">this table</a> any <code>PV</code> (Physical Volume) that supports <code>ReadWriteMany</code> will help you accomplish having the same data store for your Kafka workers. So in summary these:</p>

<ul>
<li>AzureFile</li>
<li>CephFS</li>
<li>Glusterfs</li>
<li>Quobyte</li>
<li>NFS</li>
<li>VsphereVolume - (works when pods are collocated)</li>
<li>PortworxVolume</li>
</ul>

<p>In my opinion, NFS is the easiest to implement. Note that Azurefile, Quobyte, and Portworx are paid solutions.</p>
