<p>AFAIK ceph have 2 specific trafic path:</p>
<ol>
<li>Traffic between client and ceph nodes,</li>
<li>Traffic between ceph nodes (Inter Ceph-node).</li>
</ol>
<p>So, let say my network is like this</p>
<p><a href="https://i.stack.imgur.com/wvDb7.jpg" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/wvDb7.jpg" alt="enter image description here" /></a></p>
<p>Note :</p>
<ol>
<li>Kube-node-4 is a kubernet worker that do not take part as rook node. Just a ceph-client</li>
<li>Red, Green and blue line is a seperate ethernet network.</li>
</ol>
<p>Can I do trafic seperation like this using Rook?
Is there any documentation on how to do it?</p>
<p>Sincerely</p>
<p>-bino-</p>

Bino Oetomo
<p>Check out the <a href="https://docs.ceph.com/en/latest/rados/configuration/network-config-ref/" rel="nofollow noreferrer">ceph docs</a>, what you describe is the separation of public and cluster networks. Cluster network is used for OSD &lt;--&gt; OSD traffic only (replication of PGs) while the public network is for Ceph clients as well as the other Ceph daemons (MON, MGR, etc). I'm not familiar with rook but according to the <a href="https://www.rook.io/docs/rook/v1.9/Storage-Configuration/Advanced/ceph-configuration/#osd-dedicated-network" rel="nofollow noreferrer">guide</a> you have to override the config, to get the current config map run:</p>
<pre><code>kubectl -n rook-ceph get ConfigMap rook-config-override -o yaml
</code></pre>
<blockquote>
<p>Enable the hostNetwork setting in the <a href="https://www.rook.io/docs/rook/v1.9/CRDs/Cluster/ceph-cluster-crd/#samples" rel="nofollow noreferrer">Ceph Cluster CRD configuration</a>.
For example,</p>
</blockquote>
<pre><code>  network:
    provider: host
</code></pre>
<p>and then</p>
<blockquote>
<p>Define the subnets to use for public and private OSD networks. Edit
the rook-config-override configmap to define the custom network
configuration:</p>
</blockquote>
<pre><code>kubectl -n rook-ceph edit configmap rook-config-override
</code></pre>
<blockquote>
<p>In the editor, add a custom configuration to instruct ceph which
subnet is the public network and which subnet is the private network.
For example:</p>
</blockquote>
<pre><code>apiVersion: v1
data:
  config: |
    [global]
    public network =  10.0.7.0/24
    cluster network = 10.0.10.0/24
    public addr = &quot;&quot;
    cluster addr = &quot;&quot;
</code></pre>
<blockquote>
<p>After applying the updated rook-config-override configmap, it will be
necessary to restart the OSDs by deleting the OSD pods in order to
apply the change. Restart the OSD pods by deleting them, one at a
time, and running ceph -s between each restart to ensure the cluster
goes back to &quot;active/clean&quot; state.</p>
</blockquote>
