<p>Our company is developing an application which runs in 3 seperate kubernetes-clusters in different versions (production, staging, testing).
We need to monitor our clusters and the applications over time (metrics and logs). We also need to run a mailserver.</p>

<p>So basically we have 3 different environments with different versions of our application. And we have some shared services that <em>just</em> need to run and we do not care much about them:</p>

<ul>
<li>Monitoring: We need to install influxdb and grafana. In every cluster there's a pre-installed heapster, that needs to send data to our tools.</li>
<li>Logging: We didn't decide yet.</li>
<li>Mailserver (<a href="https://github.com/tomav/docker-mailserver" rel="nofollow noreferrer">https://github.com/tomav/docker-mailserver</a>)</li>
<li>independant services: Sentry, Gitlab</li>
</ul>

<p>I am not sure where to run these external shared services. I found these options:</p>

<h2>1. Inside each cluster</h2>

<p>We need to install the tools 3 times for the 3 environments. </p>

<p>Con: </p>

<ul>
<li>We don't have <em>one</em> central point to analyze our systems.</li>
<li>If the whole cluster is down, we cannot look at anything.</li>
<li>Installing the same tools multiple times does not feel right.</li>
</ul>

<h2>2. Create an additional cluster</h2>

<p>We install the shared tools in an additional kubernetes-cluster.</p>

<p>Con:</p>

<ul>
<li>Cost for an additional cluster</li>
<li>It's probably harder to send ongoing data to external cluster (networking, security, firewall etc.).</li>
</ul>

<h2>3) Use an additional root-server</h2>

<p>We run docker-containers on an oldschool-root-server.</p>

<p>Con:</p>

<ul>
<li>Feels contradictory to use root-server instead of cutting-edge-k8s.</li>
<li>Single point of failure.</li>
<li>We need to control the docker-containers manually (or attach the machine to rancher).</li>
</ul>

<hr>

<p>I tried to google for the problem but I cannot find anything about the topic. Can anyone give me a hint or some links on this topic?
Or is it just no relevant problem that a cluster might go down?</p>

<p>To me, the second option sound less evil but I cannot estimate yet if it's hard to transfer data from one cluster to another.</p>

<p>The important questions are:</p>

<ul>
<li>Is it a problem to have monitoring-data in a cluster because one cannot see the monitoring-data if the cluster is offline?</li>
<li>Is it common practice to have an additional cluster for shared services that should not have an impact on other parts of the application?</li>
<li>Is it (easily) possible to send metrics and logs from one kubernetes-cluster to another (we are running kubernetes in <code>OpenTelekomCloud</code> which is basically <code>OpenStack</code>)?</li>
</ul>

<p>Thanks for your hints,</p>

<p>Marius</p>

Marius
<p>That is a very complex and philosophic topic, but I will give you my view on it and some facts to support it.</p>

<p>I think the best way is the second one - Create an additional cluster, and that's why:</p>

<ol>
<li><p>You need a point which should be accessible from any of your environments. With a separate cluster, you can set the same firewall rules, routes, etc. in all your environments and it doesn't affect your current workload.</p></li>
<li><p>Yes, you need to pay a bit more. However, you need resources to run your shared applications, and overhead for a Kubernetes infrastructure is not high in comparison with applications.</p></li>
<li><p>With a separate cluster, you can setup a real HA solution, which you might not need for staging and development clusters, so you will not pay for that multiple times.</p></li>
<li><p>Technically, it is also OK. You can use Heapster to collect data from multiple clusters; almost any logging solution can also work with multiple clusters. All other applications can be just run on the separate cluster, and that's all you need to do with them.</p></li>
</ol>

<p>Now, about your questions:</p>

<blockquote>
  <p>Is it a problem to have monitoring-data in a cluster because one cannot see the monitoring-data if the cluster is offline?</p>
</blockquote>

<p>No, it is not a problem with a separate cluster.</p>

<blockquote>
  <p>Is it common practice to have an additional cluster for shared services that should not have an impact on other parts of the application?</p>
</blockquote>

<p>I think, yes. At least I did it several times, and I know some other projects with similar architecture.</p>

<blockquote>
  <p>Is it (easily) possible to send metrics and logs from one kubernetes-cluster to another (we are running kubernetes in OpenTelekomCloud which is basically OpenStack)?</p>
</blockquote>

<p>Yes, nothing complex there. Usually, it does not depend on the platform.</p>
