<p>I'm working on a SaaS app that will be running in Kubernetes. We're using a Helm chart that deploys all the components into the cluster (for simplicity's sake let's assume it's a frontend service, a backend and a database). App architecture is multi-tenant (we have a single instance of each service that are being shared by all tenants) <strong>and we would like to keep it that way</strong>. What I'm currently struggling with and would like to ask for advice/best practice on is how does one go about automating the provisioning of custom sub-domains for the tenants?</p>

<p>Imagine the app is hosted at <code>exampleapp.com</code>.
A brand new customer comes and registers a new organisation <code>some-company</code>. At that moment, in addition to creating new tenant in the system, I would also like to provision a new subdomain <code>some-company.exampleapp.com</code>. I would like this provisioning to be done automatically and not require any manual intervention. </p>

<ul>
<li>What options do I have for implementing automated sub-domain provisioning in Kubernetes? </li>
<li>How does our (<code>exampleapp.com</code>) domain registrar/nameserver provider fit into the solution? 
Does it have to provide an API for dynamic DNS record creation/modification?</li>
</ul>

<p>I appreciate that the questions I'm asking are quite broad so I'm not expecting anything more than a high-level conceptual answer or pointers to some services/libraries/tools that might help me achieve this.</p>

IvanR
<p><strong>Note:</strong> Since this is more of a theoretical question, I'll give you some points from a Kubernetes Engineer, I divided your question in blocks to ease the understanding.</p>

<ul>
<li>About your multi-tenancy architecture:
Keeping "it that way" is achievable. It simplifies the Kubernetes structure, on the other hand it relies more on your app.</li>
</ul>

<p><strong>Question 1:</strong></p>

<blockquote>
  <p>Imagine the app is hosted at  <code>exampleapp.com</code>. A brand new customer comes and registers a new organisation  <code>some-company</code>. At that moment, in addition to creating new tenant in the system, I would also like to provision a new subdomain  <code>some-company.exampleapp.com</code>. I would like this provisioning to be done automatically and not require any manual intervention.</p>
</blockquote>

<p><strong>Suggestion:</strong></p>

<ul>
<li>For that, you will have to give your app admin privileges and the tools required for it to add Ingress Rules entries to your Ingress when a new client is added. A script using <code>kubectl patch</code> is the simpler solution from my viewpoint.</li>
</ul>

<p>For this approach I suggest installing the <a href="https://kubernetes.github.io/ingress-nginx/" rel="noreferrer">Nginx Ingress Controller</a> for it's versatility.</p>

<p><strong>Here is an Example:</strong></p>

<pre><code>apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: &lt;ingress-name&gt;
spec:
  rules:
  - host: client1.exampleapp.com
    http:
      paths:
      - path: /client1
        backend:
          serviceName: &lt;main-service&gt;
          servicePort: &lt;main-service-port&gt;

  - host: client2.exampleapp.com
    http:
      paths:
      - path: /client2
        backend:
          serviceName: &lt;main-service&gt;
          servicePort: &lt;main-service-port&gt;
</code></pre>

<ul>
<li>And here is the one-liner command using <code>kubectl patch</code> on how to add new rules:</li>
</ul>

<pre><code>kubectl patch ingress demo-ingress --type "json" -p '[{"op":"add","path":"/spec/rules/-","value":{"host":"client3.exampleapp.com","http":{"paths":[{"path":"/client3","backend":{"serviceName":"main-service","servicePort":80}}]}}}]'
</code></pre>

<p>POC:</p>

<pre><code>$ kubectl get ingress
NAME           HOSTS                                           ADDRESS         PORTS   AGE
demo-ingress   client1.exampleapp.com,client2.exampleapp.com   192.168.39.39   80      15m

$ kubectl patch ingress demo-ingress --type "json" -p '[{"op":"add","path":"/spec/rules/-","value":{"host":"client3.exampleapp.com","http":{"paths":[{"path":"/client3","backend":{"serviceName":"main-service","servicePort":80}}]}}}]'
ingress.extensions/demo-ingress patched

$ kubectl describe ingress demo-ingress
Rules:
  Host                    Path  Backends
  ----                    ----  --------
  client1.exampleapp.com  
                          /client1   main-service:80 (&lt;none&gt;)
  client2.exampleapp.com  
                          /client2   main-service:80 (&lt;none&gt;)
  client3.exampleapp.com  
                          /client3   main-service:80 (&lt;none&gt;)
</code></pre>

<p>This rule redirects the traffic incoming from the subdomains to subpaths inside your main app.</p>

<ul>
<li>Also, to add TLS handling, refer to: <a href="https://kubernetes.github.io/ingress-nginx/user-guide/tls/" rel="noreferrer">Nginx Ingress TLS Guide</a></li>
</ul>

<hr>

<p><strong>Question2 :</strong></p>

<blockquote>
  <p>How does our (<code>exampleapp.com</code>) domain registrar/nameserver provider fit into the solution? Does it have to provide an API for dynamic DNS record creation/modification?</p>
</blockquote>

<p><strong>Suggestion:</strong></p>

<ul>
<li>I believe you already has something similar, but you need a wildcard record in your DNS to point <code>*.example.app</code> to the IP of the ingress, I don't believe you need anything more than that, because it redirects all to the ingress and the ingress forwards it internally.</li>
</ul>

<p><strong>Question 3:</strong></p>

<blockquote>
  <p>If there are some strong arguments why multi-tenancy + Kubernetes don't go along very well, those are welcome as well.  </p>
</blockquote>

<p><strong>Opinion:</strong></p>

<ul>
<li>I  don't see any major reason why it would be a problem. You just have, once again, to adequate your app to handle scaling because I believe in the long run you would want your app to be able to scale to multi-pod structure to provide elastic availability.</li>
</ul>

<p>These are my 2 cents to your question, I hope it helps you!</p>
