<p>Trying to figure out how to expose multiple TCP/UDP services using a single LoadBalancer on Kubernetes. Let's say the services are ftpsrv1.com and ftpsrv2.com each serving at port 21. </p>

<p>Here are the options that I can think of and their limitations :</p>

<ul>
<li>One LB per svc: too expensive.</li>
<li>Nodeport :  Want to use a port outside the 30000-32767 range.</li>
<li>K8s Ingress : does not support TCP or UDP services as of now.</li>
<li>Using Nginx Ingress controller : which again <a href="https://kubernetes.github.io/ingress-nginx/user-guide/exposing-tcp-udp-services/" rel="noreferrer">will  be one on one mapping</a>: </li>
<li>Found <a href="https://github.com/DevFactory/smartnat" rel="noreferrer">this custom implementation</a> : But it doesn't seem to updated, last update was almost an year ago.</li>
</ul>

<p>Any inputs will be greatly appreciated.</p>

<p>It's actually <a href="https://kubernetes.github.io/ingress-nginx/user-guide/exposing-tcp-udp-services/" rel="noreferrer">possible</a> to do it using NGINX Ingress.</p>
<p>Ingress does not support TCP or UDP services. For this reason this Ingress controller uses the flags <code>--tcp-services-configmap</code> and <code>--udp-services-configmap</code> to point to an existing config map where the key is the external port to use and the value indicates the service to expose using the format: <code>&lt;namespace/service name&gt;:&lt;service port&gt;:[PROXY]:[PROXY]</code>.</p>
<p><a href="https://minikube.sigs.k8s.io/docs/tutorials/nginx_tcp_udp_ingress/" rel="noreferrer">This guide</a> is describing how it can be achieved using minikube but doing this on a on-premises kubernetes is different and requires a few more steps.</p>
<p>There is lack of documentation describing how it can be done on a non-minikube system and that's why I decided to go through all the steps here. This guide assumes you have a fresh cluster with no NGINX Ingress installed.</p>
<p>I'm using a GKE cluster and all commands are running from my Linux Workstation. It can be done on a Bare Metal K8S Cluster also.</p>
<p><strong>Create sample application and service</strong></p>
<p>Here we are going to create and application and it's service to expose it later using our ingress.</p>
<pre class="lang-yaml prettyprint-override"><code>apiVersion: apps/v1
kind: Deployment
metadata:
  name: redis-deployment
  namespace: default
  labels:
    app: redis
spec:
  replicas: 1
  selector:
    matchLabels:
      app: redis
  template:
    metadata:
      labels:
        app: redis
    spec:
      containers:
      - image: redis
        imagePullPolicy: Always
        name: redis
        ports:
        - containerPort: 6379
          protocol: TCP
---
apiVersion: v1
kind: Service
metadata:
  name: redis-service
  namespace: default
spec:
  selector:
    app: redis
  type: ClusterIP
  ports:
    - name: tcp-port
      port: 6379
      targetPort: 6379
      protocol: TCP
---      
apiVersion: v1
kind: Service
metadata:
  name: redis-service2
  namespace: default
spec:
  selector:
    app: redis
  type: ClusterIP
  ports:
    - name: tcp-port
      port: 6380
      targetPort: 6379
      protocol: TCP      
</code></pre>
<p>Notice that we are creating 2 different services for the same application. This is only to work as a proof of concept. I wan't to show latter that many ports can be mapped using only one Ingress.</p>
<p><strong>Installing NGINX Ingress using Helm:</strong></p>
<p>Install helm 3:</p>
<pre><code>$ curl https://raw.githubusercontent.com/helm/helm/master/scripts/get-helm-3 | bash
</code></pre>
<p>Add NGINX Ingress repo:</p>
<pre><code>$ helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx
</code></pre>
<p>Install NGINX Ingress on kube-system namespace:</p>
<pre><code>$ helm install -n kube-system ingress-nginx ingress-nginx/ingress-nginx
</code></pre>
<p><strong>Preparing our new NGINX Ingress Controller Deployment</strong></p>
<p>We have to add the following lines under spec.template.spec.containers.args:</p>
<pre><code>        - --tcp-services-configmap=$(POD_NAMESPACE)/tcp-services
        - --udp-services-configmap=$(POD_NAMESPACE)/udp-services
</code></pre>
<p>So we have to edit using the following command:</p>
<pre><code>$ kubectl edit deployments -n kube-system ingress-nginx-controller
</code></pre>
<p>And make it look like this:</p>
<pre><code>...
    spec:
      containers:
      - args:
        - /nginx-ingress-controller
        - --publish-service=kube-system/ingress-nginx-controller
        - --election-id=ingress-controller-leader
        - --ingress-class=nginx
        - --configmap=kube-system/ingress-nginx-controller
        - --tcp-services-configmap=$(POD_NAMESPACE)/tcp-services
        - --udp-services-configmap=$(POD_NAMESPACE)/udp-services
        - --validating-webhook=:8443
        - --validating-webhook-certificate=/usr/local/certificates/cert
        - --validating-webhook-key=/usr/local/certificates/key
...
</code></pre>
<p><strong>Create tcp/udp services Config Maps</strong></p>
<pre class="lang-yaml prettyprint-override"><code>apiVersion: v1
kind: ConfigMap
metadata:
  name: tcp-services
  namespace: kube-system
</code></pre>
<pre class="lang-yaml prettyprint-override"><code>apiVersion: v1
kind: ConfigMap
metadata:
  name: udp-services
  namespace: kube-system
</code></pre>
<p>Since these configmaps are centralized and may contain configurations, it is best if we only patch them rather than completely overwrite them every time you add a service:</p>
<pre class="lang-sh prettyprint-override"><code>$ kubectl patch configmap tcp-services -n kube-system --patch '{&quot;data&quot;:{&quot;6379&quot;:&quot;default/redis-service:6379&quot;}}'
</code></pre>
<pre class="lang-sh prettyprint-override"><code>$ kubectl patch configmap tcp-services -n kube-system --patch '{&quot;data&quot;:{&quot;6380&quot;:&quot;default/redis-service2:6380&quot;}}'
</code></pre>
<p>Where:</p>
<ul>
<li><code>6379</code>  : the port your service should listen to from outside the minikube virtual machine</li>
<li><code>default</code>  : the namespace that your service is installed in</li>
<li><code>redis-service</code>  : the name of the service</li>
</ul>
<p>We can verify that our resource was patched with the following command:</p>
<pre class="lang-sh prettyprint-override"><code>$ kubectl get configmap tcp-services -n kube-system -o yaml

apiVersion: v1
data:
  &quot;6379&quot;: default/redis-service:6379
  &quot;6380&quot;: default/redis-service2:6380
kind: ConfigMap
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {&quot;apiVersion&quot;:&quot;v1&quot;,&quot;kind&quot;:&quot;ConfigMap&quot;,&quot;metadata&quot;:{&quot;annotations&quot;:{},&quot;name&quot;:&quot;tcp-services&quot;,&quot;namespace&quot;:&quot;kube-system&quot;}}
  creationTimestamp: &quot;2020-04-27T14:40:41Z&quot;
  name: tcp-services
  namespace: kube-system
  resourceVersion: &quot;7437&quot;
  selfLink: /api/v1/namespaces/kube-system/configmaps/tcp-services
  uid: 11b01605-8895-11ea-b40b-42010a9a0050
</code></pre>
<p>The only value you need to validate is that there is a value under the  <code>data</code>  property that looks like this:</p>
<pre class="lang-yaml prettyprint-override"><code>  &quot;6379&quot;: default/redis-service:6379
  &quot;6380&quot;: default/redis-service2:6380
</code></pre>
<p><strong>Add ports to NGINX Ingress Controller Deployment</strong></p>
<p>We need to patch our nginx ingress controller so that it is listening on ports 6379/6380 and can route traffic to your service.</p>
<pre><code>spec:
  template:
    spec:
      containers:
      - name: controller
        ports:
         - containerPort: 6379
           hostPort: 6379
         - containerPort: 6380
           hostPort: 6380 
</code></pre>
<p>Create a file called  <code>nginx-ingress-controller-patch.yaml</code>  and paste the contents above.</p>
<p>Next apply the changes with the following command:</p>
<pre class="lang-sh prettyprint-override"><code>$ kubectl patch deployment ingress-nginx-controller -n kube-system --patch &quot;$(cat nginx-ingress-controller-patch.yaml)&quot;
</code></pre>
<p><strong>Add ports to NGINX Ingress Controller Service</strong></p>
<p>Differently from the solution presented for minikube, we have to patch our NGINX Ingress Controller Service as it is the responsible for exposing these ports.</p>
<pre><code>spec:
  ports:
  - nodePort: 31100
    port: 6379
    name: redis
  - nodePort: 31101
    port: 6380
    name: redis2
</code></pre>
<p>Create a file called  <code>nginx-ingress-svc-controller-patch.yaml</code>  and paste the contents above.</p>
<p>Next apply the changes with the following command:</p>
<pre class="lang-sh prettyprint-override"><code>$ kubectl patch service ingress-nginx-controller -n kube-system --patch &quot;$(cat nginx-ingress-svc-controller-patch.yaml)&quot;
</code></pre>
<p><strong>Check our service</strong></p>
<pre><code>$ kubectl get service -n kube-system ingress-nginx-controller
NAME                       TYPE           CLUSTER-IP      EXTERNAL-IP    PORT(S)                                                    AGE
ingress-nginx-controller   LoadBalancer   10.15.251.203   34.89.108.48   6379:31100/TCP,6380:31101/TCP,80:30752/TCP,443:30268/TCP   38m
</code></pre>
<p>Notice that our <code>ingress-nginx-controller</code> is listening to ports 6379/6380.</p>
<p>Test that you can reach your service with telnet via the following command:</p>
<pre><code>$ telnet 34.89.108.48 6379
</code></pre>
<p>You should see the following output:</p>
<pre><code>Trying 34.89.108.48...
Connected to 34.89.108.48.
Escape character is '^]'.
</code></pre>
<p>To exit telnet enter the  <code>Ctrl</code>  key and  <code>]</code>  at the same time. Then type  <code>quit</code>  and press enter.</p>
<p>We can also test port 6380:</p>
<pre><code>$ telnet 34.89.108.48 6380
Trying 34.89.108.48...
Connected to 34.89.108.48.
Escape character is '^]'.
</code></pre>
<p>If you were not able to connect please review your steps above.</p>
<p><strong>Related articles</strong></p>
<ul>
<li><a href="https://kubernetes.io/docs/Handbook/access-application-cluster/ingress-minikube/" rel="noreferrer">Routing traffic multiple services on ports 80 and 443 in minikube with the Kubernetes Ingress resource</a></li>
<li><a href="https://kubernetes.io/docs/Handbook/access-application-cluster/port-forward-access-application-cluster/" rel="noreferrer">Use port forwarding to access applications in a cluster</a></li>
</ul>
