<p>What would be the best setup to run <code>sonatype\nexus3</code> in Kubernetes that allows using the Docker repositories? </p>

<p>Currently I have a basic setup:</p>

<ul>
<li>Deployment of <code>sonatype\nexus3</code></li>
<li>Internal service exposing port 80 and 5000</li>
<li>Ingress + kube-lego provides HTTPS access to the Nexus UI</li>
</ul>

<p>How do I get around the limitation of ingress that doesn't allow more than one port?</p>

Niel de Wet
<h1>tl;dr</h1>

<p>Nexus needs to be served over SSL, otherwise docker won't connect to it. This can be achieved with a k8s ingress + <a href="https://github.com/jetstack/kube-lego/" rel="noreferrer">kube-lego</a> for a <a href="https://letsencrypt.org/" rel="noreferrer">Let's Encrypt</a> certificate. Any other real certificate will work as well. However, in order to serve both the nexus UI and the docker registry through one ingress (thus, one port) one needs a reverse proxy behind the ingress to detect the docker user agent and forward the request to the registry.</p>

<pre><code>                                                                             --(IF user agent docker) --&gt; [nexus service]nexus:5000 --&gt; docker registry
                                                                             |
[nexus ingress]nexus.example.com:80/ --&gt; [proxy service]internal-proxy:80 --&gt;|
                                                                             |
                                                                             --(ELSE                ) --&gt; [nexus service]nexus:80   --&gt; nexus UI
</code></pre>

<hr>

<h2>Start nexus server</h2>

<p><em>nexus-deployment.yaml</em>
This makes use of an azureFile volume, but you can use any volume. Also, the secret is not shown, for obvious reasons.</p>

<pre><code>apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: nexus
  namespace: default

spec:
  replicas: 1
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app: nexus
    spec:
      containers:
        - name: nexus
          image: sonatype/nexus3:3.3.1
          imagePullPolicy: IfNotPresent
          ports:
            - containerPort: 8081
            - containerPort: 5000
          volumeMounts:
            - name: nexus-data
              mountPath: /nexus-data
          resources:
            requests:
              cpu: 440m
              memory: 3.3Gi
            limits:
              cpu: 440m
              memory: 3.3Gi
      volumes:
        - name: nexus-data
          azureFile:
            secretName: azure-file-storage-secret
            shareName: nexus-data
</code></pre>

<p>It is always a good idea to add health and readiness probes, so that kubernetes can detect when the app goes down. Hitting the <code>index.html</code> page doesn't always work very well, so I'm using the REST API instead. This requires adding the Authorization header for a user with the <code>nx-script-*-browse</code> permission. Obviously you'll have to first bring the system up without probes to set up the user, then update your deployment later.</p>

<pre><code>      readinessProbe:
        httpGet:
          path: /service/siesta/rest/v1/script
          port: 8081
          httpHeaders:
            - name: Authorization
              # The authorization token is simply the base64 encoding of the `healthprobe` user's credentials:
              # $ echo -n user:password | base64
              value: Basic dXNlcjpwYXNzd29yZA==
        initialDelaySeconds: 900
        timeoutSeconds: 60
      livenessProbe:
        httpGet:
          path: /service/siesta/rest/v1/script
          port: 8081
          httpHeaders:
            - name: Authorization
              value: Basic dXNlcjpwYXNzd29yZA==
        initialDelaySeconds: 900
        timeoutSeconds: 60
</code></pre>

<p>Because nexus can sometimes take a long time to start, I use a very generous initial delay and timeout.</p>

<p><em>nexus-service.yaml</em> Expose port 80 for the UI, and port 5000 for the registry. This must correspond to the port configured for the registry through the UI.</p>

<pre><code>apiVersion: v1
kind: Service
metadata:
  labels:
    app: nexus
  name: nexus
  namespace: default
  selfLink: /api/v1/namespaces/default/services/nexus

spec:
  ports:
  - name: http
    port: 80
    targetPort: 8081
  - name: docker
    port: 5000
    targetPort: 5000
  selector:
    app: nexus
  type: ClusterIP
</code></pre>

<h2>Start reverse proxy (nginx)</h2>

<p><em>proxy-configmap.yaml</em> The <em>nginx.conf</em> is added as ConfigMap data volume. This includes a rule for detecting the docker user agent. This relies on the kubernetes DNS to access the <code>nexus</code> service as upstream.</p>

<pre><code>apiVersion: v1
data:
  nginx.conf: |
    worker_processes auto;

    events {
        worker_connections 1024;
    }

    http {
        error_log /var/log/nginx/error.log warn;
        access_log  /dev/null;
        proxy_intercept_errors off;
        proxy_send_timeout 120;
        proxy_read_timeout 300;

        upstream nexus {
            server nexus:80;
        }

        upstream registry {
            server nexus:5000;
        }

        server {
            listen 80;
            server_name nexus.example.com;

            keepalive_timeout  5 5;
            proxy_buffering    off;

            # allow large uploads
            client_max_body_size 1G;

            location / {
            # redirect to docker registry
            if ($http_user_agent ~ docker ) {
                proxy_pass http://registry;
            }
            proxy_pass http://nexus;
            proxy_set_header Host $host;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header X-Forwarded-Proto "https";
            }
        }
    }
kind: ConfigMap
metadata:
  creationTimestamp: null
  name: internal-proxy-conf
  namespace: default
  selfLink: /api/v1/namespaces/default/configmaps/internal-proxy-conf
</code></pre>

<p><em>proxy-deployment.yaml</em> </p>

<pre><code>apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: internal-proxy
  namespace: default

spec:
  replicas: 1
  template:
    metadata:
      labels:
        proxy: internal
    spec:
      containers:
        - name: nginx
          image: nginx:1.11-alpine
          imagePullPolicy: IfNotPresent
          lifecycle:
            preStop:
              exec:
                command: ["/usr/sbin/nginx","-s","quit"]
          volumeMounts:
            - name: internal-proxy-conf
              mountPath: /etc/nginx/
          env:
            # This is a workaround to easily force a restart by incrementing the value (numbers must be quoted)
            # NGINX needs to be restarted for configuration changes, especially DNS changes, to be detected
            - name: RESTART_
              value: "0"
      volumes:
        - name: internal-proxy-conf
          configMap:
            name: internal-proxy-conf
            items:
              - key: nginx.conf
                path: nginx.conf
</code></pre>

<p><em>proxy-service.yaml</em> The proxy is deliberately of type <code>ClusterIP</code> because the ingress will forward traffic to it. Port 443 is not used in this example.</p>

<pre><code>kind: Service
apiVersion: v1
metadata:
  name: internal-proxy
  namespace: default

spec:
  selector:
    proxy: internal
  ports:
    - name: http
      port: 80
      targetPort: 80
    - name: https
      port: 443
      targetPort: 443
  type: ClusterIP
</code></pre>

<h2>Create Ingress</h2>

<p><em>nexus-ingress.yaml</em> This step assumes you have an nginx ingress controller. If you have a certificate you don't need an ingress and can instead expose the proxy service, but you won't have the automation benefits of kube-lego.</p>

<pre><code>apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: nexus
  namespace: default
  annotations:
    kubernetes.io/ingress.class: "nginx"
    kubernetes.io/tls-acme: "true"

spec:
  tls:
    - hosts:
      - nexus.example.com
      secretName: nexus-tls
  rules:
    - host: nexus.example.com
      http:
        paths:
        - path: /
          backend:
            serviceName: internal-proxy
            servicePort: 80
</code></pre>
