<p>I have a database in AWS that I need to connect to from Kubernetes, but security settings in that database prevent this.  My solution is to SSH tunnel to a proxy from within the Kubernetes pod, and connect through that to the database in AWS.</p>
<p>However, I'm not quite sure how to actually get this going in Kubernetes, as the sidecar container is throwing a &quot;CrashLoopBackOff&quot; error.</p>
<p>my Dockerfile is pretty thin.  It's an alpine container that really doesn't do anything at all, other than copy a shell script which handles the tunneling.</p>
<p><em>Dockerfile</em></p>
<pre><code>FROM alpine:3.14.0

COPY tunnel.sh /

RUN apk update &amp;&amp; apk add curl \
    wget \
    nano \
    bash \
    ca-certificates \
    openssh-client

RUN chmod +x /tunnel.sh
RUN mkdir ~/.ssh

RUN ssh-keyscan -Ht ecdsa proxysql-sshtunnel.domain.com &gt; ~/.ssh/known_hosts

CMD /bin/bash
</code></pre>
<p><em>tunnel.sh</em></p>
<pre><code>#!/bin/bash
ssh -i /keys/sql_proxy.private -L 3306:10.0.0.229:6033 centos@proxysql-sshtunnel.domain.com -N
</code></pre>
<p>They SSH keys are mounted to the pod from a secret volume in Kubernetes.  My deployment looks like this:</p>
<p><em>deployment.yaml</em></p>
<pre><code>apiVersion: apps/v1
kind: Deployment
metadata:
  name: accounts-deployment
  namespace: default
spec:
  progressDeadlineSeconds: 600
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app: api-accounts
  strategy:
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 25%
    type: RollingUpdate
  template:
    spec:
      containers:
      - image: gcr.io/xxxxxxxx/accounts:VERSION-2.0.6
        imagePullPolicy: Always
        name: accounts
        resources: {}
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /tmp
          name: accounts-keys
          readOnly: true
        - mountPath: /var/www/html/var/spool
          name: mail-spool
      - image: gcr.io/xxxxxxxx/sql-proxy:latest
        imagePullPolicy: IfNotPresent
        name: sql-proxy
        args:
          - -c
          - /tunnel.sh
        command:
          - /bin/bash
        resources: {}
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /keys
          name: keys-sql-proxy
          readOnly: true
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      terminationGracePeriodSeconds: 30
      volumes:
      - name: accounts-keys
        secret:
          defaultMode: 420
          secretName: accounts-keys
      - name: spoonity-sql-proxy
        secret:
          defaultMode: 420
          secretName: spoonity-sql-proxy
      - emptyDir: {}
        name: mail-spool
status:
</code></pre>
<p>&lt;------- The relevant portion is here -------&gt;</p>
<pre><code>...
- image: gcr.io/xxxxxxxx/sql-proxy:latest
  imagePullPolicy: IfNotPresent
  name: sql-proxy
  args:
    - -c
    - /tunnel.sh
  command:
    - /bin/bash
  resources: {}
  terminationMessagePath: /dev/termination-log
  terminationMessagePolicy: File
  volumeMounts:
    - mountPath: /keys
      name: keys-sql-proxy
      readOnly: true
...
</code></pre>
<p>The only logs I get from Kubernetes is: &quot;<strong>/bin/bash: line 1: /tunnel.sh: No such file or directory</strong>&quot;</p>
<p>If I try to run the container locally in docker with <code>docker run sql-proxy:latest /tunnel.sh</code>, then I get a different error complaining that the keys don't exist (which is exactly what I'd be expecting to see).</p>
<p>Not sure where the issue is with this one.</p>
<p>EDIT: tried rebuilding the container locally and including the keys manually. I was able to successfully launch the container.  So it looks like it's definitely a Kubernetes issue, but I'm really not sure why.</p>

<p>So the problem was here:</p>
<pre><code>volumes:
      - name: accounts-keys
        secret:
          defaultMode: 420
          secretName: accounts-keys
      - name: spoonity-sql-proxy
        secret:
          defaultMode: 420 #&lt;----------- this is wrong
          secretName: spoonity-sql-proxy
</code></pre>
<p>SSH requires specific key permissions in order to connect. Kubernetes uses decimal-based file permissions, so the correct value here should be <strong>384</strong>, which will mount the keys with the proper permissions of 0600 in Linux.</p>
<p>Because the permissions were wrong, every time the script tried to execute, it would fail and exit, triggering Kubernetes to try to restart it.</p>
<p>Still not sure why those logs were never generated, but I found this by arbitrarily changing the <code>command</code> and <code>args</code> in my deployment manifest to instead just ping localhost continuously so the container would at least start:</p>
<pre><code>...
 - image: gcr.io/xxxxxxxxx/sql-proxy:latest
   command: [&quot;ping&quot;]
   args: [&quot;127.0.0.1&quot;]
...
</code></pre>
<p>I then connected to the now-running pod, and tried to run the tunnel.sh command by hand.  Now that I could actually see why it was failing, I could fix it.</p>
