<p>I have a pair of <code>kubernetes</code> pods, one for <code>nginx</code> and one for <code>Python Flask</code> + <code>uWSGI</code>. I have tested my setup locally in <code>docker-compose</code>, and it has worked fine, however after deploying to <code>kubernetes</code> somehow it seems there is no communication between the two. The end result is that I get 502 Gateway Error when trying to reach my location.</p>

<p>So my question is not really about what is wrong with my setup, but rather what tools can I use to debug this scenario. Is there a test-client for uwsgi? Can I use ncat? I don't seem to get any useful log output from nginx, and I don't know if uwsgi even has a log.</p>

<p><strong>How can I debug this?</strong></p>

<p>For reference, here is my nginx location:</p>

<pre><code>location / {
        # Trick to avoid nginx aborting at startup (set server in variable)
        set $upstream_server ${APP_SERVER};

        include            uwsgi_params;
        uwsgi_pass         $upstream_server;
        uwsgi_read_timeout 300;
        uwsgi_intercept_errors on;
}
</code></pre>

<p>Here is my wsgi.ini:</p>

<pre><code>[uwsgi]
module = my_app.app
callable = app
master = true
processes = 5
socket = 0.0.0.0:5000
die-on-term = true

uid = www-data
gid = www-data
</code></pre>

<p>Here is the kubernetes deployment.yaml for nginx:</p>

<pre><code>apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    service: nginx
  name: nginx
spec:
  replicas: 1
  revisionHistoryLimit: 2
  selector:
    matchLabels:
      service: nginx
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        service: nginx
    spec:
      imagePullSecrets:
      - name: docker-reg
      containers:
      - name: nginx
        image: &lt;custom image url&gt;
        imagePullPolicy: Always
        env:
          - name: APP_SERVER
            valueFrom:
              secretKeyRef:
                name: my-environment-config
                key: APP_SERVER
          - name: FK_SERVER_NAME
            valueFrom:
              secretKeyRef:
                name: my-environment-config
                key: SERVER_NAME
        ports:
        - containerPort: 80
        - containerPort: 10443
        - containerPort: 10090
        resources:
          requests:
            cpu: 1m
            memory: 200Mi
        volumeMounts:
        - mountPath: /etc/letsencrypt
          name: my-storage
          subPath: nginx
        - mountPath: /dev/shm
          name: dshm
      restartPolicy: Always
      volumes:
      - name: my-storage
        persistentVolumeClaim:
          claimName: my-storage-claim-nginx
      - name: dshm
        emptyDir:
          medium: Memory
</code></pre>

<p>Here is the kubernetes service.yaml for nginx:</p>

<pre><code>apiVersion: v1
kind: Service
metadata:
  labels:
    service: nginx
  name: nginx
spec:
  type: LoadBalancer
  ports:
  - name: "nginx-port-80"
    port: 80
    targetPort: 80
    protocol: TCP
  - name: "nginx-port-443"
    port: 443
    targetPort: 10443
    protocol: TCP
  - name: "nginx-port-10090"
    port: 10090
    targetPort: 10090
    protocol: TCP
  selector:
    service: nginx
</code></pre>

<p>Here is the kubernetes deployment.yaml for python flask:</p>

<pre><code>apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    service: my-app
  name: my-app
spec:
  replicas: 1
  revisionHistoryLimit: 2
  selector:
    matchLabels:
      service: my-app
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        service: my-app
    spec:
      imagePullSecrets:
      - name: docker-reg
      containers:
      - name: my-app
        image: &lt;custom image url&gt;
        imagePullPolicy: Always
        ports:
        - containerPort: 5000
        resources:
          requests:
            cpu: 1m
            memory: 100Mi
        volumeMounts:
        - name: merchbot-storage
          mountPath: /app/data
          subPath: my-app
        - name: dshm
          mountPath: /dev/shm
        - name: local-config
          mountPath: /app/secrets/local_config.json
          subPath: merchbot-local-config-test.json
      restartPolicy: Always
      volumes:
      - name: merchbot-storage
        persistentVolumeClaim:
          claimName: my-storage-claim-app
      - name: dshm
        emptyDir:
          medium: Memory
      - name: local-config
        secret:
          secretName: my-app-local-config
</code></pre>

<p>Here is the kubernetes service.yaml for nginx:</p>

<pre><code>apiVersion: v1
kind: Service
metadata:
  labels:
    service: my-app
  name: my-app
spec:
  ports:
  - name: "my-app-port-5000"
    port: 5000
    targetPort: 5000
  selector:
    service: my-app
</code></pre>

<p>Debugging in kubernetes is not very different from debugging outside, there's just some concepts that need to be overlaid for the kubernetes world. </p>

<p>A Pod in kubernetes is what you would conceptually see as a host in the VM world. Every container running in a Pod will see each others services on localhost. From there, a Pod to anything else will have a network connection involved (even if the endpoint is node local).  So start testing with services on localhost and work your way out through pod IP, service IP, service name.</p>

<p>Some complexity comes from having the debug tools available in the containers. Generally containers are built slim and don't have everything available. So you either need to install tools while a container is running (if you can) or build a special "debug" container you can deploy on demand in the same environment. You can always fall back to testing from the cluster nodes which also have access.</p>

<p>Where you have python available you can <a href="https://stackoverflow.com/a/32893520/1318694">test with <code>uswgi_curl</code></a></p>

<pre><code>pip install uwsgi-tools
uwsgi_curl hostname:port /path
</code></pre>

<p>Otherwise <code>nc</code>/<code>curl</code> will suffice, to a point. </p>

<h3>Pod to localhost</h3>

<p>First step is to make sure the container itself is responding. In this case you are likely to have python/pip available to use <code>uwsgi_curl</code></p>

<pre><code>kubectl exec -ti my-app-XXXX-XXXX sh
nc -v localhost 5000
uwsgi_curl localhost:5000 /path
</code></pre>

<h3>Pod to Pod/Service</h3>

<p>Next include the kubernetes networking. Start with IP's and finish with names. </p>

<p>Less likely to have <code>python</code> here, or even <code>nc</code> but I think testing the environment variables is important here:</p>

<pre><code>kubectl exec -ti nginx-XXXX-XXXX sh
nc -v my-app-pod-IP 5000
nc -v my-app-service-IP 5000
nc -v my-app-service-name 5000

echo $APP_SERVER
echo $FK_SERVER_NAME
nc -v $APP_SERVER 5000
# or 
uwsgi_curl $APP_SERVER:5000 /path
</code></pre>

<h3>Debug Pod to Pod/Service</h3>

<p>If you do need to use a debug pod, try and mimic the pod you are testing as much as possible. It's great to have a generic debug pod/deployment to quickly test anything, but if that doesn't reveal the issue you may need to customise the deployment to mimic the pod you are testing more closely. </p>

<p>In this case the environment variables play a part in the connection setup, so that should be emulated for a debug pod. </p>

<h3>Node to Pod/Service</h3>

<p>Pods/Services will be available from the cluster nodes (if you are not using restrictive network policies) so usually the quick test is to check Pods/Services are working from there:</p>

<pre><code>nc -v &lt;pod_ip&gt; &lt;container_port&gt;
nc -v &lt;service_ip&gt; &lt;service_port&gt;
nc -v &lt;service__dns&gt; &lt;service_port&gt;
</code></pre>

<p>In this case:</p>

<pre><code>nc -v &lt;my_app_pod_ip&gt; 5000
nc -v &lt;my_app_service_ip&gt; 5000
nc -v my-app.svc.&lt;namespace&gt;.cluster.local 5000
</code></pre>
