<p>What I am craving for is to have 2 applications running in a pod, each of those applications has its own container. The Application A is a simple spring-boot application which makes HTTP requests to the other application which is deployed on Kubernetes. The purpose of Application B (proxy) is to intercept that HTTP request and add an Authorization token to its header. The Application B is a mitmdump with a python script. The issue I am having is that when I have deployed in on Kubernetes, the proxy seems to not intercept any traffic at all ( I tried to reproduce this issue on my local machine and I didn't find any troubles, so I guess the issue lies somewhere withing networking inside a pod). Can someone have a look into it and guide me how to solve it?</p>

<p><a href="https://i.stack.imgur.com/CpQ8k.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/CpQ8k.png" alt="enter image description here"></a></p>

<p>Here's the deployment and service file.</p>

<pre><code>apiVersion: apps/v1
kind: Deployment
metadata:
  name: proxy-deployment
  namespace: myown
  labels:
    app: application-a
spec:
  replicas: 1
  selector:
    matchLabels:
      app: application-a
  template:
    metadata:
      labels:
        app: application-a
    spec:
      containers:
      - name: application-a
        image: registry.gitlab.com/application-a
        resources:
          requests:
            memory: "230Mi"
            cpu: "100m"
          limits:
            memory: "460Mi"
            cpu: "200m"
        imagePullPolicy: Always
        ports:
        - containerPort: 8090
        env:
        - name: "HTTP_PROXY"
          value: "http://localhost:1030"
      - name:
        image: registry.gitlab.com/application-b-proxy
        resources:
          requests:
            memory: "230Mi"
            cpu: "100m"
          limits:
            memory: "460Mi"
            cpu: "200m"
        imagePullPolicy: Always
        ports:
        - containerPort: 1080
---
kind: Service
apiVersion: v1
metadata:
  name: proxy-svc
  namespace: myown
spec:
  ports:
  - nodePort: 31000
    port: 8090
    protocol: TCP
    targetPort: 8090
  selector:
    app: application-a
  sessionAffinity: None
  type: NodePort
</code></pre>

<p>And here's how i build the docker image of mitmproxy/mitmdump</p>

<pre><code>FROM mitmproxy/mitmproxy:latest

ADD get_token.py .
WORKDIR ~/mit_docker
COPY get_token.py .
EXPOSE 1080:1080
ENTRYPOINT ["mitmdump","--listen-port", "1030", "-s","get_token.py"]
</code></pre>

<p><strong>EDIT</strong></p>

<p>I created two dummy docker images in order to have this scenario recreated locally.</p>

<p><strong>APPLICATION A</strong> - a spring boot application with a job to create an HTTP GET request every 1 minute for specified but irrelevant address, the address should be accessible. The response should be 302 FOUND. Every time an HTTP request is made, a message in the logs of the application appears.</p>

<p><strong>APPLICATION B</strong> - a proxy application which is supposed to proxy the docker container with application A. Every request is logged.</p>

<ol>
<li><p>Make sure your docker proxy config is set to listen to <a href="http://localhost:8080" rel="nofollow noreferrer">http://localhost:8080</a> - <a href="https://docs.docker.com/network/proxy/" rel="nofollow noreferrer">you can check how to do so here</a></p></li>
<li><p>Open a terminal and run this command:</p></li>
</ol>

<pre><code> docker run -p 8080:8080 -ti registry.gitlab.com/dyrekcja117/proxyexample:application-b-proxy
</code></pre>

<ol start="3">
<li>Open another terminal and run this command:</li>
</ol>

<pre><code>    docker run --network="host" registry.gitlab.com/dyrekcja117/proxyexample:application-a
</code></pre>

<ol start="4">
<li>Go into the shell with the container of application A in 3rd terminal:</li>
</ol>

<pre><code>    docker exec -ti &lt;name of docker container&gt; sh
</code></pre>

<p>and try to make curl to whatever address you want. </p>

<p>And the issue I am struggling with is that when I make curl from inside the container with Application A it is intercepted by my proxy and it can be seen in the logs. But whenever Application A itself makes the same request it is not intercepted. The same thing happens on Kubernetes</p>

<p>Let's first wrap up the facts we discover over our troubleshooting discussion in the comments:</p>

<ul>
<li>Your need is that APP-A receives a HTTP request and a token needs to be added inflight by PROXY before sending the request to your datastorage.</li>
<li>Every container in a Pod shares the network namespace, including the IP address and network ports. Containers <em>inside a Pod</em> can communicate with one another using <code>localhost</code>, source <a href="https://kubernetes.io/docs/concepts/workloads/pods/pod-overview/#networking" rel="nofollow noreferrer">here</a>.</li>
<li>You was able to login to container <code>application-a</code> and send a <code>curl</code> request to container <code>
application-b-proxy</code> on port <code>1030</code>, proving the above statement.</li>
<li>The problem is that your proxy is not intercepting the request as expected.</li>
<li>You mention that in you was able to make it work on localhost, but in localhost the proxy has more power than inside a container.</li>
<li>Since I don't have access neither to your <code>app-a</code> code nor the mitmproxy <code>token.py</code> I will give you a general example how to redirect traffic from <code>container-a</code> to <code>container-b</code></li>
<li>In order to make it work, I'll use <a href="http://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_pass" rel="nofollow noreferrer">NGINX Proxy Pass</a>: it simply proxies the request to <code>container-b</code>.</li>
</ul>

<hr>

<p><strong>Reproduction:</strong></p>

<ul>
<li><p>I'll use a nginx server as  <code>container-a</code>.</p></li>
<li><p>I'll build it with this <code>Dockerfile</code>:</p></li>
</ul>

<pre><code>FROM nginx:1.17.3
RUN rm /etc/nginx/conf.d/default.conf
COPY frontend.conf /etc/nginx/conf.d
</code></pre>

<ul>
<li>I'll add  this configuration file  <code>frontend.conf</code>:</li>
</ul>

<pre><code>server {
    listen 80;

    location / {
        proxy_pass http://127.0.0.1:8080;
    }
}
</code></pre>

<p>It's ordering the traffic should be sent to <code>container-b</code> that is listening in <code>port 8080</code> inside the same pod.</p>

<ul>
<li>I'll build this image as <code>nginxproxy</code> in my local repo: </li>
</ul>

<pre><code>$ docker build -t nginxproxy .

$ docker images 
REPOSITORY        TAG       IMAGE ID        CREATED          SIZE
nginxproxy    latest    7c203a72c650    4 minutes ago    126MB
</code></pre>

<ul>
<li>Now the <code>full.yaml</code> deployment:</li>
</ul>

<pre><code>apiVersion: apps/v1
kind: Deployment
metadata:
  name: proxy-deployment
  labels:
    app: application-a
spec:
  replicas: 1
  selector:
    matchLabels:
      app: application-a
  template:
    metadata:
      labels:
        app: application-a
    spec:
      containers:
      - name: container-a
        image: nginxproxy:latest
        ports:
        - containerPort: 80
        imagePullPolicy: Never
      - name: container-b
        image: echo8080:latest
        ports:
        - containerPort: 8080
        imagePullPolicy: Never
---
apiVersion: v1
kind: Service
metadata:
  name: proxy-svc
spec:
  ports:
  - nodePort: 31000
    port: 80
    protocol: TCP
    targetPort: 80
  selector:
    app: application-a
  sessionAffinity: None
  type: NodePort    
</code></pre>

<p><strong>NOTE:</strong> I set <code>imagePullPolicy</code> as <code>Never</code> because I'm using my local docker image cache.</p>

<p>I'll list the changes I made to help you link it to your current environment:</p>

<ul>
<li><code>container-a</code> is doing the work of your <code>application-a</code> and I'm serving <code>nginx</code> on <code>port 80</code> where you are using <code>port 8090</code></li>
<li><p><code>container-b</code> is receiving the request, as your <code>application-b-proxy</code>. The image I'm using was based on <code>mendhak/http-https-echo</code>, normally it listens on <code>port 80</code>, I've made a custom image just changing to listen on <code>port 8080</code> and named it <code>echo8080</code>.</p></li>
<li><p>First I created a <code>nginx</code> pod and exposed it alone to show you it's running (since it's empty in content, it will return <code>bad gateway</code> but you can see the output is from nginx:</p></li>
</ul>

<pre><code>$ kubectl apply -f nginx.yaml 
pod/nginx created
service/nginx-svc created

$ kubectl get pods
NAME                               READY   STATUS    RESTARTS   AGE
nginx                              1/1     Running   0          64s
$ kubectl get svc
NAME         TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)        AGE
nginx-svc    NodePort    10.103.178.109   &lt;none&gt;        80:31491/TCP   66s

$ curl http://192.168.39.51:31491
&lt;html&gt;
&lt;head&gt;&lt;title&gt;502 Bad Gateway&lt;/title&gt;&lt;/head&gt;
&lt;body&gt;
&lt;center&gt;&lt;h1&gt;502 Bad Gateway&lt;/h1&gt;&lt;/center&gt;
&lt;hr&gt;&lt;center&gt;nginx/1.17.3&lt;/center&gt;
&lt;/body&gt;
&lt;/html&gt;
</code></pre>

<ul>
<li>I deleted the <code>nginx</code> pod and created a <code>echo-app</code>pod and exposed it to show you the response it gives when directly curled from outside:</li>
</ul>

<pre><code>$ kubectl apply -f echo.yaml 
pod/echo created
service/echo-svc created

$ kubectl get pods
NAME                               READY   STATUS    RESTARTS   AGE
echo                               1/1     Running   0          118s
$ kubectl get svc
NAME         TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)          AGE
echo-svc     NodePort    10.102.168.235   &lt;none&gt;        8080:32116/TCP   2m

$ curl http://192.168.39.51:32116
{
  "path": "/",
  "headers": {
    "host": "192.168.39.51:32116",
    "user-agent": "curl/7.52.1",
  },
  "method": "GET",
  "hostname": "192.168.39.51",
  "ip": "::ffff:172.17.0.1",
  "protocol": "http",
  "os": {
    "hostname": "echo"
  },
</code></pre>

<ul>
<li>Now I'll apply the <code>full.yaml</code>:</li>
</ul>

<pre><code>$ kubectl apply -f full.yaml 
deployment.apps/proxy-deployment created
service/proxy-svc created
$ kubectl get pods
NAME                               READY   STATUS    RESTARTS   AGE
proxy-deployment-9fc4ff64b-qbljn   2/2     Running   0          1s

$ k get service
NAME         TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)        AGE
proxy-svc    NodePort    10.103.238.103   &lt;none&gt;        80:31000/TCP   31s
</code></pre>

<ul>
<li>Now the Proof of concept, from outside the cluster, I'll send a curl to my node IP <code>192.168.39.51</code> in port <code>31000</code> which is sending the request to <code>port 80</code> on the pod (handled by <code>nginx</code>):</li>
</ul>

<pre><code>$ curl http://192.168.39.51:31000
{
  "path": "/",
  "headers": {
    "host": "127.0.0.1:8080",
    "user-agent": "curl/7.52.1",
  },
  "method": "GET",
  "hostname": "127.0.0.1",
  "ip": "::ffff:127.0.0.1",
  "protocol": "http",
  "os": {
    "hostname": "proxy-deployment-9fc4ff64b-qbljn"
  },
</code></pre>

<ul>
<li>As you can see, the response has all the parameters of the pod, indicating it was sent from <code>127.0.0.1</code> instead of a public IP, showing that the NGINX is proxying the request to <code>container-b</code>.</li>
</ul>

<hr>

<p><strong>Considerations:</strong></p>

<ul>
<li>This example was created to show you how the communication works inside kubernetes.</li>
<li>You will have to check how your <code>application-a</code> is handling the requests and edit it to send the traffic to your proxy.</li>
<li>Here are a few links with tutorials and explanation that could help you port your application to kubernetes environment:

<ul>
<li><a href="https://gist.github.com/soheilhy/8b94347ff8336d971ad0" rel="nofollow noreferrer">Virtual Hosts on nginx</a></li>
<li><a href="https://www.magalix.com/blog/implemeting-a-reverse-proxy-server-in-kubernetes-using-the-sidecar-pattern" rel="nofollow noreferrer">Implementing a Reverse proxy Server in Kubernetes Using the Sidecar Pattern</a></li>
<li><a href="https://www.nginx.com/blog/validating-oauth-2-0-access-tokens-nginx/" rel="nofollow noreferrer">Validating OAuth 2.0 Access Tokens with NGINX and NGINX Plus</a></li>
<li><a href="https://developer.okta.com/blog/2018/08/28/nginx-auth-request" rel="nofollow noreferrer">Use nginx to Add Authentication to Any Application</a></li>
<li><a href="https://kubernetes.io/docs/tasks/access-application-cluster/connecting-frontend-backend/" rel="nofollow noreferrer">Connecting a Front End to a Back End Using a Service</a></li>
<li><a href="https://cloud.google.com/community/tutorials/transparent-proxy-and-filtering-on-k8s" rel="nofollow noreferrer">Transparent Proxy and Filtering on K8s</a></li>
</ul></li>
</ul>

<p>I Hope to help you with this example.</p>
