<p>I want to send data from a sensor written in Python to a Go http Server which are deployed with Kubernetes (k3s) on two Raspberry Pi's. The sensor will read every minute the temperatur and luminosity and send the data as a json with a timestamp to the server. At first when I run the setup it works, but after a while the sensor gets a <code>ConnectionRefusedError: [Errno 111] Connection refused</code> Error in its POST request. However after a while it will continue to work normal until it will break again. I do not know what causes this, since it works part-time. It will just suddenly refuse the connection.</p>
<p>When I use <code>kubectl describe pod weather-sensor-5b88dd65d8-m8zn2</code> I get under Events:</p>
<pre><code>Events:
  Type     Reason   Age                     From     Message
  ----     ------   ----                    ----     -------
  Warning  BackOff  3m16s (x3684 over 18h)  kubelet  Back-off restarting failed container
</code></pre>
<p>running <code>kubectl logs weather-sensor-5b88dd65d8-m8zn2</code> says:</p>
<pre><code>send this data:  {&quot;time&quot;: 1635531114, &quot;temp&quot;: &quot;23.25&quot;, &quot;lux&quot;: &quot;254&quot;}
DEBUG:urllib3.connectionpool:Starting new HTTP connection (1): weather-server:8080
DEBUG:urllib3.connectionpool:http://weather-server:8080 &quot;POST /weather HTTP/1.1&quot; 200 0
send this data:  {&quot;time&quot;: 1635531175, &quot;temp&quot;: &quot;23.25&quot;, &quot;lux&quot;: &quot;252&quot;}
DEBUG:urllib3.connectionpool:Starting new HTTP connection (1): weather-server:8080
Traceback (most recent call last):
  File &quot;/usr/local/lib/python3.8/site-packages/urllib3/connection.py&quot;, line 174, in _new_conn
    conn = connection.create_connection(
  File &quot;/usr/local/lib/python3.8/site-packages/urllib3/util/connection.py&quot;, line 96, in create_connection
    raise err
  File &quot;/usr/local/lib/python3.8/site-packages/urllib3/util/connection.py&quot;, line 86, in create_connection
    sock.connect(sa)
ConnectionRefusedError: [Errno 111] Connection refused
</code></pre>
<p>The sensor will then continue to try to connect to the server until it gets a MaxRetryError. Then Kubernetes will terminate the pod because of <code>CrashLoopBackOff</code></p>
<p>On the sensor I use this url for the post request: <code>URL = &quot;http://weather-server:8080/weather&quot;</code></p>
<p>In the logs on the server side, I haven't seen anything unusual except that it only gets data erratically.</p>
<p>Relevant Python code:</p>
<pre><code>def create_data(temp, lux):
    weather = {
        'time': int(time.time()),
        'temp': temp,
        'lux': lux
    }
    return json.dumps(weather)


def send_data(data):
    try:
        headers = {'Content-Type': 'application/json'}
        requests.post(url=URL, data=data, headers=headers)
    except ConnectionError as e:
        print(e)
</code></pre>
<p>Here my yml files:</p>
<p>sensor_deployment.yml</p>
<pre><code>apiVersion: apps/v1
kind: Deployment
metadata:
  name: weather-sensor
  labels:
    app: weather
spec:
  replicas: 1
  selector:
    matchLabels:
      app: weather
  template:
    metadata:
      labels:
        app: weather
    spec:
      containers:
      - name: weather-sensor
        image: weather-sensor:pi-1.14
        imagePullPolicy: IfNotPresent
        securityContext:
          privileged: true
</code></pre>
<p>server_deployment.yml</p>
<pre><code>apiVersion: apps/v1
kind: Deployment
metadata:
  name: weather-server
  labels:
    app: weather
spec:
  replicas: 2
  selector:
    matchLabels:
      app: weather
  template:
    metadata:
      labels:
        app: weather
    spec:
      containers:
      - name: weather-server
        image: weather-server:pi-1.15
        imagePullPolicy: IfNotPresent
        ports:
        - containerPort: 8080
</code></pre>
<p>server_service.yml</p>
<pre><code>apiVersion: v1
kind: Service
metadata:
  name: weather-server
  labels:
    app: weather
spec:
  type: NodePort
  selector:
    app: weather
  ports:
    - port: 8080
      protocol: TCP
      targetPort: 8080
</code></pre>
<p>Any ideas?</p>

TheQuestioner
<p>I think the problem is the weather server (k8s service) is broken.</p>
<p>This is because the selector is checking for pods with label <code>app=weather</code> which includes both server pods and sensor pods.</p>
<p>If a sensor tries to send data (through the k8s service) to another sensor pod, then it will result in the error because the sensor does not listen for HTTP requests.</p>
<p>To fix it, ensure that the app label is unique for each pod type. For example, weather-server has <code>app=weather-server</code> and weather-sensor has <code>app=weather-sensor</code>.</p>
<p><strong>server_deployment.yml</strong></p>
<pre class="lang-yaml prettyprint-override"><code>apiVersion: apps/v1
kind: Deployment
metadata:
  name: weather-server
  labels:
    app: weather-server
spec:
  replicas: 2
  selector:
    matchLabels:
      app: weather-server
  template:
    metadata:
      labels:
        app: weather-server
    spec:
      containers:
      - name: weather-server
        image: weather-server:pi-1.15
        imagePullPolicy: IfNotPresent
        ports:
        - containerPort: 8080
</code></pre>
<p><strong>server_service.yml</strong></p>
<pre class="lang-yaml prettyprint-override"><code>apiVersion: v1
kind: Service
metadata:
  name: weather-server
  labels:
    app: weather-server
spec:
  type: NodePort
  selector:
    app: weather-server
  ports:
    - port: 8080
      protocol: TCP
      targetPort: 8080
</code></pre>
