<p>I am running my backend using Python and Django with uWSGI. We recently migrated it to Kubernetes (GKE) and our pods are consuming a LOT of memory and the rest of the cluster is starving for resources. We think that this might be related to the uWSGI configuration.</p>

<p>This is our yaml for the pods:</p>

<pre><code>apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: my-pod
  namespace: my-namespace
spec:
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 10
      maxUnavailable: 10
  selector:
    matchLabels:
      app: my-pod
  template:
    metadata:
      labels:
        app: my-pod
    spec:
      containers:
      - name: web
        image: my-img:{{VERSION}}
        imagePullPolicy: IfNotPresent
        ports:
          - containerPort: 8000
            protocol: TCP
        command: ["uwsgi", "--http", ":8000", "--wsgi-file", "onyo/wsgi.py", "--workers", "5", "--max-requests", "10", "--master", "--vacuum", "--enable-threads"]
        resources:
          requests:
            memory: "300Mi"
            cpu: 150m
          limits:
            memory: "2Gi"
            cpu: 1
        livenessProbe:
          httpGet:
            httpHeaders:
              - name: Accept
                value: application/json
            path: "/healthcheck"
            port: 8000
          initialDelaySeconds: 15
          timeoutSeconds: 5
          periodSeconds: 30
        readinessProbe:
          httpGet:
            httpHeaders:
              - name: Accept
                value: application/json
            path: "/healthcheck"
            port: 8000
          initialDelaySeconds: 15
          timeoutSeconds: 5
          periodSeconds: 30
        envFrom:
          - configMapRef:
              name: configmap
          - secretRef:
              name: secrets
        volumeMounts:
        - name: service-account-storage-credentials-volume
          mountPath: /credentials
          readOnly: true
      - name: csql-proxy
        image: gcr.io/cloudsql-docker/gce-proxy:1.11
        command: ["/cloud_sql_proxy",
                  "-instances=my-project:region:backend=tcp:1234",
                  "-credential_file=/secrets/credentials.json"]
        ports:
          - containerPort: 1234
            name: sql
        securityContext:
          runAsUser: 2  # non-root user
          allowPrivilegeEscalation: false
        volumeMounts:
          - name: credentials
            mountPath: /secrets/sql
            readOnly: true
      volumes:
        - name: credentials
          secret:
            secretName: credentials
        - name: volume
          secret:
            secretName: production
            items:
            - key: APPLICATION_CREDENTIALS_CONTENT
              path: key.json
</code></pre>

<p>We are using the same uWSGI configuration that we had before the migration (when the backend was being executed in a VM).</p>

<p>Is there a best practice config for running uWSGI in K8s? Or maybe something that I am doing wrong in this particular config?</p>

<p>You activated 5 workers in uwsgi, that could mean 5 times the need of memory if your application is using lazy-loading techniques (my advice: load everything at startup and trust pre-fork <a href="https://engineering.ticketea.com/uwsgi-preforking-lazy-apps/" rel="nofollow noreferrer">check this</a>). However, you could try reducing number of workers and instead raising number of threads.</p>

<p>Also, you should drop max-requests, this makes your app reload every 10 requests, that's non-sense in a production environment <a href="https://uwsgi-docs.readthedocs.io/en/latest/Options.html#max-requests" rel="nofollow noreferrer">(doc)</a>. If you have troubles with memory leaks, use reload-on-rss instead.</p>

<p>I would do something like this, maybe less or more threads per worker depending on how your app uses it (adjust according to cpu usage/availability per pod in production):</p>

<p><code>command: ["uwsgi", "--http", ":8000", "--wsgi-file", "onyo/wsgi.py", "--workers", "2", "--threads", "10", "--master", "--vacuum", "--enable-threads"]</code></p>

<p>ps: as zerg said in comment you should of course ensure your app is not running DEBUG mode, together with low logging output.</p>
