<p>I have an Istio-enabled EKS Cluster, and my nodes are constantly running out of disk space.</p>

<p>Calculating the overall disk usage lead me to the istio-mixer container, which has a <strong>log file</strong> using more than <strong>50GB</strong> of disk space in only <strong>12 days</strong> of uptime:</p>

<pre><code>[root@ip-some-ip containers]# pwd
/var/lib/docker/containers

[root@ip-some-ip containers]# du -schx .[!.]* * | sort -h | tail -n 10
66M 8bf5e8ee5a03096c589ad8f53b9e1a3d3088ca67b0064f3796e406f00336b532
73M 657eca261461d10c5b1b81ab3078d2058b931a357395903808b0145b617c1662
101M    bb338296ff06ef42ae6177c8a88e63438c26c398a457dc3f5301ffcb4ef2682b
127M    21f2da86055ad76882730abf65d4465386bb85598f797f451e7ad66726243613
134M    9c2be24e8b9345659b6f208c9f2d4650bb1ece11e0c4b0793aa01fdfebadb44e
383M    5d5fdbe6813ddc3ff2f6eb96f62f8317bd73e24730e2f44ebc537367d9987142
419M    475f8dfc74c3df2bc95c47df56e37d1dfb9181fae9aa783dafabba8283023115
592M    9193c50e586e0c7ecaeb87cecd8be13714a5d6ccd6ea63557c034ef56b07772f
52G 9c6b3e4f26603471d0aa9b6a61b3da5a69001e6b9be34432ffa62d577738c149
54G total

[root@ip-192-168-228-194 containers]# du -hs 9c6b3e4*/*.log
52G 9c6b3e4f26603471d0aa9b6a61b3da5a69001e6b9be34432ffa62d577738c149-json.log

[root@ip-ip-some-ip containers]# docker ps -a | grep 9c6b3e4f2660
9c6b3e4f2660        d559bdcd7a88                                                       "/usr/local/bin/mi..."   12 days ago         Up 12 days                                     k8s_mixer_istio-telemetry-6b5579595f-fvm5x_istio-system_6324c262-f3b5-11e8-b615-0eccb0bb4724_0
</code></pre>

<p><strong>My questions are:</strong>  </p>

<ul>
<li>This amount of log output is expected?</li>
<li>The mixer log level can be decreased? How? Changing it affects my telemetry metrics?</li>
<li>There is a way to configure a log "retention period"?</li>
</ul>

<p><strong>Additional info:</strong></p>

<ul>
<li>Istio v1.0.2 (deployed with the offical helm charts; no custom configs)</li>
<li>k8s v1.10.11-eks</li>
<li>The cluster has approximately 20 pods running in Istio-enabled namespaces</li>
</ul>

<p>The default value of logging level in <strong>Mixer</strong> is info. And the logs provided by you, confirms that you have this settings. Therefore, a lot of redundant information gathered in logs and it is possible to decrease logging level for some sources.</p>

<p>You can change it in two ways:</p>

<ol>
<li><p><strong>On working pod without restart.</strong></p>

<p>In your logs you can find the following line:</p>

<pre><code>2018-12-12T17:54:55.461261Z info    ControlZ available at 192.168.87.249:9876
</code></pre>

<p>It means, that in the mixer container on 9876 port you can find Istio ControlZ web-interface. To get an access to it from a computer with installed <code>kubectl</code>, you need to run the following command:</p>

<pre><code>kubectl -n istio-system port-forward $(kubectl -n istio-system get pod -l istio=mixer,istio-mixer-type=telemetry -o jsonpath='{.items[0].metadata.name}') 9876:9876 &amp;
</code></pre>

<p>After that, in your browser go to the link <a href="http://localhost:9876/scopez/" rel="nofollow noreferrer">http://localhost:9876/scopez/</a>, and you will see the following dashboard, where you can change log levels:</p>

<p><a href="https://i.stack.imgur.com/yHkID.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/yHkID.png" alt="enter image description here"></a></p></li>
<li><p><strong>Add <code>--log_output_level</code> flag to the <code>istio-telemetry</code> deployment for the <code>mixer</code> container.</strong></p>

<p>Here is the description for the flag from the mixer's documentation:</p>

<blockquote>
  <p><strong>--log_output_level string</strong><br>
  Comma-separated minimum per-scope logging level of messages to output, in the form of :,:,... where scope can be one of [adapters, api, attributes, default, grpcAdapter, loadshedding] and level can be one of [debug, info, warn, error, none] (default "default:info")</p>
</blockquote>

<p>Note, that for key <code>--log_output_level attributes:warn,api:error</code> in yaml file you need to use one of the following:</p>

<ul>
<li>value <code>- --log_output_level=attributes:warn,api:error</code> or</li>
<li>values <code>- --log_output_level</code> and <code>- attributes:warn,api:error</code> on different lines      </li>
</ul>

<p>The example of the deployment:    </p>

<pre><code>apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  annotations:
  labels:
    chart: mixer-1.0.4
    istio: mixer
    release: istio
  name: istio-telemetry
  namespace: istio-system
spec:
  progressDeadlineSeconds: 600
  replicas: 2
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app: telemetry
      istio: mixer
      istio-mixer-type: telemetry
  strategy:
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 1
    type: RollingUpdate
  template:
    metadata:
      annotations:
        scheduler.alpha.kubernetes.io/critical-pod: ""
        sidecar.istio.io/inject: "false"
      creationTimestamp: null
      labels:
        app: telemetry
        istio: mixer
        istio-mixer-type: telemetry
    spec:
      containers:
      - args:                                                #Flags for the Mixer process
        - --address                                          #Flag on two different lines
        - unix:///sock/mixer.socket
        - --configStoreURL=k8s://                            #Flag with '='
        - --configDefaultNamespace=istio-system
        - --trace_zipkin_url=http://zipkin:9411/api/v1/spans
        - --log_output_level=attributes:warn,api:error       # &lt;------ THIS LINE IS WHAT YOU ARE LOOKING FOR
        env:
        - name: GODEBUG
          value: gctrace=2
        image: docker.io/istio/mixer:1.0.4
        imagePullPolicy: IfNotPresent
        livenessProbe:
          failureThreshold: 3
          httpGet:
            path: /version
            port: 9093
            scheme: HTTP
          initialDelaySeconds: 5
          periodSeconds: 5
          successThreshold: 1
          timeoutSeconds: 1
        name: mixer
        ports:
        - containerPort: 9093
          protocol: TCP
        - containerPort: 42422
          protocol: TCP
        resources:
          requests:
            cpu: 10m
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /sock
          name: uds-socket
      - args:
        - proxy
        - --serviceCluster
        - istio-telemetry
        - --templateFile
        - /etc/istio/proxy/envoy_telemetry.yaml.tmpl
        - --controlPlaneAuthPolicy
        - MUTUAL_TLS
        env:
        - name: POD_NAME
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.name
        - name: POD_NAMESPACE
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.namespace
        - name: INSTANCE_IP
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: status.podIP
        image: docker.io/istio/proxyv2:1.0.4
        imagePullPolicy: IfNotPresent
        name: istio-proxy
        ports:
        - containerPort: 15090
          name: http-envoy-prom
          protocol: TCP
        resources:
          requests:
            cpu: 10m
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /etc/certs
          name: istio-certs
          readOnly: true
        - mountPath: /sock
          name: uds-socket
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      serviceAccount: istio-mixer-service-account
      serviceAccountName: istio-mixer-service-account
      terminationGracePeriodSeconds: 30
      volumes:
      - name: istio-certs
        secret:
          defaultMode: 420
          optional: true
          secretName: istio.istio-mixer-service-account
      - emptyDir: {}
        name: uds-socket
</code></pre></li>
</ol>

<p><strong>Additionally, you can configure log rotation for mixer process using the following flags:</strong></p>

<blockquote>
  <p>--log_rotate string                   The path for the optional rotating log file</p>
  
  <p>--log_rotate_max_age int              The maximum age in days of a log file beyond which the file is rotated (0 indicates no limit) (default 30)</p>
  
  <p>--log_rotate_max_backups int          The maximum number of log file backups to keep before older files are deleted (0 indicates no limit) (default 1000)</p>
  
  <p>--log_rotate_max_size int             The maximum size in megabytes of a log file beyond which the file is rotated (default 104857600)</p>
</blockquote>

<p>However, I have no possibility to generate a huge amount of such logs and test how it works.</p>

<p><strong>Links:</strong></p>

<p>Unfortunately, the <a href="https://istio.io/help/ops/component-logging/" rel="nofollow noreferrer">official documentation</a> is not good, but maybe it helps somehow.</p>

<p>And as a bonus, <a href="https://istio.io/docs/reference/commands/mixs/#mixs-server" rel="nofollow noreferrer">here</a> is the list of all mixer server flags.</p>
