<p>I am trying to add audit logging in one of my k8s clusters.
So far YAML manifest are below</p>
<p>Policy file - /etc/kubernetes/policy/audit/policy.yaml</p>
<pre><code>apiVersion: audit.k8s.io/v1
kind: Policy
rules:

# log Secret resources audits, level Metadata
- level: Metadata
  resources:
  - group: &quot;&quot;
    resources: [&quot;secrets&quot;]

# log node related audits, level RequestResponse
- level: RequestResponse
  userGroups: [&quot;system:nodes&quot;]

# for everything else don't log anything
- level: None
</code></pre>
<p>kuber-apiserver file - manifests/kube-apiserver.yaml</p>
<pre><code>apiVersion: v1
kind: Pod
metadata:
  annotations:
    kubeadm.kubernetes.io/kube-apiserver.advertise-address.endpoint: 192.168.0.XX:6443
  creationTimestamp: null
  labels:
    component: kube-apiserver
    tier: control-plane
  name: kube-apiserver
  namespace: kube-system
spec:
  containers:
  - command:
    - --audit-policy-file=/etc/kubernetes/audit-policy/policy.yaml
    - --audit-log-path=/etc/kubernetes/audit-logs/audit.log
    - --audit-log-maxsize=7
    - kube-apiserver
    - --encryption-provider-config=/etc/kubernetes/encryption/encryptionconfiguration.yaml
    - --advertise-address=192.168.0.XX
    - --allow-privileged=true
    - --authorization-mode=Node,RBAC
    - --client-ca-file=/etc/kubernetes/pki/ca.crt
    - --enable-admission-plugins=NodeRestriction
    - --enable-bootstrap-token-auth=true
    - --etcd-cafile=/etc/kubernetes/pki/etcd/ca.crt
    - --etcd-certfile=/etc/kubernetes/pki/apiserver-etcd-client.crt
    - --etcd-keyfile=/etc/kubernetes/pki/apiserver-etcd-client.key
    - --etcd-servers=https://127.0.0.1:2379
    - --kubelet-client-certificate=/etc/kubernetes/pki/apiserver-kubelet-client.crt
    - --kubelet-client-key=/etc/kubernetes/pki/apiserver-kubelet-client.key
    - --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname
    - --proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.crt
    - --proxy-client-key-file=/etc/kubernetes/pki/front-proxy-client.key
    - --requestheader-allowed-names=front-proxy-client
    - --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.crt
    - --requestheader-extra-headers-prefix=X-Remote-Extra-
    - --requestheader-group-headers=X-Remote-Group
    - --requestheader-username-headers=X-Remote-User
    - --secure-port=6443
    - --service-account-issuer=https://kubernetes.default.svc.cluster.local
    - --service-account-key-file=/etc/kubernetes/pki/sa.pub
    - --service-account-signing-key-file=/etc/kubernetes/pki/sa.key
    - --service-cluster-ip-range=10.96.0.0/12
    - --tls-cert-file=/etc/kubernetes/pki/apiserver.crt
    - --tls-private-key-file=/etc/kubernetes/pki/apiserver.key
    image: k8s.gcr.io/kube-apiserver:v1.24.3
    imagePullPolicy: IfNotPresent
    livenessProbe:
      failureThreshold: 8
      httpGet:
        host: 192.168.0.XX
        path: /livez
        port: 6443
        scheme: HTTPS
      initialDelaySeconds: 10
      periodSeconds: 10
      timeoutSeconds: 15
    name: kube-apiserver
    readinessProbe:
      failureThreshold: 3
      httpGet:
        host: 192.168.0.XX
        path: /readyz
        port: 6443
        scheme: HTTPS
      periodSeconds: 1
      timeoutSeconds: 15
    resources:
      requests:
        cpu: 250m
    startupProbe:
      failureThreshold: 24
      httpGet:
        host: 192.168.0.XX
        path: /livez
        port: 6443
        scheme: HTTPS
      initialDelaySeconds: 10
      periodSeconds: 10
      timeoutSeconds: 15
    volumeMounts:
    - mountPath: /etc/ssl/certs
      name: ca-certs
      readOnly: true
    - mountPath: /etc/ca-certificates
      name: etc-ca-certificates
      readOnly: true
    - mountPath: /etc/kubernetes/pki
      name: k8s-certs
      readOnly: true
    - mountPath: /usr/local/share/ca-certificates
      name: usr-local-share-ca-certificates
      readOnly: true
    - mountPath: /usr/share/ca-certificates
      name: usr-share-ca-certificates
      readOnly: true
    - mountPath: /etc/kubernetes/encryption/
      name: enc-conf
      readOnly: true
    - mountPath: /etc/kubernetes/audit-policy/policy.yaml
      name: audit-policy
      readOnly: true
    - mountPath: /etc/kubernetes/audit-logs
      name: audit-logs
      readOnly: false
  hostNetwork: true
  priorityClassName: system-node-critical
  securityContext:
    seccompProfile:
      type: RuntimeDefault
  volumes:
  - name: audit-policy
    hostPath:
      path: /etc/kubernetes/audit-policy/policy.yaml
      type: File
  - name: audit-logs
    hostPath:
      path: /etc/kubernetes/audit-logs
      type: DirectoryOrCreate
  - hostPath:
      path: /etc/ssl/certs
      type: DirectoryOrCreate
    name: ca-certs
  - hostPath:
      path: /etc/ca-certificates
      type: DirectoryOrCreate
    name: etc-ca-certificates
  - hostPath:
      path: /etc/kubernetes/pki
      type: DirectoryOrCreate
    name: k8s-certs
  - hostPath:
      path: /usr/local/share/ca-certificates
      type: DirectoryOrCreate
    name: usr-local-share-ca-certificates
  - hostPath:
      path: /usr/share/ca-certificates
      type: DirectoryOrCreate
    name: usr-share-ca-certificates
  - hostPath:
      path: /etc/kubernetes/encryption
      type: DirectoryOrCreate
    name: enc-conf
status: {}

</code></pre>
<p>I double checked conf and path, The strange part is kube-apiserver is not logging into <code>cat kube-system_kube-apiserver-XX-kube-master-1_c514a6246640287303eb130a626552f2/kube-apiserver/5.log</code>, there are no logs in any of the files under <code>kube-system_kube-apiserver-XX-kube-master-1_c514a6246640287303eb130a626552f2</code></p>
<pre><code>crictl ps
</code></pre>
<p>Doesn't show a container running with name <em>api</em>, so not able to get any logs.</p>
<p>From <code>journalctl -u kubelet | grep &quot;policy&quot;</code></p>
<pre><code>Sep 22 15:25:32 i11806-kube-master-1 kubelet[1187]: E0922 15:25:32.312777    1187 pod_workers.go:951] &quot;Error syncing pod, skipping&quot; err=&quot;failed to \&quot;StartContainer\&quot; for \&quot;kube-apiserver\&quot; with RunContainerError: \&quot;failed to create containerd task: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: exec: \\\&quot;--audit-policy-file=/etc/kubernetes/audit-policy/policy.yaml\\\&quot;: stat --audit-policy-file=/etc/kubernetes/audit-policy/policy.yaml: no such file or directory: unknown\&quot;&quot; pod=&quot;kube-system/kube-apiserver-i11806-kube-master-1&quot; podUID=43aa05fda9c680dd3c0c77b8e1c95dac

</code></pre>
<p>Any help is appreciated.</p>

Rahul Sharma
<p>Your <code>Pod</code> definition for <code>kube-apiserver</code> is incorrect. Take a look:</p>
<ul>
<li>Incorrect:</li>
</ul>
<pre class="lang-yaml prettyprint-override"><code>spec: 
  containers:
  - command:
    - --audit-policy-file=/etc/kubernetes/audit-policy/policy.yaml
    - --audit-log-path=/etc/kubernetes/audit-logs/audit.log
    - --audit-log-maxsize=7
    - kube-apiserver # &lt;-- WRONG
    - --encryption-provider-config=/etc/kubernetes/encryption/encryptionconfiguration.yaml
    - --advertise-address=192.168.0.XX
</code></pre>
<ul>
<li>Correct:</li>
</ul>
<pre class="lang-yaml prettyprint-override"><code>  containers:
  - command:
    - kube-apiserver # &lt;-- CORRECT
    - --audit-policy-file=/etc/kubernetes/audit-policy/policy.yaml
    - --audit-log-path=/etc/kubernetes/audit-logs/audit.log
    - --audit-log-maxsize=7
    - --encryption-provider-config=/etc/kubernetes/encryption/encryptionconfiguration.yaml
    - --advertise-address=192.168.0.XX
</code></pre>
<p>You have put the parameters for <code>kube-apiserver</code> before the actual command to run the <code>kubeapi-server</code>.</p>
