<p>I am running <code>Kafka Streams 3.1.0</code> on <code>AWS OCP</code> cluster, and I am facing this error during restart of the pod:</p>
<pre><code>10:33:18,529 [INFO ] Loaded Kafka Streams properties {topology.optimization=all, processing.guarantee=at_least_once, bootstrap.servers=PLAINTEXT://app-kafka-headless.app.svc.cluster.local:9092, state.dir=/var/data/state-store, metrics.recording.level=INFO, consumer.auto.offset.reset=earliest, cache.max.bytes.buffering=10485760, producer.compression.type=lz4, num.stream.threads=3, application.id=AppProcessor}
10:33:18,572 [ERROR] Error changing permissions for the directory /var/data/state-store
java.nio.file.FileSystemException: /var/data/state-store: Operation not permitted
    at java.base/sun.nio.fs.UnixException.translateToIOException(Unknown Source)
    at java.base/sun.nio.fs.UnixException.rethrowAsIOException(Unknown Source)
    at java.base/sun.nio.fs.UnixException.rethrowAsIOException(Unknown Source)
    at java.base/sun.nio.fs.UnixFileAttributeViews$Posix.setMode(Unknown Source)
    at java.base/sun.nio.fs.UnixFileAttributeViews$Posix.setPermissions(Unknown Source)
    at java.base/java.nio.file.Files.setPosixFilePermissions(Unknown Source)
    at org.apache.kafka.streams.processor.internals.StateDirectory.configurePermissions(StateDirectory.java:154)
    at org.apache.kafka.streams.processor.internals.StateDirectory.&lt;init&gt;(StateDirectory.java:144)
    at org.apache.kafka.streams.KafkaStreams.&lt;init&gt;(KafkaStreams.java:867)
    at org.apache.kafka.streams.KafkaStreams.&lt;init&gt;(KafkaStreams.java:851)
    at org.apache.kafka.streams.KafkaStreams.&lt;init&gt;(KafkaStreams.java:821)
    at org.apache.kafka.streams.KafkaStreams.&lt;init&gt;(KafkaStreams.java:733)
    at com.xyz.app.kafka.streams.AbstractProcessing.run(AbstractProcessing.java:54)
    at com.xyz.app.kafka.streams.AppProcessor.main(AppProcessor.java:97)
10:33:18,964 [INFO ] Topologies:
   Sub-topology: 0
    Source: app-stream (topics: [app-app-stream])
      --&gt; KSTREAM-AGGREGATE-0000000002
    Processor: KSTREAM-AGGREGATE-0000000002 (stores: [KSTREAM-AGGREGATE-STATE-STORE-0000000001])
      --&gt; none
      &lt;-- app-stream
10:33:18,991 [WARN ] stream-thread [main] Failed to delete state store directory of /var/data/state-store/AppProcessor for it is not empty
</code></pre>
<p>On OCP cluster, the user running the app is provided by the cluster, and the state store is provided by an persistent volume (allowing pod to restart on same context), so the <code>/var/data/state-store/</code> folder have following permissions <code>drwxrwsr-x. (u:root g:1001030000)</code> :</p>
<pre><code>1001030000@app-processor-0:/$ ls -al /var/data/state-store/
total 24
drwxrwsr-x. 4 root       1001030000  4096 Mar 21 10:43 .
drwxr-xr-x. 3 root       root          25 Mar 23 11:04 ..
drwxr-x---. 2 1001030000 1001030000  4096 Mar 23 11:04 AppProcessor
drwxrws---. 2 root       1001030000 16384 Mar 21 10:36 lost+found

1001030000@app-processor-0:/$ chmod 750 /var/data/state-store/
chmod: changing permissions of '/var/data/state-store/': Operation not permitted
</code></pre>
<p>POD manifest relevant parts are:</p>
<pre><code>spec:
  containers:
  - name: app-processor
     volumeMounts:
     - mountPath: /var/data/state-store
       name: data
     securityContext:
       capabilities:
         drop:
         - KILL
         - MKNOD
         - SETGID
         - SETUID
  securityContext:
    fsGroup: 1001030000
    runAsUser: 1001030000
    seLinuxOptions:
      level: s0:c32,c19
  volumes:
  - name: data
    persistentVolumeClaim:
      claimName: data-app-processor-0
</code></pre>
<p>How to handle that ?
Should we use a <code>subPath</code> on <code>volumeMount</code> ?</p>
<p>Thanks for your insights.</p>

<p>As suggested, the fix I found was to set a <code>subPath</code> below the mountPath:</p>
<p>Here is the relevant part of helm template used:</p>
<pre><code>spec:
  containers:
  - name: app-processor
    volumeMounts:
    - name: data
      mountPath: {{ dir .Values.streams.state_dir | default &quot;/var/data/&quot; }}
      subPath: {{ base .Values.streams.state_dir | default &quot;state-store&quot; }}
</code></pre>
<p>Where <code>.Values.streams.state_dir</code> is mapped to stream property <code>state.dir</code>.
Note this value is mandatory, and must be initialized in the values.</p>
<p>In that case the <code>state-store</code> directory is created by <code>securityContext.runAsUser</code> user, instead of root, so the <code>org.apache.kafka.streams.processor.internals.StateDirectory</code> class can enforce the permissions.</p>
