<p>I'm running a Nextflow workflow using the Kubernetes executor in a shared cluster that cannot allow privileged containers for security reasons.</p>
<p>When I run my nextflow workflow I get the error that clearly indicates that it's attempting to run my job in privileged mode:</p>
<pre><code>job-controller  Error creating: admission webhook &quot;validation.gatekeeper.sh&quot; denied the request: 
[psp-privileged-container] Privileged container is not allowed: nf-f48d29c6300b8f61af05447de0072d69, 
securityContext: {&quot;privileged&quot;: true}
</code></pre>
<p>The Nextflow documentation suggests that I can set <code>privileged=false</code> in the <code>k8s.securityContext</code>. I've tried two different values to set privileged to <code>false</code> (one based on Nextflow docs, <code>privileged</code>, the other based on Kubernetes docs, <code>allowPrivilegeEscalation</code>). Both still yield the same error.</p>
<pre><code>// Kubernetes config
process.executor = 'k8s'
k8s.namespace = 'mynamespace'
k8s.computeResourceType = 'Job'
k8s.securityContext.privileged = false                // nextflow doc format
k8s.securityContext.allowPrivilegeEscalation = false  // k8s doc format
</code></pre>
<p>I see the following related discussions that seem to suggest that this should work, but I'm having a hard time following these threads, or at least these threads seem to indicate that my <code>nextflow.config</code> is correct.</p>
<ul>
<li><p>[ERROR controller-runtime.controller - Privileged containers are not allowed spec.containers[1].securityContext.privileged #792][1]</p>
</li>
<li><p>[feat: Support arbitrarily setting privileged: true for runner container #1383][2]</p>
</li>
</ul>
<hr />
<p>Update:</p>
<p>When nextflow submits my job, here's what the yaml looks like that Kubernetes receives. It looks like the job security context is empty, but that the pod template at <code>spec.template.spec.containers[0].securityContext.privileged = true</code>.</p>
<p>Noticing that I also tried setting <code>k8s.containers.securityContext.privileged = false</code> in <code>nextflow.config</code> but that was a wrong guess, I don't know how to index into the pod template settings.</p>
<p>It seems like I just need to figure out how to reference the pod template <code>securityContext</code> from <code>nextflow.config</code>.</p>
<pre><code>$ kubectl get job nf-878eb722258681cf0031aeeabe2fb132 -n mynamespace -o yaml
apiVersion: batch/v1
kind: Job
metadata:
  annotations:
    batch.kubernetes.io/job-tracking: &quot;&quot;
  creationTimestamp: &quot;2023-09-12T17:16:58Z&quot;
  generation: 1
  labels:
    nextflow.io/app: nextflow
    nextflow.io/processName: helloWorld1
    nextflow.io/runName: focused_pesquet
    nextflow.io/sessionId: uuid-2a634652-ba40-4905-a9ca-ad0b948df4f0
    nextflow.io/taskName: helloWorld1
  name: nf-878eb722258681cf0031aeeabe2fb132
  namespace: mynamespace
  resourceVersion: &quot;6387865826&quot;
  uid: 6a4ce939-cc01-4298-9d94-84219d750e84
spec:
  backoffLimit: 0
  completionMode: NonIndexed
  completions: 1
  parallelism: 1
  selector:
    matchLabels:
      controller-uid: 6a4ce939-cc01-4298-9d94-84219d750e84
  suspend: false
  template:
    metadata:
      creationTimestamp: null
      labels:
        controller-uid: 6a4ce939-cc01-4298-9d94-84219d750e84
        job-name: nf-878eb722258681cf0031aeeabe2fb132
    spec:
      containers:
      - args:
        - /usr/bin/fusion
        - bash
        - /fusion/s3/mybucket/nextflow/87/8eb722258681cf0031aeeabe2fb132/.command.run
        env:
        - name: FUSION_WORK
          value: /fusion/s3/mybucket/nextflow/87/8eb722258681cf0031aeeabe2fb132
        - name: AWS_S3_ENDPOINT
          value: custom.ceph.endpoint
        - name: FUSION_TAGS
          value: '[.command.*|.exitcode|.fusion.*](nextflow.io/metadata=true),[*](nextflow.io/temporary=true)'
        image: wave.seqera.io/wt/8602f8b269fe/library/ubuntu:latest
        imagePullPolicy: Always
        name: nf-878eb722258681cf0031aeeabe2fb132
        resources:
          limits:
            ephemeral-storage: 1Gi
            memory: 1Gi
          requests:
            cpu: &quot;1&quot;
            ephemeral-storage: 1Gi
            memory: 1Gi
        securityContext:
          privileged: true
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
      dnsPolicy: ClusterFirst
      restartPolicy: Never
      schedulerName: default-scheduler
      securityContext: {}
      serviceAccount: default
      serviceAccountName: default
      terminationGracePeriodSeconds: 30
  ttlSecondsAfterFinished: 604800
status:
  ready: 0
  startTime: &quot;2023-09-12T17:16:58Z&quot;
  uncountedTerminatedPods: {}
``


  [1]: https://github.com/actions/actions-runner-controller/issues/792
  [2]: https://github.com/actions/actions-runner-controller/pull/1383
</code></pre>

David Parks
<p>In the <a href="https://www.nextflow.io/docs/latest/executor.html#k8s-executor" rel="nofollow noreferrer">Nextflow's Kubernetes executor</a>, you have the ability to control the security context of the pods that are created.</p>
<p>You need to correctly specify the security context in such a way that it applies to the correct level within your Kubernetes job specification. It appears that Nextflow is <em>not</em> applying the <code>securityContext</code> to the individual containers within the pods based on the settings in your <code>nextflow.config</code>.</p>
<p>You can try and utilize the <a href="https://www.nextflow.io/docs/latest/process.html#process-pod" rel="nofollow noreferrer"><code>pod</code> directive</a> within your Nextflow process definitions to specify custom pod settings. You would specify the security context within the pod directive using Groovy syntax to create a YAML snippet that represents the correct Kubernetes configuration.</p>
<pre><code>process {
    ...
    
    pod = [
        'spec': [
            'containers': [[
                'name': 'main',
                'securityContext': [
                    'privileged': false
                ]
            ]]
        ]
    ]
}
</code></pre>
<p>You can see the <code>pod</code> directive, used to define a custom pod configuration, using a map of configurations (and not a YAML multi-line string) to represent the YAML configuration for the Kubernetes pod. We specify a <code>securityContext</code> at the pod level with <code>runAsUser: 1000</code> to dictate the user ID that the process will run as (replace <code>1000</code> with an appropriate user ID for your environment).</p>
<p>Within the <code>containers</code> list, we specify a container named &quot;main&quot; and set <code>privileged: false</code> in its <code>securityContext</code> to disable privileged mode for that container.</p>
<p>That configuration should override the defaults applied by Nextflow and allow your job to run without privileged mode.</p>
