<p>I am deploying my application in a read only kubernetes cluster, so I am using volumes and volumeMounts for tmp folder for apache server. Upon start of apache server within read only pod, I am getting this error:</p>
<pre><code>chown: changing ownership of '/var/lock/apache2.fm2cgWmnxk': Operation not permitted
</code></pre>
<p>I came across this issue <a href="https://stackoverflow.com/questions/43544370/kubernetes-how-to-set-volumemount-user-group-and-file-permissions">Kubernetes: how to set VolumeMount user group and file permissions</a> and tried using SecurityContext.fsGroup but still getting same issue.</p>
<p>Here is my deployment.yaml for reference:</p>
<pre><code>apiVersion: apps/v1
kind: Deployment
metadata:
  name: &amp;DeploymentName abc
spec:
  replicas: 1
  selector:
    matchLabels: &amp;appName
      app: *DeploymentName
  template:
    metadata:
      name: main
      labels:
        &lt;&lt;: *appName
    spec:
      securityContext:
        fsGroup: 2000
        runAsNonRoot: true
        runAsUser: 1000
        runAsGroup: 3000
        fsGroupChangePolicy: &quot;OnRootMismatch&quot;
      volumes:
        - name: var-lock
          emptyDir: {}
      containers:
        - name: *DeploymentName
          image: abc-image
          ports:
            - containerPort: 80
          volumeMounts:
            - mountPath: /var/lock
              name: var-lock
          readinessProbe:
              tcpSocket:
                port: 80
              initialDelaySeconds: 180
              periodSeconds: 60
          livenessProbe:
              tcpSocket:
                port: 80
              initialDelaySeconds: 300
              periodSeconds: 180
          imagePullPolicy: Always
          tty: true
          env:
            - name: POD_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
          envFrom:
            - configMapRef:
                name: *DeploymentName
          resources:
            limits:
              cpu: 1
              memory: 2Gi
            requests:
              cpu: 1
              memory: 2Gi

---
apiVersion: autoscaling/v1
kind: HorizontalPodAutoscaler
metadata:
  name: &amp;hpaName abc
spec:
  maxReplicas: 1
  minReplicas: 1
  scaleTargetRef:
    apiVersion: extensions/v1beta1
    kind: Deployment
    name: *hpaName
  targetCPUUtilizationPercentage: 60
</code></pre>
<p>Any help is appreciated.</p>

<p>Hello, hope you are envoying your Kubernetes journey !</p>
<p>I wanted to try this on my kind (Kubernetes in docker) cluster locally. So this is what I've done:</p>
<p>First I have setup a kind cluster locally with this configuration (info here: <a href="https://kind.sigs.k8s.io/docs/user/quick-start/" rel="nofollow noreferrer">https://kind.sigs.k8s.io/docs/user/quick-start/</a>):</p>
<pre><code>kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
name: so-cluster-1
nodes:
- role: control-plane
  image: kindest/node:v1.23.5
- role: control-plane
  image: kindest/node:v1.23.5
- role: control-plane
  image: kindest/node:v1.23.5
- role: worker
  image: kindest/node:v1.23.5
- role: worker
  image: kindest/node:v1.23.5
- role: worker
  image: kindest/node:v1.23.5
</code></pre>
<p>after this I created my cluster with this command:</p>
<pre><code>kind create cluster --config=config.yaml
</code></pre>
<p>Next, i have created a test namespace (manifest obtained with: kubectl create ns so-tests -o yaml --dry-run):</p>
<pre><code>apiVersion: v1
kind: Namespace
metadata:
  name: so-tests
</code></pre>
<p>From there, i got my environment setted up, so I used your deployment config and replaced the deploymentName, appName and hpaName occurences by &quot;so-71823613&quot; (stack-overflow and you question id), but for the test, I decided to not use the hpa config.</p>
<p>next, since you did not provide the image you are using for apache, I used the dockerhub image httpd:2.4.53 (<a href="https://hub.docker.com/layers/httpd/library/httpd/2.4.53/images/sha256-10ed1591781d9fdbaefaafee77067f12e833c699c84ed4e21706ccbd5229fd0a?context=explore" rel="nofollow noreferrer">https://hub.docker.com/layers/httpd/library/httpd/2.4.53/images/sha256-10ed1591781d9fdbaefaafee77067f12e833c699c84ed4e21706ccbd5229fd0a?context=explore</a>)</p>
<p>again, since i dont have your configmap config, i decided to comment out the part where you get env variables from the configmap.</p>
<p>since the default user in httpd image is &quot;www-data&quot;, I first deployed the pod without any securityContext just to get the id of that user:</p>
<pre><code>❯ k exec -it pod/so-71823613-555d8b454-z5ks5 -- id www-data
uid=33(www-data) gid=33(www-data) groups=33(www-data)
</code></pre>
<p>Once that i knew what was the id of the www-data user, I modified the securityContext. I kept the rest of the configuration (probes, volume etc.) as you configured them, here is the manifest now:</p>
<p>In the configuration file, the runAsUser field specifies that for any Containers in the Pod, all processes run with user ID 33(www-data).
The runAsGroup field specifies the primary group ID of 33 for all processes within any containers of the Pod. If this field is omitted, the primary group ID of the containers will be root(0).
Any files created will also be owned by user 33 and group 33 when runAsGroup is specified.
Since fsGroup field is specified, all processes of the container are also part of the supplementary group ID 33. The owner for volume &quot;/var/lock&quot; and any files created in that volume will be Group ID 33.
...
fsGroupChangePolicy - fsGroupChangePolicy defines behavior for changing ownership and permission of the volume before being exposed inside a Pod. This field only applies to volume types that support fsGroup controlled ownership and permissions. This field has two possible values:</p>
<p>OnRootMismatch: Only change permissions and ownership if permission and ownership of root directory does not match with expected permissions of the volume. This could help shorten the time it takes to change ownership and permission of a volume.
Always: Always change permission and ownership of the volume when volume is mounted.</p>
<p>( description from here: <a href="https://kubernetes.io/docs/tasks/configure-pod-container/security-context/" rel="nofollow noreferrer">https://kubernetes.io/docs/tasks/configure-pod-container/security-context/</a>)</p>
<p>So, once i deployed my configuration using:</p>
<pre><code>kubectl apply -f deployment.yaml
deployment.apps/so-71823613 created
</code></pre>
<p>I got this error:</p>
<pre><code> k logs -f pod/so-71823613-7c5b65df4d-6scg5
AH00558: httpd: Could not reliably determine the server's fully qualified domain name, using 10.244.1.2. Set the 'ServerName' directive globally to suppress this message
(13)Permission denied: AH00072: make_sock: could not bind to address [::]:80
(13)Permission denied: AH00072: make_sock: could not bind to address 0.0.0.0:80
no listening sockets available, shutting down
AH00015: Unable to open logs
</code></pre>
<p>So, first to fix the first line error, I reconnected into the pod to fetch the httpd.conf file with:</p>
<pre><code>k exec -it pod/so-71823613-555d8b454-fgjcs -- cat /usr/local/apache2/conf/httpd.conf &gt; httpd.conf
</code></pre>
<p>once i get the http.conf file, I modified it, by adding:</p>
<pre><code>ServerName localhost:8080
</code></pre>
<p>(cf <a href="https://ixnfo.com/en/solution-ah00558-apache2-could-not-reliably-determine-the-servers-fully-qualified-domain-name.html" rel="nofollow noreferrer">https://ixnfo.com/en/solution-ah00558-apache2-could-not-reliably-determine-the-servers-fully-qualified-domain-name.html</a>)</p>
<p>Then I put the new httpd.conf file into a configmap named &quot;httpconf&quot;, and modified the deployment to mount the configmap into the right place, to replace the first one (here -&gt; &quot;/usr/local/apache2/conf/httpd.conf&quot;) with:</p>
<pre><code>    ...
    volumeMounts:
    ...
      - name: &quot;config&quot;
        mountPath: &quot;/usr/local/apache2/conf/httpd.conf&quot;
        subPath: &quot;httpd.conf&quot;
  volumes:
  ...
    - name: &quot;config&quot;
      configMap:
        name: &quot;httpconf&quot;
  ...



❯ kubectl apply -f configmap.yaml -f deployment.yaml
configmap/httpconf created
deployment.apps/so-71823613 created
</code></pre>
<p>Then i got this error remaining:</p>
<pre><code>(13)Permission denied: AH00072: make_sock: could not bind to address [::]:80
</code></pre>
<p>So, to fix it, I changed to listening port of apache directly into the configmap http.conf file (according to this: <a href="https://askubuntu.com/questions/338218/why-am-i-getting-permission-denied-make-sock-could-not-bind-to-address-when">https://askubuntu.com/questions/338218/why-am-i-getting-permission-denied-make-sock-could-not-bind-to-address-when</a>)</p>
<pre><code>Listen 8080
ServerName localhost:8080
</code></pre>
<p>since I am now exposing the 8080 port, I also modified the probes and the port in consequence:</p>
<pre><code>...
ports:
  - containerPort: 8080
readinessProbe:
    tcpSocket:
      port: 8080
    initialDelaySeconds: 180
    periodSeconds: 60
livenessProbe:
    tcpSocket:
      port: 8080
...
</code></pre>
<p>After reapplying my config I got this new error:</p>
<pre><code>❯ k  logs -f pod/so-71823613-7dd7bdb66d-qtf9t
[Wed Apr 20 05:50:57.863971 2022] [core:error] [pid 1:tid 139771999915328] (13)Permission denied: AH00099: could not create /usr/local/apache2/logs/httpd.pid.KYUI5g
[Wed Apr 20 05:50:57.864061 2022] [core:error] [pid 1:tid 139771999915328] AH00100: httpd: could not log pid to file /usr/local/apache2/logs/httpd.pid
</code></pre>
<p>To fix that issue, i used your workaround with the emptyDir and added this:</p>
<pre><code>    volumeMounts:
    ...
      - mountPath: /usr/local/apache2/logs/
        name: apache2-logs
  volumes:
  ...
    - name: apache2-logs
      emptyDir: {}
</code></pre>
<p>here are the manifests:</p>
<pre><code>apiVersion: apps/v1
kind: Deployment
metadata:
  name: so-71823613
spec:
  replicas: 1
  selector:
    matchLabels:
      app: so-71823613
  template:
    metadata:
      name: main
      labels:
        app: so-71823613
    spec:
      securityContext:
        fsGroup: 33
        runAsNonRoot: true
        runAsUser: 33
        runAsGroup: 33
        fsGroupChangePolicy: &quot;OnRootMismatch&quot;
      containers:
      - name: so-71823613
        image: httpd:2.4.53
        ports:
          - containerPort: 8080
        readinessProbe:
            tcpSocket:
              port: 8080
            initialDelaySeconds: 180
            periodSeconds: 60
        livenessProbe:
            tcpSocket:
              port: 8080
            initialDelaySeconds: 300
            periodSeconds: 180
        imagePullPolicy: Always
        tty: true
        env:
          - name: POD_NAME
            valueFrom:
              fieldRef:
                fieldPath: metadata.name
        # envFrom:
        #   - configMapRef:
        #       name: so-71823613
        resources:
          limits:
            cpu: 1
            memory: 2Gi
          requests:
            cpu: 1
            memory: 2Gi
        volumeMounts:
          - mountPath: /var/lock
            name: var-lock
          - mountPath: /usr/local/apache2/logs/
            name: apache2-logs
          - name: &quot;config&quot;
            mountPath: &quot;/usr/local/apache2/conf/httpd.conf&quot;
            subPath: &quot;httpd.conf&quot;
      volumes:
        - name: var-lock
          emptyDir: {}
        - name: apache2-logs
          emptyDir: {}
        - name: &quot;config&quot;
          configMap:
            name: &quot;httpconf&quot;

---

apiVersion: v1
kind: ConfigMap
metadata:
  name: httpconf
data:
  httpd.conf: |
    ServerRoot &quot;/usr/local/apache2&quot;
    Listen 8080

    LoadModule mpm_event_module modules/mod_mpm_event.so
    LoadModule authn_file_module modules/mod_authn_file.so
    LoadModule authn_core_module modules/mod_authn_core.so
    LoadModule authz_host_module modules/mod_authz_host.so
    LoadModule authz_groupfile_module modules/mod_authz_groupfile.so
    LoadModule authz_user_module modules/mod_authz_user.so
    LoadModule authz_core_module modules/mod_authz_core.so
    LoadModule access_compat_module modules/mod_access_compat.so
    LoadModule auth_basic_module modules/mod_auth_basic.so
    LoadModule reqtimeout_module modules/mod_reqtimeout.so
    LoadModule filter_module modules/mod_filter.so
    LoadModule mime_module modules/mod_mime.so
    LoadModule log_config_module modules/mod_log_config.so
    LoadModule env_module modules/mod_env.so
    LoadModule headers_module modules/mod_headers.so
    LoadModule setenvif_module modules/mod_setenvif.so
    LoadModule version_module modules/mod_version.so
    LoadModule unixd_module modules/mod_unixd.so
    LoadModule status_module modules/mod_status.so
    LoadModule autoindex_module modules/mod_autoindex.so

    &lt;IfModule !mpm_prefork_module&gt;
    &lt;/IfModule&gt;
    &lt;IfModule mpm_prefork_module&gt;
    &lt;/IfModule&gt;

    LoadModule dir_module modules/mod_dir.so
    LoadModule alias_module modules/mod_alias.so

    &lt;IfModule unixd_module&gt;

    User www-data
    Group www-data

    &lt;/IfModule&gt;

    ServerAdmin you@example.com
    ServerName localhost:8080

    &lt;Directory /&gt;
        AllowOverride none
        Require all denied
    &lt;/Directory&gt;

    DocumentRoot &quot;/usr/local/apache2/htdocs&quot;
    &lt;Directory &quot;/usr/local/apache2/htdocs&quot;&gt;
        Options Indexes FollowSymLinks

        AllowOverride None

        Require all granted
    &lt;/Directory&gt;

    &lt;IfModule dir_module&gt;
        DirectoryIndex index.html
    &lt;/IfModule&gt;

    &lt;Files &quot;.ht*&quot;&gt;
        Require all denied
    &lt;/Files&gt;

    ErrorLog /proc/self/fd/2

    LogLevel warn

    &lt;IfModule log_config_module&gt;
 
        LogFormat &quot;%h %l %u %t \&quot;%r\&quot; %&gt;s %b \&quot;%{Referer}i\&quot; \&quot;%{User-Agent}i\&quot;&quot; combined
        LogFormat &quot;%h %l %u %t \&quot;%r\&quot; %&gt;s %b&quot; common

        &lt;IfModule logio_module&gt;
          LogFormat &quot;%h %l %u %t \&quot;%r\&quot; %&gt;s %b \&quot;%{Referer}i\&quot; \&quot;%{User-Agent}i\&quot; %I %O&quot; combinedio
        &lt;/IfModule&gt;

        CustomLog /proc/self/fd/1 common

    &lt;/IfModule&gt;

    &lt;IfModule alias_module&gt;


        ScriptAlias /cgi-bin/ &quot;/usr/local/apache2/cgi-bin/&quot;

    &lt;/IfModule&gt;

    &lt;IfModule cgid_module&gt;
    &lt;/IfModule&gt;

    &lt;Directory &quot;/usr/local/apache2/cgi-bin&quot;&gt;
        AllowOverride None
        Options None
        Require all granted
    &lt;/Directory&gt;

    &lt;IfModule headers_module&gt;

        RequestHeader unset Proxy early
    &lt;/IfModule&gt;

    &lt;IfModule mime_module&gt;

        TypesConfig conf/mime.types

        AddType application/x-compress .Z
        AddType application/x-gzip .gz .tgz

    &lt;/IfModule&gt;

    &lt;IfModule proxy_html_module&gt;
    Include conf/extra/proxy-html.conf
    &lt;/IfModule&gt;

    &lt;IfModule ssl_module&gt;
    SSLRandomSeed startup builtin
    SSLRandomSeed connect builtin
    &lt;/IfModule&gt;



# ---
# apiVersion: autoscaling/v1
# kind: HorizontalPodAutoscaler
# metadata:
#   name: so-71823613
# spec:
#   maxReplicas: 1
#   minReplicas: 1
#   scaleTargetRef:
#     apiVersion: extensions/v1beta1
#     kind: Deployment
#     name: so-71823613
#   targetCPUUtilizationPercentage: 60
</code></pre>
<p>after waiting the initialDelaySeconds of the probes, I finally get my pod up and running correctly:</p>
<pre><code>Every 1.0s: kubectl get po,svc,cm -o wide                                                                                       DESKTOP-6PBJAOK: Wed Apr 20 03:15:02 2022

NAME                              READY   STATUS    RESTARTS   AGE     IP           NODE                   NOMINATED NODE   READINESS GATES
pod/so-71823613-897768549-mcmb4   1/1     Running   0          4m13s   10.244.4.4   so-cluster-1-worker3   &lt;none&gt;           &lt;none&gt;

NAME                         DATA   AGE
configmap/httpconf           1      4m14s
</code></pre>
<p>Bonus:</p>
<p>I then decided to expose the http deployment with a service, here is the manifest (obtained from  &quot;  k expose deployment so-71823613 --port 80 --target-port 8080 --dry-run=client -o yaml&quot;:</p>
<pre><code>apiVersion: v1
kind: Service
metadata:
  name: so-71823613
spec:
  ports:
  - port: 80
    protocol: TCP
    targetPort: 8080
  selector:
    app: so-71823613
</code></pre>
<p>as you can see, I port-forwarded the 8080 pod port to 80 in the service (you can also use an ingress controller to expose the service outside of the cluster )</p>
<p>tried this on my machine:</p>
<pre><code>❯ k port-forward service/so-71823613 8080:80
Forwarding from 127.0.0.1:8080 -&gt; 8080
Forwarding from [::1]:8080 -&gt; 8080
Handling connection for 8080
</code></pre>
<p>and here is the result:</p>
<p><a href="https://i.stack.imgur.com/HWIXE.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/HWIXE.png" alt="enter image description here" /></a></p>
<p>TADA !</p>
<p>To conclude, I tried to reproduce the best i could with your provided information (It was kinda cool), so if this does not work for you, it means that i need more information.
Thank you for your lecture.
bguess.</p>
