<p>I have an OpenShift origin (OKD) environment setup on AWS. I am trying to deploy <code>jenkins-persistent</code> image to bring up <code>jenkins</code> pod in an openshift project. For persistent storage, I am using Persistent Volume and Persistent Volume claims based on <a href="https://kubernetes.io/docs/tasks/configure-pod-container/configure-persistent-volume-storage/" rel="nofollow noreferrer">Kubernetes documentation</a>. </p>

<pre><code>[root@ip-10-0-0-7 BookStore]# oc get pv
NAME           CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS    CLAIM               STORAGECLASS   REASON    AGE
jenkins        5Gi        RWO            Retain           Bound     cicd/jenkins        manual                   1h
jenkins-data   5Gi        RWO            Retain           Bound     cicd/jenkins-data   manual                   3h
[root@ip-10-0-0-7 BookStore]# oc get pvc
NAME           STATUS    VOLUME         CAPACITY   ACCESS MODES   STORAGECLASS   AGE
jenkins        Bound     jenkins        5Gi        RWO            manual         1h
jenkins-data   Bound     jenkins-data   5Gi        RWO            manual         3h
</code></pre>

<p>I gave access permissions based on <a href="https://docs.openshift.com/enterprise/3.0/admin_guide/manage_scc.html#grant-access-to-the-privileged-scc" rel="nofollow noreferrer">OpenShift docs</a>. Now, whenever I try to deploy this image, I see error <strong>Copying Jenkins configuration to /var/lib/jenkins ...
cp: cannot create regular file ‘/var/lib/jenkins/config.xml’: Permission denied</strong></p>

<p>Error Log:</p>

<pre><code>OPENSHIFT_JENKINS_JVM_ARCH='', CONTAINER_MEMORY_IN_MB='512', using /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.191.b12-1.el7_6.i386/jre/bin/java and /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.191.b12-1.el7_6.i386/bin/javac
Picked up JAVA_TOOL_OPTIONS: -XX:+UnlockExperimentalVMOptions -XX:+UseCGroupMemoryLimitForHeap -Dsun.zip.disableMemoryMapping=true
mkdir: cannot create directory ‘/var/lib/jenkins/logs’: Permission denied
/usr/libexec/s2i/run: line 306: /var/lib/jenkins/image-version: Permission denied
Generating kubernetes-plugin configuration (/opt/openshift/configuration/config.xml.tpl) ...
Generating administrative monitor configuration (/opt/openshift/configuration/config.xml.tpl) ...
Generating kubernetes-plugin credentials (/var/lib/jenkins/credentials.xml.tpl) ...
Copying Jenkins configuration to /var/lib/jenkins ...
cp: cannot create regular file ‘/var/lib/jenkins/config.xml’: Permission denied
cp: cannot create regular file ‘/var/lib/jenkins/config.xml.tpl’: Permission denied
cp: cannot create regular file ‘/var/lib/jenkins/credentials.xml’: Permission denied
cp: cannot create regular file ‘/var/lib/jenkins/credentials.xml.tpl’: Permission denied
cp: cannot create directory ‘/var/lib/jenkins/init.groovy.d’: Permission denied
cp: cannot create regular file ‘/var/lib/jenkins/jenkins.CLI.xml’: Permission denied
cp: cannot create directory ‘/var/lib/jenkins/jobs’: Permission denied
cp: cannot create regular file ‘/var/lib/jenkins/org.jenkinsci.main.modules.sshd.SSHD.xml’: Permission denied
cp: cannot create directory ‘/var/lib/jenkins/users’: Permission denied
mkdir: cannot create directory ‘/var/lib/jenkins/plugins’: Permission denied
Copying 104 files to /var/lib/jenkins ...
cp: cannot create regular file ‘/var/lib/jenkins/plugins’: Permission denied
cp: cannot create regular file ‘/var/lib/jenkins/plugins’: Permission denied
cp: cannot create regular file ‘/var/lib/jenkins/plugins’: Permission denied
cp: cannot create regular file ‘/var/lib/jenkins/plugins’: Permission denied
cp: cannot create regular file ‘/var/lib/jenkins/plugins’: Permission denied
cp: cannot create regular file ‘/var/lib/jenkins/plugins’: Permission denied
cp: cannot create regular file ‘/var/lib/jenkins/plugins’: Permission denied
cp: cannot create regular file ‘/var/lib/jenkins/plugins’: Permission denied
cp: cannot create regular file ‘/var/lib/jenkins/plugins’: Permission denied
cp: cannot create regular file ‘/var/lib/jenkins/plugins’: Permission denied
cp: cannot create regular file ‘/var/lib/jenkins/plugins’: Permission denied
cp: cannot create regular file ‘/var/lib/jenkins/plugins’: Permission denied
cp: cannot create regular file ‘/var/lib/jenkins/plugins’: Permission denied
cp: cannot create regular file ‘/var/lib/jenkins/plugins’: Permission denied
cp: cannot create regular file ‘/var/lib/jenkins/plugins’: Permission denied
cp: cannot create regular file ‘/var/lib/jenkins/plugins’: Permission denied
cp: cannot create regular file ‘/var/lib/jenkins/plugins’: Permission denied
cp: cannot create regular file ‘/var/lib/jenkins/plugins’: Permission denied
cp: cannot create regular file ‘/var/lib/jenkins/plugins’: Permission denied
cp: cannot create regular file ‘/var/lib/jenkins/plugins’: Permission denied
cp: cannot create regular file ‘/var/lib/jenkins/plugins’: Permission denied
cp: cannot create regular file ‘/var/lib/jenkins/plugins’: Permission denied
cp: cannot create regular file ‘/var/lib/jenkins/plugins’: Permission denied
cp: cannot create regular file ‘/var/lib/jenkins/plugins’: Permission denied
cp: cannot create regular file ‘/var/lib/jenkins/plugins’: Permission denied
cp: cannot create regular file ‘/var/lib/jenkins/plugins’: Permission denied
cp: cannot create regular file ‘/var/lib/jenkins/plugins’: Permission denied
cp: cannot create regular file ‘/var/lib/jenkins/plugins’: Permission denied
cp: cannot create regular file ‘/var/lib/jenkins/plugins’: Permission denied
cp: cannot create regular file ‘/var/lib/jenkins/plugins’: Permission denied
cp: cannot create regular file ‘/var/lib/jenkins/plugins’: Permission denied
cp: cannot create regular file ‘/var/lib/jenkins/plugins’: Permission denied
cp: cannot create regular file ‘/var/lib/jenkins/plugins’: Permission denied
cp: cannot create regular file ‘/var/lib/jenkins/plugins’: Permission denied
cp: cannot create regular file ‘/var/lib/jenkins/plugins’: Permission denied
cp: cannot create regular file ‘/var/lib/jenkins/plugins’: Permission denied
cp: cannot create regular file ‘/var/lib/jenkins/plugins’: Permission denied
cp: cannot create regular file ‘/var/lib/jenkins/plugins’: Permission denied
cp: cannot create regular file ‘/var/lib/jenkins/plugins’: Permission denied
cp: cannot create regular file ‘/var/lib/jenkins/plugins’: Permission denied
cp: cannot create regular file ‘/var/lib/jenkins/plugins’: Permission denied
cp: cannot create regular file ‘/var/lib/jenkins/plugins’: Permission denied
cp: cannot create regular file ‘/var/lib/jenkins/plugins’: Permission denied
cp: cannot create regular file ‘/var/lib/jenkins/plugins’: Permission denied
cp: cannot create regular file ‘/var/lib/jenkins/plugins’: Permission denied
cp: cannot create regular file ‘/var/lib/jenkins/plugins’: Permission denied
cp: cannot create regular file ‘/var/lib/jenkins/plugins’: Permission denied
cp: cannot create regular file ‘/var/lib/jenkins/plugins’: Permission denied
cp: cannot create regular file ‘/var/lib/jenkins/plugins’: Permission denied
cp: cannot create regular file ‘/var/lib/jenkins/plugins’: Permission denied
cp: cannot create regular file ‘/var/lib/jenkins/plugins’: Permission denied
cp: cannot create regular file ‘/var/lib/jenkins/plugins’: Permission denied
cp: cannot create regular file ‘/var/lib/jenkins/plugins’: Permission denied
cp: cannot create regular file ‘/var/lib/jenkins/plugins’: Permission denied
cp: cannot create regular file ‘/var/lib/jenkins/plugins’: Permission denied
cp: cannot create regular file ‘/var/lib/jenkins/plugins’: Permission denied
cp: cannot create regular file ‘/var/lib/jenkins/plugins’: Permission denied
cp: cannot create regular file ‘/var/lib/jenkins/plugins’: Permission denied
cp: cannot create regular file ‘/var/lib/jenkins/plugins’: Permission denied
cp: cannot create regular file ‘/var/lib/jenkins/plugins’: Permission denied
cp: cannot create regular file ‘/var/lib/jenkins/plugins’: Permission denied
cp: cannot create regular file ‘/var/lib/jenkins/plugins’: Permission denied
cp: cannot create regular file ‘/var/lib/jenkins/plugins’: Permission denied
cp: cannot create regular file ‘/var/lib/jenkins/plugins’: Permission denied
cp: cannot create regular file ‘/var/lib/jenkins/plugins’: Permission denied
cp: cannot create regular file ‘/var/lib/jenkins/plugins’: Permission denied
cp: cannot create regular file ‘/var/lib/jenkins/plugins’: Permission denied
cp: cannot create regular file ‘/var/lib/jenkins/plugins’: Permission denied
cp: cannot create regular file ‘/var/lib/jenkins/plugins’: Permission denied
cp: cannot create regular file ‘/var/lib/jenkins/plugins’: Permission denied
cp: cannot create regular file ‘/var/lib/jenkins/plugins’: Permission denied
cp: cannot create regular file ‘/var/lib/jenkins/plugins’: Permission denied
cp: cannot create regular file ‘/var/lib/jenkins/plugins’: Permission denied
cp: cannot create regular file ‘/var/lib/jenkins/plugins’: Permission denied
cp: cannot create regular file ‘/var/lib/jenkins/plugins’: Permission denied
cp: cannot create regular file ‘/var/lib/jenkins/plugins’: Permission denied
cp: cannot create regular file ‘/var/lib/jenkins/plugins’: Permission denied
cp: cannot create regular file ‘/var/lib/jenkins/plugins’: Permission denied
cp: cannot create regular file ‘/var/lib/jenkins/plugins’: Permission denied
cp: cannot create regular file ‘/var/lib/jenkins/plugins’: Permission denied
cp: cannot create regular file ‘/var/lib/jenkins/plugins’: Permission denied
cp: cannot create regular file ‘/var/lib/jenkins/plugins’: Permission denied
cp: cannot create regular file ‘/var/lib/jenkins/plugins’: Permission denied
cp: cannot create regular file ‘/var/lib/jenkins/plugins’: Permission denied
cp: cannot create regular file ‘/var/lib/jenkins/plugins’: Permission denied
cp: cannot create regular file ‘/var/lib/jenkins/plugins’: Permission denied
cp: cannot create regular file ‘/var/lib/jenkins/plugins’: Permission denied
cp: cannot create regular file ‘/var/lib/jenkins/plugins’: Permission denied
cp: cannot create regular file ‘/var/lib/jenkins/plugins’: Permission denied
cp: cannot create regular file ‘/var/lib/jenkins/plugins’: Permission denied
cp: cannot create regular file ‘/var/lib/jenkins/plugins’: Permission denied
cp: cannot create regular file ‘/var/lib/jenkins/plugins’: Permission denied
cp: cannot create regular file ‘/var/lib/jenkins/plugins’: Permission denied
cp: cannot create regular file ‘/var/lib/jenkins/plugins’: Permission denied
cp: cannot create regular file ‘/var/lib/jenkins/plugins’: Permission denied
cp: cannot create regular file ‘/var/lib/jenkins/plugins’: Permission denied
cp: cannot create regular file ‘/var/lib/jenkins/plugins’: Permission denied
cp: cannot create regular file ‘/var/lib/jenkins/plugins’: Permission denied
cp: cannot create regular file ‘/var/lib/jenkins/plugins’: Permission denied
cp: cannot create regular file ‘/var/lib/jenkins/plugins’: Permission denied
cp: cannot create regular file ‘/var/lib/jenkins/plugins’: Permission denied
cp: cannot create regular file ‘/var/lib/jenkins/plugins’: Permission denied
cp: cannot create regular file ‘/var/lib/jenkins/plugins’: Permission denied
cp: cannot create regular file ‘/var/lib/jenkins/plugins’: Permission denied
Creating initial Jenkins 'admin' user ...
ls: cannot access /var/lib/jenkins/users: No such file or directory
ls: cannot access /var/lib/jenkins/users/: No such file or directory
sed: can't read /var/lib/jenkins/users//config.xml: No such file or directory
/usr/libexec/s2i/run: line 23: /var/lib/jenkins/password: Permission denied
touch: cannot touch ‘/var/lib/jenkins/configured’: Permission denied
Administrative monitors that contact the update center will remain active
Migrating slave image configuration to current version tag ...
sed: can't read /var/lib/jenkins/config.xml: No such file or directory
sed: can't read /var/lib/jenkins/config.xml: No such file or directory
sed: can't read /var/lib/jenkins/config.xml: No such file or directory
sed: can't read /var/lib/jenkins/config.xml: No such file or directory
sed: can't read /var/lib/jenkins/config.xml: No such file or directory
sed: can't read /var/lib/jenkins/config.xml: No such file or directory
sed: can't read /var/lib/jenkins/config.xml: No such file or directory
sed: can't read /var/lib/jenkins/config.xml: No such file or directory
+ exec java -XX:+UseParallelGC -XX:MinHeapFreeRatio=5 -XX:MaxHeapFreeRatio=10 -XX:GCTimeRatio=4 -XX:AdaptiveSizePolicyWeight=90 -Xmx256m -Dfile.encoding=UTF8 -Djavamelody.displayed-counters=log,error -Duser.home=/var/lib/jenkins -Djavamelody.application-name=JENKINS -jar /usr/lib/jenkins/jenkins.war
Picked up JAVA_TOOL_OPTIONS: -XX:+UnlockExperimentalVMOptions -XX:+UseCGroupMemoryLimitForHeap -Dsun.zip.disableMemoryMapping=true
OpenJDK Server VM warning: If the number of processors is expected to increase from one, then you should configure the number of parallel GC threads appropriately using -XX:ParallelGCThreads=N
Running from: /usr/lib/jenkins/jenkins.war
webroot: EnvVars.masterEnvVars.get("JENKINS_HOME")
Mar 25, 2019 6:04:16 AM org.eclipse.jetty.util.log.Log initialized
INFO: Logging initialized @971ms to org.eclipse.jetty.util.log.JavaUtilLog
Mar 25, 2019 6:04:16 AM winstone.Logger logInternal
INFO: Beginning extraction from war file
Mar 25, 2019 6:04:16 AM winstone.Logger logInternal
INFO: Jetty shutdown successfully
java.io.FileNotFoundException: /var/lib/jenkins/war/META-INF/MANIFEST.MF (No such file or directory)
    at java.io.FileOutputStream.open0(Native Method)
    at java.io.FileOutputStream.open(FileOutputStream.java:270)
    at java.io.FileOutputStream.&lt;init&gt;(FileOutputStream.java:213)
    at java.io.FileOutputStream.&lt;init&gt;(FileOutputStream.java:162)
    at winstone.HostConfiguration.getWebRoot(HostConfiguration.java:278)
    at winstone.HostConfiguration.&lt;init&gt;(HostConfiguration.java:81)
    at winstone.HostGroup.initHost(HostGroup.java:66)
    at winstone.HostGroup.&lt;init&gt;(HostGroup.java:45)
    at winstone.Launcher.&lt;init&gt;(Launcher.java:169)
    at winstone.Launcher.main(Launcher.java:354)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at Main._main(Main.java:344)
    at Main.main(Main.java:160)
Mar 25, 2019 6:04:16 AM winstone.Logger logInternal
SEVERE: Container startup failed
java.io.FileNotFoundException: /var/lib/jenkins/war/META-INF/MANIFEST.MF (No such file or directory)
    at java.io.FileOutputStream.open0(Native Method)
    at java.io.FileOutputStream.open(FileOutputStream.java:270)
    at java.io.FileOutputStream.&lt;init&gt;(FileOutputStream.java:213)
    at java.io.FileOutputStream.&lt;init&gt;(FileOutputStream.java:162)
    at winstone.HostConfiguration.getWebRoot(HostConfiguration.java:278)
    at winstone.HostConfiguration.&lt;init&gt;(HostConfiguration.java:81)
    at winstone.HostGroup.initHost(HostGroup.java:66)
    at winstone.HostGroup.&lt;init&gt;(HostGroup.java:45)
    at winstone.Launcher.&lt;init&gt;(Launcher.java:169)
    at winstone.Launcher.main(Launcher.java:354)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at Main._main(Main.java:344)
    at Main.main(Main.java:160)
</code></pre>

Pavan Jadda
<p>Finally, I figured out the issue. This was due to the fact the Persistent volumes created are not accessible as they present on AWS and not created through OpenShift commands. Execute step 1 and 2 before deploying the cluster and rest of the steps after that</p>

<ol>
<li>Add following content to inventory or hosts files. Make sure <code>openshift_clusterid</code> is unique for simplicity you can use same name as cloud formation stack name</li>
</ol>

<pre><code>openshift_clusterid=egis
# AWS (Using API Credentials)
openshift_cloudprovider_kind=aws
openshift_cloudprovider_aws_access_key=&lt;key&gt;
openshift_cloudprovider_aws_secret_key=&lt;key secret&gt;
openshift_master_bootstrap_auto_approve=true
</code></pre>

<p>it will create and installs required config files</p>

<ol start="2">
<li>Add Cluster Id (Stack Name) as a tag to all instances(nodes), VPC, Security groups and Subnets</li>
</ol>

<p><strong>Key</strong> = kubernetes.io/cluster/egis
<strong>Value</strong> = owned</p>

<ol start="3">
<li>Now create persistent volume using following format and replace <code>volumeID</code> with available and free volume id from ec2 volumes
<strong>jenkins-persistent-volume.yml</strong></li>
</ol>

<pre><code>apiVersion: "v1"
kind: "PersistentVolume"
metadata:
  name: "jenkins"
spec:
  storageClassName: manual
  capacity:
    storage: "8Gi"
  accessModes:
    - "ReadWriteOnce"
  awsElasticBlockStore:
    fsType: "ext4"
    volumeID: "&lt;&gt;"

</code></pre>

<ol start="4">
<li>Create a persistent volume claim file with the following template
<strong>jenkins-persistent-volume-claim.yml</strong></li>
</ol>

<pre><code>apiVersion: "v1"
kind: "PersistentVolumeClaim"
metadata:
  name: "jenkins"
spec:
  storageClassName: manual
  accessModes:
    - "ReadWriteOnce"
  resources:
    requests:
      storage: "8Gi"
  volumeName: "jenkins"
</code></pre>

<ol start="5">
<li>Create persistent volume on openshift master using following command</li>
</ol>

<pre><code>oc create -f jenkins-persistent-volume.yml
</code></pre>

<ol start="6">
<li>Create persistent volume claim on openshift master using the following command</li>
</ol>

<pre><code>oc create -f jenkins-persistent-volume-claim.yml
</code></pre>
