<p>For years we have built base PHP-FPM container images locally with code like this to include Oracle DB support:</p>
<pre><code>ARG PHP_VERSION=7.4
ARG PHP_TYPE=fpm

FROM php:${PHP_VERSION}-${PHP_TYPE}

ENV LD_LIBRARY_PATH /usr/local/instantclient
ENV ORACLE_BASE     /usr/local/instantclient
ENV ORACLE_HOME     /usr/local/instantclient
ENV TNS_ADMIN       /etc/oracle

COPY oracle /etc/oracle

RUN echo 'instantclient,/usr/local/instantclient' | pecl install oci8-${OCI8_VERSION} \
    &amp;&amp; docker-php-ext-configure oci8 --with-oci8=instantclient,/usr/local/instantclient \
    &amp;&amp; docker-php-ext-install oci8 \
    &amp;&amp; docker-php-ext-configure pdo_oci --with-pdo-oci=instantclient,/usr/local/instantclient \
    &amp;&amp; docker-php-ext-install pdo_oci \
    &amp;&amp; rm -rf /tmp/pear
</code></pre>
<p>From this image we build application specific images that are deployed to a Kubernetes cluster and the <code>TNS_ADMIN</code> variable and value have persisted without issue.</p>
<p>We recently changed how the images are built (using Kaniko and GitLab CI instead of building them locally) and found that now when the image is deployed to the Kubernetes cluster (via Helm) the <code>TNS_ADMIN</code> variable is now missing (not just a blank value, the entire variable). Another change made was how the Oracle pieces are installed (using <a href="https://github.com/mlocati/docker-php-extension-installer" rel="nofollow noreferrer">docker-php-extension-installer</a>), so the pertinent Dockerfile code looks like this now:</p>
<pre><code>ADD https://github.com/mlocati/docker-php-extension-installer/releases/latest/download/install-php-extensions /usr/local/bin/
RUN chmod +x /usr/local/bin/install-php-extensions &amp;&amp; \
    install-php-extensions oci8 pdo_oci

# Oracle client config
ENV TNS_ADMIN=/etc/oracle
COPY php.cli/oracle /etc/oracle
</code></pre>
<p>And, here is the GitLab CI Kaniko related code to build the application specific images (only the <code>$PHP_TYPE</code> applies to the image in question):</p>
<pre><code>- |
      LOCAL_REPOSITORY=${CI_REGISTRY}/&lt;internal namespace path&gt;/$REPOSITORY
      # Build config.json for credentials
      echo &quot;{\&quot;auths\&quot;:{\&quot;${CI_REGISTRY}\&quot;:{\&quot;auth\&quot;:\&quot;$(printf &quot;%s:%s&quot; &quot;${CI_REGISTRY_USER}&quot; &quot;${CI_REGISTRY_PASSWORD}&quot; | base64 | tr -d '\n')\&quot;}}}&quot; &gt; /kaniko/.docker/config.json
      /kaniko/executor --context $CI_PROJECT_DIR --dockerfile $CI_PROJECT_DIR/$DOCKER_FILE_PATH/Dockerfile --build-arg PHP_VERSION=$PHP_VERSION --build-arg PHP_TYPE=$PHP_TYPE --build-arg PHPUNIT_VERSION=$PHPUNIT_VERSION --build-arg PHPCS_VERSION=$PHPCS_VERSION --build-arg PHPCSFIXER_VERSION=$PHPCSFIXER_VERSION --destination $LOCAL_REPOSITORY:$PHP_VERSION-$TAG_NAME
</code></pre>
<p>Thinking this was possibly due to how Kaniko works, or the changes to the Oracle install process, we pulled the base image and application image separately and ran them with a bash shell. When pulled locally, the <code>TNS_ADMIN</code> variable is present. This suggests whatever is occurring is happening once Helm deploys it to the cluster.</p>
<p>What is vexing is on the surface neither of the changes we made should affect the setting of an environment variable in this manner in the image, but those were the only changes made that coincide with the issue arising. So, the issue seems to be when deploying the image to our cluster. <em>This process itself has not changed at all</em>. The Helm chart has not changed, which indicates it is not part of this issue; that being said, the issue occurs when Helm deploys the chart that uses the image.</p>
<p>Has anyone else seen something like this, or have any ideas where to center our search for answers?</p>

Bryan Powell
<p>Well, our issue was one that is probably endemic to many people running applications in Kubernetes: our image pull policy for the Helm deployment was set to <code>IfNotPresent</code> and a cached image without the ENV value set was being used (the image was built using a Dockerfile that did not set <code>TNS_ADMIN</code>). We have a lot of moving parts in our process and made multiple changes that were not seen due to this.</p>
<p>I am of course chastened by this explanation and so I will offer the advice to always make sure you are pulling a fresh image as the first step in troubleshooting issues with Kubernetes/Helm deployments.</p>
