<p>We have python services running in pods in a kubernetes cluster. The services are setup to receive their log-level from an environment variable. Those env vars are set during the deployment of the service in a gitlab pipeline. For debugging purposes I want to be able to just change the env var on a single pod and restart it, without having to redeploy the service from gitlab.</p>
<p>Before we moved to kubernetes, we were running our containers in rancher, where the described change was very easy to do in the GUI. Change the env var -&gt; hit update -&gt; container restarts automatically.</p>
<p>I found <a href="https://medium.com/@tszumowski/how-to-inject-environment-variables-into-kubernetes-pods-after-deployment-489306fff1a8" rel="nofollow noreferrer">this article</a> that suggest to change the replica set using a command like</p>
<pre class="lang-sh prettyprint-override"><code>kubectl set env rs [REPLICASET_NAME] [ENV_VAR]=[VALUE]
</code></pre>
<p>And then terminating the pod, after which it will be recreated with the env var set accordingly.</p>
<p>But it also states to</p>
<blockquote>
<p>Never do it on a production system.</p>
<p>Never even do this on a dev environment without taking care in how it may impact your deployment workflow.</p>
</blockquote>
<p>Is that the only / best way to achieve my goal of quickly changing an env var in a running pod for debug purposes?</p>

<blockquote>
  <p>Is that the only / best way to achieve my goal of quickly changing an
  env var in a running pod for debug purposes?</p>
</blockquote>

<p>Short answer: Yes.</p>

<p>Long answer: I've never used or read up on Rancher, but I suspect that it was also changing the ReplicaSet or Deployment template env var, which triggered a Pod update. It's really the only way to change an env var in a Pod. You can't change the env vars on a running container or a running Pod. You can't do that in Docker containers, and you can't do it in Kubernetes, so I assume that you can't do it in Rancher. You can only restart a Pod with a different spec. </p>

<p>Why?</p>

<p>Because containers are just processes running on the host machine. Once the process is started, it's not possible to change a process's environment without <a href="https://unix.stackexchange.com/questions/38205/change-environment-of-a-running-process">resorting to nasty hacks</a>.</p>

<p>If you're just concerned about the warnings that state to not do this in dev or prod, I would say that the same warnings apply to the Rancher workflow you described, so if you were willing to take the risks there, it won't be any different here.</p>
