<p>I want to automate <code>kubectl</code> and <code>helm</code> commands using Ansible. The target machine is configured properly, so that both works on the cli in a manual shell (e.g. <code>kubectl get nodes</code> or <code>helm list</code>). But when trying to make any API calls like get the server version</p>

<pre><code>- name: List charts
  shell: kubectl version -v=8
</code></pre>

<p>It breaks with a <em>Forbidden</em> error. The verbose logging doesn't give me much more details:</p>

<pre><code>fatal: [127.0.0.1]: FAILED! =&gt; changed=true 
  cmd: kubectl version -v=10
  delta: '0:00:00.072452'
  end: '2020-02-27 15:22:36.227928'
  msg: non-zero return code
  rc: 255
  start: '2020-02-27 15:22:36.155476'
  stderr: |-
    I0227 15:22:36.224517   27321 loader.go:359] Config loaded from file /home/user/.kube/config
    I0227 15:22:36.225211   27321 round_trippers.go:386] curl -k -v -XGET  -H "Accept: application/json, */*" -H "User-Agent: kubectl/v1.11.3 (linux/amd64) kubernetes/a452946" 'https://k8smaster01:6443/version?timeout=32s'
    I0227 15:22:36.225975   27321 round_trippers.go:405] GET https://k8smaster01:6443/version?timeout=32s  in 0 milliseconds
    I0227 15:22:36.225986   27321 round_trippers.go:411] Response Headers:
    I0227 15:22:36.226062   27321 helpers.go:219] Connection error: Get https://k8smaster01:6443/version?timeout=32s: Forbidden
    F0227 15:22:36.226080   27321 helpers.go:119] Unable to connect to the server: Forbidden
  stderr_lines: &lt;omitted&gt;
  stdout: 'Client Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.3", GitCommit:"a4529464e4629c21224b3d52edfe0ea91b072862", GitTreeState:"clean", BuildDate:"2018-09-09T18:02:47Z", GoVersion:"go1.10.3", Compiler:"gc", Platform:"linux/amd64"}'
  stdout_lines: &lt;omitted&gt;
</code></pre>

<p>However, when sending a manual request to those API url like this</p>

<pre><code>- name: Test master connection
  shell: curl -k https://k8smaster01:6443/version?timeout=32s
</code></pre>

<p>It works:</p>

<pre><code>stderr_lines: &lt;omitted&gt;
  stdout: |-
    {
      "major": "1",
      "minor": "11",
      "gitVersion": "v1.11.3",
      "gitCommit": "a4529464e4629c21224b3d52edfe0ea91b072862",
      "gitTreeState": "clean",
      "buildDate": "2018-09-09T17:53:03Z",
      "goVersion": "go1.10.3",
      "compiler": "gc",
      "platform": "linux/amd64"
    }
</code></pre>

<h3>Why API calls with <code>kubectl</code> doesn't work when executed with Ansible?</h3>

<p>I'm behind a proxy server, but <code>k8smaster01</code> is set in <code>no_proxy</code>. Ansible got it, I printed <code>$no_proxy</code> in the task for testing purpose.</p>

<p>For <code>curl</code> I used <code>-k</code> since its a self signed cert from k8s. This sould harm <code>kubectl</code> (which itself works when not running from Ansible). It also doesn't work when calling <code>kubectl --insecure-skip-tls-verify=true get node</code> with Ansible.</p>

<p>I tried to unset the env variables from the proxy (since the proxy is required just for internet access) by setting empty environment variables:</p>

<pre><code>- name: Kubectl test
  become: false
  shell: kubectl get no -v=10
  environment:
    http_proxy:
    https_proxy:
    no_proxy:
    HTTP_PROXY:
    HTTPS_PROXY:
    NO_PROXY:
</code></pre>

<p>This was a bad idea, since <code>curl</code> (which seems to be used inside <code>kubectl</code>) parse this to <code>None</code> and fail. Strangely, <code>kubectl</code> failed with an dns error:</p>

<pre><code>skipped caching discovery info due to Get https://k8smaster01:6443/api?timeout=32s: proxyconnect tcp: dial tcp: lookup None on 127.0.0.53:53: server misbehaving
</code></pre>

<p>Found out that the main problem was that I set <code>NO_PROXY=$no_proxy</code> in <code>/etc/environment</code>, where <code>no_proxy</code> contains the hostname <code>k8smaster01</code>. Since <code>/etc/environment</code> doesn't resolve bash variables, the uppercase <code>NO_PROXY</code> just contains <code>$no_proxy</code> as string. So it was enough to replace <code>NO_PROXY=$no_proxy</code> by the corresponding value (e.h. <code>NO_PROXY=k8smaster01</code>). </p>

<p>It wasn't an issue before, because most applications seems to follow the Linux specification of using lowercase environment variables for proxy usage. </p>
