<p>The issue I have is that <code>kubeadm</code> will never fully initialize. The output:</p>
<pre><code>...
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory &quot;/etc/kubernetes/manifests&quot;. This can take up to 4m0s
[kubelet-check] Initial timeout of 40s passed.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get &quot;http://localhost:10248/healthz&quot;: dial tcp [::1]:10248: connect: connection refused.
...
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get &quot;http://localhost:10248/healthz&quot;: dial tcp [::1]:10248: connect: connection refused.
...
</code></pre>
<p>and <code>journalctl -xeu kubelet</code> shows the following interesting info:</p>
<pre><code>Dec 03 17:54:08 ip-10-83-62-10.ec2.internal kubelet[14709]: W1203 17:54:08.017925   14709 plugins.go:105] WARNING: aws built-in cloud provider is now deprecated. The AWS provider is deprecated. The AWS provider is deprecated and will be removed in a future release
Dec 03 17:54:08 ip-10-83-62-10.ec2.internal kubelet[14709]: I1203 17:54:08.018044   14709 aws.go:1235] Building AWS cloudprovider
Dec 03 17:54:08 ip-10-83-62-10.ec2.internal kubelet[14709]: I1203 17:54:08.018112   14709 aws.go:1195] Zone not specified in configuration file; querying AWS metadata service
Dec 03 17:56:08 ip-10-83-62-10.ec2.internal kubelet[14709]: F1203 17:56:08.332951   14709 server.go:265] failed to run Kubelet: could not init cloud provider &quot;aws&quot;: error finding instance  i-03e00e9192370ca0d: &quot;error listing AWS instances: \&quot;RequestError: send request failed\\ncaused by: Post \\\&quot;https://ec2.us-east-1.amazonaws.com/\\\&quot;: dial tcp 10.83.60.11:443: i/o timeout

</code></pre>
<p>The context is: it's a fully private AWS VPC. There is a proxy that is propagated to k8s manifests.</p>
<p>the kubeadm.yaml config is pretty innocent and looks like this</p>
<pre><code>---
apiVersion: kubeadm.k8s.io/v1beta2
kind: ClusterConfiguration
apiServer:
  extraArgs:
    cloud-provider: aws
clusterName: cdspidr
controlPlaneEndpoint: ip-10-83-62-10.ec2.internal
controllerManager:
  extraArgs:
    cloud-provider: aws
    configure-cloud-routes: &quot;false&quot;
kubernetesVersion: stable
networking:
  dnsDomain: cluster.local
  podSubnet: 10.83.62.0/24
---
apiVersion: kubeadm.k8s.io/v1beta2
kind: InitConfiguration
nodeRegistration:
  name: ip-10-83-62-10.ec2.internal
  kubeletExtraArgs:
    cloud-provider: was
</code></pre>
<p>I'm looking for help to figure out a couple of things here:</p>
<ol>
<li><p>why does kubeadm use this address (<a href="https://ec2.us-east-1.amazonaws.com" rel="nofollow noreferrer">https://ec2.us-east-1.amazonaws.com</a>) to retrieve availability zones? It does not look correct. IMO, it should be something like <code>http://169.254.169.254/latest/dynamic/instance-identity/document</code></p>
</li>
<li><p>why does it fail? With the same proxy settings, a curl request from the terminal returns the web page.</p>
</li>
<li><p>To workaround it, how can I specify availability zones on my own in kubeadm.yaml or via a command like for kubeadm?</p>
</li>
</ol>
<p>I would appreciate any help or thoughts.</p>

<p>You can create a VPC endpoint for accessing Ec2 (service name - com.amazonaws.us-east-1.ec2), this will allow the kubelet to talk to Ec2 without internet and fetch the required info.</p>
<p>While creating the VPC endpoint please make sure to enable private DNS resolution option.</p>
<p>Also from the error it looks like that kubelet is trying to fetch the instance not just availability zone. (&quot;aws&quot;: error finding instance  i-03e00e9192370ca0d: &quot;error listing AWS instances).</p>
