<p>I deployed my cluster with the --pod-network-cidr added, and have created the new ip pool using calicoctl to change the pods to this range. The problem I am having is exactly what I need to change on the kubernetes side to make the pod cidr range changes? Do I make changes in the API server, Controller manager, and scheduler or is there only specific parts I need to change. I have attempted only changing the controller manager, and those control plane pods go into a crash loop after changing the --cluster-cidr in the yaml. </p>

<p>The output in the controller-manager logs are below?</p>

<p>controllermanager.go:235] error starting controllers: failed to mark cidr[192.168.0.0/24] at idx [0] as occupied for node: : cidr 192.168.0.0/24 is out the range of cluster cidr 10.0.0.0/16</p>

mmiara
<p>Changing a cluster CIDR isn't a simple task. I managed to reproduce your scenario and I managed to change it using the following steps. </p>

<p><strong>Changing an IP pool</strong></p>

<p>The process is as follows :</p>

<ol>
<li>Install calicoctl as a Kubernetes pod (<a href="https://docs.projectcalico.org/getting-started/calicoctl/install#installing-calicoctl-as-a-kubernetes-pod" rel="noreferrer">Source</a>)</li>
<li>Add a new IP pool (<a href="https://docs.projectcalico.org/v3.6/networking/changing-ip-pools" rel="noreferrer">Source</a>).</li>
<li>Disable the old IP pool. This prevents new IPAM allocations from the old IP pool without affecting the networking of existing workloads.</li>
<li>Change nodes <code>podCIDR</code> parameter (<a href="https://serverfault.com/questions/976513/is-it-possible-to-change-cidr-network-flannel-and-kubernetes">Source</a>) </li>
<li>Change <code>--cluster-cidr</code> on <code>kube-controller-manager.yaml</code> on master node. (Credits to <a href="https://stackoverflow.com/users/6759406/mm-wvu18">OP</a> on that)</li>
<li>Recreate all existing workloads that were assigned an address from the old IP pool.</li>
<li>Remove the old IP pool.</li>
</ol>

<p>Let’s get started.</p>

<p>In this example, we are going to replace <code>192.168.0.0/16</code> to <code>10.0.0.0/8</code>. </p>

<ol>
<li><strong>Installing calicoctl as a Kubernetes pod</strong>

<pre><code>$ kubectl apply -f https://docs.projectcalico.org/manifests/calicoctl.yaml
</code></pre>

Setting an alias:

<pre><code>$ alias calicoctl="kubectl exec -i -n kube-system calicoctl -- /calicoctl "
</code></pre></li>
<li><p>Add a new IP pool:</p>

<pre><code>calicoctl create -f -&lt;&lt;EOF
apiVersion: projectcalico.org/v3
kind: IPPool
metadata:
  name: new-pool
spec:
  cidr: 10.0.0.0/8
  ipipMode: Always
  natOutgoing: true
EOF
</code></pre>

<p>We should now have two enabled IP pools, which we can see when running  <code>calicoctl get ippool -o wide</code>:</p>

<pre><code>NAME                  CIDR             NAT    IPIPMODE   DISABLED
default-ipv4-ippool   192.168.0.0/16   true   Always     false
new-pool              10.0.0.0/8       true   Always     false
</code></pre></li>
<li><p>Disable the old IP pool.</p>

<p>First save the IP pool definition to disk:</p>

<pre><code>calicoctl get ippool -o yaml &gt; pool.yaml
</code></pre>

<p><code>pool.yaml</code>  should look like this:</p>

<pre><code>apiVersion: projectcalico.org/v3
items:
- apiVersion: projectcalico.org/v3
  kind: IPPool
  metadata:
    name: default-ipv4-ippool
  spec:
    cidr: 192.168.0.0/16
    ipipMode: Always
    natOutgoing: true
- apiVersion: projectcalico.org/v3
  kind: IPPool
  metadata:
    name: new-pool
  spec:
    cidr: 10.0.0.0/8
    ipipMode: Always
    natOutgoing: true
</code></pre>

<blockquote>
  <p>Note: Some extra cluster-specific information has been redacted to improve readibility.</p>
</blockquote>

<p>Edit the file, adding  <code>disabled: true</code>  to the  <code>default-ipv4-ippool</code>  IP pool:</p>

<pre><code>apiVersion: projectcalico.org/v3
kind: IPPool
metadata:5
  name: default-ipv4-ippool
spec:
  cidr: 192.168.0.0/16
  ipipMode: Always
  natOutgoing: true
  disabled: true
</code></pre>

<p>Apply the changes:</p>

<pre><code>calicoctl apply -f pool.yaml
</code></pre>

<p>We should see the change reflected in the output of  <code>calicoctl get ippool -o wide</code>:</p>

<pre><code>NAME                  CIDR             NAT    IPIPMODE   DISABLED
default-ipv4-ippool   192.168.0.0/16   true   Always     true
new-pool              10.0.0.0/8       true   Always     false
</code></pre></li>
<li><p>Change nodes <code>podCIDR</code> parameter: </p>

<p>Override <code>podCIDR</code> parameter on the particular k8s Node resource with a new IP source range, desirable way with the following commands:</p>

<pre><code>$ kubectl get no kubeadm-0 -o yaml &gt; file.yaml; sed -i "s~192.168.0.0/24~10.0.0.0/16~" file.yaml; kubectl delete no kubeadm-0 &amp;&amp; kubectl create -f file.yaml
$ kubectl get no kubeadm-1 -o yaml &gt; file.yaml; sed -i "s~192.168.1.0/24~10.1.0.0/16~" file.yaml; kubectl delete no kubeadm-1 &amp;&amp; kubectl create -f file.yaml
$ kubectl get no kubeadm-2 -o yaml &gt; file.yaml; sed -i "s~192.168.2.0/24~10.2.0.0/16~" file.yaml; kubectl delete no kubeadm-2 &amp;&amp; kubectl create -f file.yaml    
</code></pre>

<p>We had to perform this action for every node we have. Pay attention to the IP Ranges, they are different from one node to the other. </p></li>
<li><p>Change CIDR on kubeadm-config ConfigMap and kube-controller-manager.yaml</p></li>
</ol>

<p>Edit kubeadm-config ConfigMap and change podSubnet to the new IP Range:</p>

<pre><code>kubectl -n kube-system edit cm kubeadm-config
</code></pre>

<p>Also, change the <code>--cluster-cidr</code> on /etc/kubernetes/manifests/kube-controller-manager.yaml located in the master node. </p>

<pre><code>$ sudo cat /etc/kubernetes/manifests/kube-controller-manager.yaml
apiVersion: v1
kind: Pod
metadata:
  creationTimestamp: null
  labels:
    component: kube-controller-manager
    tier: control-plane
  name: kube-controller-manager
  namespace: kube-system
spec:
  containers:
  - command:
    - kube-controller-manager
    - --allocate-node-cidrs=true
    - --authentication-kubeconfig=/etc/kubernetes/controller-manager.conf
    - --authorization-kubeconfig=/etc/kubernetes/controller-manager.conf
    - --bind-address=127.0.0.1
    - --client-ca-file=/etc/kubernetes/pki/ca.crt
    - --cluster-cidr=10.0.0.0/8
    - --cluster-signing-cert-file=/etc/kubernetes/pki/ca.crt
    - --cluster-signing-key-file=/etc/kubernetes/pki/ca.key
    - --controllers=*,bootstrapsigner,tokencleaner
    - --kubeconfig=/etc/kubernetes/controller-manager.conf
    - --leader-elect=true
    - --node-cidr-mask-size=24
    - --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.crt
    - --root-ca-file=/etc/kubernetes/pki/ca.crt
    - --service-account-private-key-file=/etc/kubernetes/pki/sa.key
    - --service-cluster-ip-range=10.96.0.0/12
    - --use-service-account-credentials=true
</code></pre>

<ol start="7">
<li><p>Recreate all existing workloads using IPs from the disabled pool. In this example, kube-dns is the only workload networked by Calico:</p>

<pre><code>kubectl delete pod -n kube-system kube-dns-6f4fd4bdf-8q7zp
</code></pre>

<p>Check that the new workload now has an address in the new IP pool by running  <code>calicoctl get wep --all-namespaces</code>:</p>

<pre><code>NAMESPACE     WORKLOAD                   NODE      NETWORKS            INTERFACE
kube-system   kube-dns-6f4fd4bdf-8q7zp   vagrant   10.0.24.8/32   cali800a63073ed
</code></pre></li>
<li><p>Delete the old IP pool:</p>

<pre><code>calicoctl delete pool default-ipv4-ippool
</code></pre></li>
</ol>

<p><strong>Creating it correctly from scratch</strong></p>

<p>To deploy a cluster under a specific IP range using Kubeadm and Calico you need to init the cluster with <code>--pod-network-cidr=192.168.0.0/24</code> (where <code>192.168.0.0/24</code> is your desired range) and than you need to tune the Calico manifest before applying it in your fresh cluster. </p>

<p>To tune Calico before applying, you have to download it's yaml file and change the network range. </p>

<ol>
<li>Download the Calico networking manifest for the Kubernetes.

<pre><code>$ curl https://docs.projectcalico.org/manifests/calico.yaml -O
</code></pre></li>
<li>If you are using pod CIDR  <code>192.168.0.0/24</code>, skip to the next step. If you are using a different pod CIDR, use the following commands to set an environment variable called  <code>POD_CIDR</code>  containing your pod CIDR and replace  <code>192.168.0.0/24</code>  in the manifest with your pod CIDR.

<pre><code>$ POD_CIDR="&lt;your-pod-cidr&gt;" \
sed -i -e "s?192.168.0.0/16?$POD_CIDR?g" calico.yaml
</code></pre></li>
<li>Apply the manifest using the following command.

<pre><code>$ kubectl apply -f calico.yaml
</code></pre></li>
</ol>
