<p>I have Installed Istio-1.8.3 via Rancher UI long back and Istio Pods and</p>
<p>Ingress Gateway Pods are Up and Running and My Application is getting served by Istio.</p>
<p>Now, Recently We have Upgraded the K8's Cluster version from 1.21 to 1.22 and then 1.22 to 1.23.</p>
<p>Once We restart the kubelet, Istio Pods used to come up with No Issues.</p>
<p>Now, Because of few Issues We have rebooted the Node and Istio Pods got restarted, They are in Running State but Readiness Probe is getting failed.</p>
<p>The Error I was able to find is</p>
<pre><code>failed to list CRDs: the server could not find the requested resource
</code></pre>
<p>Below are the full logs of Istio Pod.</p>
<p><div class="snippet" data-lang="js" data-hide="false" data-console="true" data-babel="false">
<div class="snippet-code">
<pre class="snippet-code-html lang-html prettyprint-override"><code>stream logs failed container "discovery" in pod "istiod-5fbc9568cd-qgqkk" is waiting to start: ContainerCreating for istio-system/istiod-5fbc9568cd-qgqkk (discovery)
2022-06-27T05:35:32.772949Z    info    FLAG: --log_rotate_max_age="30"
2022-06-27T05:35:32.772952Z    info    FLAG: --log_rotate_max_backups="1000"
2022-06-27T05:35:32.772955Z    info    FLAG: --log_rotate_max_size="104857600"
2022-06-27T05:35:32.772958Z    info    FLAG: --log_stacktrace_level="default:none"
2022-06-27T05:35:32.772963Z    info    FLAG: --log_target="[stdout]"
2022-06-27T05:35:32.772971Z    info    FLAG: --mcpInitialConnWindowSize="1048576"
2022-06-27T05:35:32.772974Z    info    FLAG: --mcpInitialWindowSize="1048576"
2022-06-27T05:35:32.772977Z    info    FLAG: --mcpMaxMsgSize="4194304"
2022-06-27T05:35:32.772980Z    info    FLAG: --meshConfig="./etc/istio/config/mesh"
2022-06-27T05:35:32.772982Z    info    FLAG: --monitoringAddr=":15014"
2022-06-27T05:35:32.772985Z    info    FLAG: --namespace="istio-system"
2022-06-27T05:35:32.772988Z    info    FLAG: --networksConfig="/etc/istio/config/meshNetworks"
2022-06-27T05:35:32.772999Z    info    FLAG: --plugins="[authn,authz,health]"
2022-06-27T05:35:32.773002Z    info    FLAG: --profile="true"
2022-06-27T05:35:32.773005Z    info    FLAG: --registries="[Kubernetes]"
2022-06-27T05:35:32.773008Z    info    FLAG: --resync="1m0s"
2022-06-27T05:35:32.773011Z    info    FLAG: --secureGRPCAddr=":15012"
2022-06-27T05:35:32.773013Z    info    FLAG: --tlsCertFile=""
2022-06-27T05:35:32.773016Z    info    FLAG: --tlsKeyFile=""
2022-06-27T05:35:32.773018Z    info    FLAG: --trust-domain=""
2022-06-27T05:35:32.801976Z    info    klog    Config not found: /var/run/secrets/remote/config[]
2022-06-27T05:35:32.803516Z    info    initializing mesh configuration ./etc/istio/config/mesh
2022-06-27T05:35:32.804499Z    info    mesh configuration: {
    "proxyListenPort": 15001,
    "connectTimeout": "10s",
    "protocolDetectionTimeout": "0s",
    "ingressClass": "istio",
    "ingressService": "istio-ingressgateway",
    "ingressControllerMode": "STRICT",
    "enableTracing": true,
    "defaultConfig": {
        "configPath": "./etc/istio/proxy",
        "binaryPath": "/usr/local/bin/envoy",
        "serviceCluster": "istio-proxy",
        "drainDuration": "45s",
        "parentShutdownDuration": "60s",
        "discoveryAddress": "istiod.istio-system.svc:15012",
        "proxyAdminPort": 15000,
        "controlPlaneAuthPolicy": "MUTUAL_TLS",
        "statNameLength": 189,
        "concurrency": 2,
        "tracing": {
            "zipkin": {
                "address": "zipkin.istio-system:9411"
            }
        },
        "envoyAccessLogService": {

        },
        "envoyMetricsService": {

        },
        "proxyMetadata": {
            "DNS_AGENT": ""
        },
        "statusPort": 15020,
        "terminationDrainDuration": "5s"
    },
    "outboundTrafficPolicy": {
        "mode": "ALLOW_ANY"
    },
    "enableAutoMtls": true,
    "trustDomain": "cluster.local",
    "trustDomainAliases": [
    ],
    "defaultServiceExportTo": [
        "*"
    ],
    "defaultVirtualServiceExportTo": [
        "*"
    ],
    "defaultDestinationRuleExportTo": [
        "*"
    ],
    "rootNamespace": "istio-system",
    "localityLbSetting": {
        "enabled": true
    },
    "dnsRefreshRate": "5s",
    "certificates": [
    ],
    "thriftConfig": {

    },
    "serviceSettings": [
    ],
    "enablePrometheusMerge": true
}
2022-06-27T05:35:32.804516Z    info    version: 1.8.3-e282a1f927086cc046b967f0171840e238a9aa8c-Clean
2022-06-27T05:35:32.804699Z    info    flags: 
2022-06-27T05:35:32.804706Z    info    initializing mesh networks
2022-06-27T05:35:32.804877Z    info    mesh networks configuration: {
   "networks": {
   }
}
2022-06-27T05:35:32.804938Z    info    initializing mesh handlers
2022-06-27T05:35:32.804949Z    info    initializing controllers
2022-06-27T05:35:32.804952Z    info    No certificates specified, skipping K8S DNS certificate controller
2022-06-27T05:35:32.814002Z    error    kube    failed to list CRDs: the server could not find the requested resource
2022-06-27T05:35:33.816596Z    error    kube    failed to list CRDs: the server could not find the requested resource
2022-06-27T05:35:35.819157Z    error    kube    failed to list CRDs: the server could not find the requested resource
2022-06-27T05:35:39.821510Z    error    kube    failed to list CRDs: the server could not find the requested resource
2022-06-27T05:35:47.823675Z    error    kube    failed to list CRDs: the server could not find the requested resource
2022-06-27T05:36:03.827023Z    error    kube    failed to list CRDs: the server could not find the requested resource
2022-06-27T05:36:35.829441Z    error    kube    failed to list CRDs: the server could not find the requested resource
2022-06-27T05:37:35.831758Z    error    kube    failed to list CRDs: the server could not find the requested resource</code></pre>
</div>
</div>
</p>

RajNikhil Marpu
<p>Upgrading Istio Pilot and Istio Ingress Gateway from 1.8.3 to 1.10.2 will work.</p>
<p><a href="https://github.com/istio/istio/issues/34665" rel="nofollow noreferrer">https://github.com/istio/istio/issues/34665</a></p>
