<h3>Present state</h3>
<p>In v1.22 Kubernetes dropped support for <code>v1beta1</code> API. That made our release pipeline crash and we are not sure how to fix it.</p>
<p>We use build pipelines to build .NET Core applications and deploy them to the Azure Container Registry. Then there are release pipelines that use <code>helm</code> to upgrade them in the cluster from that ACR. This is how it looks exactly.</p>
<p>Build pipeline:</p>
<ol>
<li>.NET download, restore, build, test, publish</li>
<li>Docker task v0: Build task</li>
<li>Docker task v0: Push to the <code>ACR</code> task</li>
<li>Artifact publish to Azure Pipelines</li>
</ol>
<p>Release pipeline:</p>
<ol>
<li>Helm tool installer: Install <code>helm</code> v3.2.4 (check for latest version of Helm unchecked) and install newest <code>Kubectl</code> (Check for latest version checked)</li>
<li>Bash task:</li>
</ol>
<pre><code>az acr login --name &lt;acrname&gt;
az acr helm repo add --name &lt;acrname&gt;
</code></pre>
<ol start="3">
<li>Helm upgrade task:
<ul>
<li>chart name <code>&lt;acrname&gt;/&lt;chartname&gt;</code></li>
<li>version <code>empty</code></li>
<li>release name `</li>
</ul>
</li>
</ol>
<p>After the upgrade to Kubernetes v1.22 we are getting the following error in Release step 3.:</p>
<p><code>Error: UPGRADE FAILED: unable to recognize &quot;&quot;: no matches for kind &quot;Ingress&quot; in version &quot;extensions/v1beta1&quot;</code>.</p>
<h3>What I've already tried</h3>
<p>Error is pretty obvious and from <a href="https://helm.sh/docs/topics/version_skew/" rel="nofollow noreferrer">Helm compatibility table</a> it states clearly that I need to upgrade the release pipelines to use at least Helm v3.7.x. Unfortunately in this version OCI functionality (about this shortly) is still in experimental phase so at least v3.8.x has to be used.</p>
<h6>Bumping helm version to v3.8.0</h6>
<p>That makes release step 3. report:</p>
<p><code>Error: looks like &quot;https://&lt;acrname&gt;.azurecr.io/helm/v1/repo&quot; is not a valid chart repository or cannot be reached: error unmarshaling JSON: while decoding JSON: json: unknown field &quot;acrMetadata&quot;</code></p>
<p>After reading Microsoft tutorial on <a href="https://learn.microsoft.com/en-us/azure/container-registry/container-registry-helm-repos" rel="nofollow noreferrer">how to live with <code>helm</code> and <code>ACR</code></a> I learned that <code>az acr helm</code> commands use helm v2 so are deprecated and <code>OCI</code> artifacts should be used.</p>
<h6>Switching to OCI part 1</h6>
<p>After reading that I changed release step 2. to a one-liner:</p>
<p><code>helm registry login &lt;acrname&gt;.azurecr.io  --username &lt;username&gt; --password &lt;password&gt;</code></p>
<p>That now gives me <code>Login Succeeded</code> in release step 2. but release step 3. fails with</p>
<p><code>Error: failed to download &quot;&lt;acrname&gt;/&lt;reponame&gt;&quot;</code>.</p>
<h6>Switching to OCI part 2</h6>
<p>I thought that the helm task is incompatible or something with the new approach so I removed release step 3. and decided to make it from the command line in step 2. So now step 2. looks like this:</p>
<pre><code>helm registry login &lt;acrname&gt;.azurecr.io  --username &lt;username&gt; --password &lt;password&gt;
helm upgrade --install --wait -n &lt;namespace&gt; &lt;deploymentName&gt; oci://&lt;acrname&gt;.azurecr.io/&lt;reponame&gt; --version latest --values ./values.yaml
</code></pre>
<p>Unfortunately, that still gives me:</p>
<p><code>Error: failed to download &quot;oci://&lt;acrname&gt;.azurecr.io/&lt;reponame&gt;&quot; at version &quot;latest&quot;</code></p>
<h6>Helm pull, export, upgrade instead of just upgrade</h6>
<p>The next try was to split the <code>help upgrade</code> into separately <code>helm pull</code>, <code>helm export</code> and then <code>helm upgrade</code> but</p>
<p><code>helm pull oci://&lt;acrname&gt;.azurecr.io/&lt;reponame&gt; --version latest</code></p>
<p>gives me:</p>
<p><code>Error: manifest does not contain minimum number of descriptors (2), descriptors found: 0</code></p>
<h6>Changing <code>docker build</code> and <code>docker push</code> tasks to v2</h6>
<p>I also tried changing the docker tasks in the build pipelines to v2. But that didn't change anything at all.</p>

<p>Have you tried changing the Ingress object's <code>apiVersion</code> to <code>networking.k8s.io/v1beta1</code> or <code>networking.k8s.io/v1</code>? <a href="https://kubernetes.io/blog/2019/07/18/api-deprecations-in-1-16/" rel="nofollow noreferrer">Support for Ingress in the extensions/v1beta1 API version is dropped in k8s 1.22</a>.</p>
<p>Our <code>ingress.yaml</code> file in our helm chart looks something like this to support multiple k8s versions. You can ignore the AWS-specific annotations since you're using Azure. Our chart has a global value of  <code>ingress.enablePathType</code> because at the time of writing the yaml file, <a href="https://github.com/kubernetes-sigs/aws-load-balancer-controller/pull/1772" rel="nofollow noreferrer">AWS Load Balancer did not support pathType</a> and so we set the value to false.</p>
<pre class="lang-yaml prettyprint-override"><code>{{- if .Values.global.ingress.enabled -}}
{{- $useV1Ingress := and (.Capabilities.APIVersions.Has &quot;networking.k8s.io/v1/Ingress&quot;) .Values.global.ingress.enablePathType -}}
{{- if $useV1Ingress -}}
apiVersion: networking.k8s.io/v1
{{- else if semverCompare &quot;&gt;=1.14-0&quot; .Capabilities.KubeVersion.GitVersion -}}
apiVersion: networking.k8s.io/v1beta1
{{- else -}}
apiVersion: extensions/v1beta1
{{- end }}
kind: Ingress
metadata:
  name: example-ingress
  labels:
    {{- include &quot;my-chart.labels&quot; . | nindent 4 }}
  annotations:
    {{- if .Values.global.ingress.group.enabled }}
    alb.ingress.kubernetes.io/group.name: {{ required &quot;ingress.group.name is required when ingress.group.enabled is true&quot; .Values.global.ingress.group.name }}
    {{- end }}
    {{- with .Values.global.ingress.annotations }}
    {{- toYaml . | nindent 4 }}
    {{- end }}
    # Add these tags to the AWS Application Load Balancer
    alb.ingress.kubernetes.io/tags: k8s.namespace/{{ .Release.Namespace }}={{ .Release.Namespace }}
spec:
  rules:
    - host: {{ include &quot;my-chart.applicationOneServerUrl&quot; . | quote }}
      http:
        paths:
          {{- if $useV1Ingress }}
          - path: /
            pathType: Prefix
            backend:
              service:
                name: {{ $applicationOneServiceName }}
                port:
                  name: http-grails
          {{- else }}
          - path: /*
            backend:
              serviceName: {{ $applicationOneServiceName }}
              servicePort: http-grails
          {{- end }}
    - host: {{ include &quot;my-chart.applicationTwoServerUrl&quot; . | quote }}
      http:
        paths:
          {{- if $useV1Ingress }}
          - path: /
            pathType: Prefix
            backend:
              service:
                name: {{ .Values.global.applicationTwo.serviceName }}
                port:
                  name: http-grails
          {{- else }}
          - path: /*
            backend:
              serviceName: {{ .Values.global.applicationTwo.serviceName }}
              servicePort: http-grails
          {{- end }}
{{- end }}
</code></pre>
