<p>We use helm to manage all our resources in a k8s cluster. Recently we had an incident where some k8s resources were modified outside of helm (see below for details on the root cause).</p>

<p>The end result is however, that we have some k8s resources in our cluster that do not match what is specified in the helm chart of the release.</p>

<p>Example:</p>

<p>We have a helm chart that contains a <code>HorizontalPodAutoscaler</code>. If I do something like:</p>

<pre><code>helm get myservice-release
</code></pre>

<p>I will see something like this:</p>

<pre><code>---
# Source: myservice/charts/default-deployment/templates/default_deployment.yaml
apiVersion: autoscaling/v2beta1
kind: HorizontalPodAutoscaler
metadata:
  name: myservice-autoscaler
  labels:     
    app: myservice
spec:
  minReplicas: 2
  maxReplicas: 10
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: myservice-deployment
  metrics:
  - type: Resource
    resource:
      name: cpu
      targetAverageUtilization: 85
</code></pre>

<p>However, if I do:</p>

<pre><code>kubectl get hpa myservice-autoscaler -o yaml
</code></pre>

<p>The <code>spec.{max,min}Replicas</code> does not match the Chart:</p>

<pre><code>apiVersion: autoscaling/v1
kind: HorizontalPodAutoscaler
metadata:
  annotations:
    autoscaling.alpha.kubernetes.io/conditions: '{REDACTED}'
    autoscaling.alpha.kubernetes.io/current-metrics: '{REDACTED}'
  creationTimestamp: "{REDACTED}"
  labels:
    app: myservice
  name: myservice-autoscaler
  namespace: default
  resourceVersion: "174526833"
  selfLink: /apis/autoscaling/v1/namespaces/default/horizontalpodautoscalers/myservice-autoscaler
  uid: {REDACTED}
spec:
  maxReplicas: 1
  minReplicas: 1
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: myservice-deployment
  targetCPUUtilizationPercentage: 85
status:
  currentCPUUtilizationPercentage: 9
  currentReplicas: 1
  desiredReplicas: 1
  lastScaleTime: "{REACTED}"
</code></pre>

<p>I suspect, there are more than this one occurrence of drift in the k8s resources.</p>

<ul>
<li>How do I verify which resources have drifted?</li>
<li>How do I inform helm of that drift, so the next deployment could take it into account when applying the release diff?</li>
</ul>

<p><strong>EDIT:</strong></p>

<p>For those of you interested, this was caused by two <code>helm</code> charts managing the same resources (autoscaling) both setting different values.</p>

<p>This occurred because two helm releases that were meant for different namespaces ended up in the same and were updated with <code>--force</code>.</p>

themarex
<p>We figured out a way to do this in a scalable way. Note, that this solution requires Kubernetes 1.13 for support of <code>kubectl diff</code>.</p>

<p>The overall idea is to fetch the helm state and apply it using <code>kubectl</code> to sync the two again. <strong>This might be unsafe on your cluster</strong>, please verify the changes with <code>kubectl diff</code>.</p>

<ol>
<li><p>Fetch the state from helm: <code>helm get manifest {service}-release &gt; {service}-release.yaml</code></p></li>
<li><p>Check if there is a difference to the k8s objects: <code>kubectl diff -f {service}-release.yaml</code></p></li>
<li><p>Overwrite the k8s state with the helm state: <code>kubectl apply -f {service}-release.yaml</code></p></li>
</ol>
