<p>I have a Kubernetes deployment which is the Owner/Parent of two ACTIVE ReplicaSets with different configurations.</p>

<p>This setup is managed by Helm.</p>

<p>I have tried to make the <code>revisionHistory: 0</code>. This doesn't work as the ReplicaSet is not inactive. This old ReplicaSet tries to spin up a pod and it stays in pending because of the resource limitations on the node.</p>

<p>I tried to update the Deployment and only the new ReplicaSet is updated. The old one remains the same.</p>

<p>I am not able to delete this ReplicaSet as well. This is causing a lot of troubles for me.</p>

<p>Could somebody help me with this issue?</p>

<p>Helm Deployment Template -</p>

<pre><code>apiVersion: apps/v1
kind: Deployment
metadata:
  name: example
  namespace: kube-system
spec:
  selector:
    matchLabels:
      k8s-app: example
  replicas: 1
  template:
    metadata:
      labels:
        k8s-app: example
    spec:
      serviceAccountName: example
      nodeSelector:
        node-role: example-node
      containers:
      - name: example
        image: example-image:vX.X.X
        resources:
          requests:
            cpu: 100m
        ports:
        - name: example-port
          containerPort: XXXX
        - name: example-port-1
          containerPort: XXXX
        readinessProbe:
          httpGet:
            path: /example
            port: XXXX
          initialDelaySeconds: 5
          timeoutSeconds: 5
      - name: example-sidecar
        image: example-image-sidecar:vX.X.X
        resources:
          limits:
            memory: 400Mi
          requests:
            cpu: 100m
        env:
          - name: MY_POD_NAME
            valueFrom:
              fieldRef:
                fieldPath: metadata.name
          - name: MY_POD_NAMESPACE
            valueFrom:
              fieldRef:
                fieldPath: metadata.namespace
        command:
          - command
          - --container=example
          - --cpu=200m
          - --extra-cpu=10m
          - --memory=300Mi
          - --extra-memory=2Mi
          - --threshold=5
          - --deployment=example
</code></pre>

<p>Stack:  Standalone K8s deployed on AWS EC2s using Kops and Helm 2.13.1 </p>

<p>OutPuts - </p>

<pre><code>kubectl get rs -o wide -n kube-system | grep example

NAME               DESIRED  CURRENT READY   AGE     CONTAINERS IMAGES SELECTOR
example-6d4f99bc54 0        0       0       12h     example,example-sidecar example-image:vX.X.X,example-image-sidecar:vX.X.X k8s-app=example,pod-template-hash=6d4f99bc54
example-59d46955b6 0        0       0       13h     example,example-sidecar example-image:vX.X.X,example-image-sidecar:vX.X.X k8s-app=example,pod-template-hash=59d46955b6
example-5855866cdb 0        0       0       18h     example,example-sidecar example-image:vX.X.X,example-image-sidecar:vX.X.X k8s-app=example,pod-template-hash=5855866cdb
example-ccc5cf5cd0 0        0       0       18h     example,example-sidecar example-image:vX.X.X,example-image-sidecar:vX.X.X k8s-app=example,pod-template-hash=ccc5cf5cd
example-66db79f578 1        1       0       19h     example,example-sidecar example-image:vX.X.X,example-image-sidecar:vX.X.X k8s-app=example,pod-template-hash=66db79f578
example-759469945f 1        1       1       19h     example,example-sidecar example-image:vX.X.X,example-image-sidecar:vX.X.X k8s-app=example,pod-template-hash=759469945f
example-ff8f986960 0        0       0       19h     example,example-sidecar example-image:vX.X.X,example-image-sidecar:vX.X.X k8s-app=example,pod-template-hash=ff8f98696

</code></pre>

<pre><code>kubectl describe deployments example -n kube-system

Name:                   example
Namespace:              kube-system
CreationTimestamp:      Tue, 03 Mar 2020 00:48:18 +0530
Labels:                 k8s-app=example
Annotations:            deployment.kubernetes.io/revision: 27
Selector:               k8s-app=example
Replicas:               1 desired | 1 updated | 2 total | 1 available | 1 unavailable
StrategyType:           RollingUpdate
MinReadySeconds:        0
RollingUpdateStrategy:  25% max unavailable, 25% max surge
Pod Template:
  Labels:           k8s-app=example
  Service Account:  example
  Containers:
   example:
    Image:       example-image:vX.X.X
    Ports:       8080/TCP, 8081/TCP
    Host Ports:  0/TCP, 0/TCP
    Limits:
      cpu:     1630m
      memory:  586Mi
    Requests:
      cpu:        1630m
      memory:     586Mi
    Readiness:    http-get http://:8080/healthz delay=5s timeout=5s period=10s #success=1 #failure=3
    Environment:  &lt;none&gt;
    Mounts:       &lt;none&gt;
   example-sidecar:
    Image:      example-image-sidecar:vX.X.X
    Port:       &lt;none&gt;
    Host Port:  &lt;none&gt;
    Command:
      command
      --container=example
      --cpu=200m
      --extra-cpu=10m
      --memory=300Mi
      --extra-memory=2Mi
      --threshold=5
      --deployment=example
    Limits:
      memory:  400Mi
    Requests:
      cpu:  100m
    Environment:
      MY_POD_NAME:        (v1:metadata.name)
      MY_POD_NAMESPACE:   (v1:metadata.namespace)
    Mounts:              &lt;none&gt;
  Volumes:               &lt;none&gt;
Conditions:
  Type           Status  Reason
  ----           ------  ------
  Available      True    MinimumReplicasAvailable
OldReplicaSets:  example-759469945f (1/1 replicas created)
NewReplicaSet:   example-66db79f578 (1/1 replicas created)
Events:          &lt;none&gt;
</code></pre>

<pre><code>kubectl rollout history deployments example -n kube-system

deployment.extensions/example 
REVISION  CHANGE-CAUSE
1         &lt;none&gt;
16        &lt;none&gt;
17        &lt;none&gt;
21        &lt;none&gt;
24        &lt;none&gt;
26        &lt;none&gt;
27        &lt;none&gt;
</code></pre>

<p>We have the similar issue with Openshift that 2 replicasets are owned by the same deployment. And unfortunately, this problem is causing their pods are restarting frequently for about every 100 seconds. This is because the 2 replicasets are using different images. (We are upgrading the image frequently)</p>
<p>There are 2 replicasets (ua-operator-776546f4ff, ua-operator-b6c858456) owned by the same deployment (Deployment/ua-operator). We can see the &quot;restartCount&quot; parameter of pod status is always 0. But the pod can live for about only 100 seconds. It is not caused by app issue. It is the 2 replicasets to do something like rolling upgrade (but not rolling upgrade).</p>
<p>Looking into the 2 replicasets (ua-operator-776546f4ff, ua-operator-b6c858456), you can see they are unfortunately using 2 different images.
replicasets / ua-operator-776546f4ff is using ua-operator:APM_202006202301
replicasets / ua-operator-b6c858456 is using ua-operator:APM_202006212301</p>
<p>While the deployment (Deployment/ua-operator) only specifies the newer image:
Deployment/ua-operator is using ua-operator:APM_202006212301</p>
<p>So the problem is:
Someone created the new Deployment/ua-operator while not completely deleted the old Deployment/ua-operator, so the old replicasets / ua-operator-776546f4ff which was using old ua-operator:APM_202006202301 image was not deleted and is now still there and become a child of the new Deployment/ua-operator. And unfortunately the 2 replicasets are using different images. This is the reason to always restart the ua-operator pods.</p>
<p>I am not sure if it is a Kubernetes or Openshift defect. But we can avoid it by wait the old deployment to be completely deleted then to create the new ua-operator deployment.</p>
