<p>My task is to add a label named "app" to all <code>deployments</code>, <code>daemonsets</code>, and <code>cronjobs</code> so that it's easier to query our apps across the stack in our monitoring tools. This way, we can build dashboards that use a single selector, namely app.</p>

<p>To avoid downtime I've decided to resolve this issue in the following steps:</p>

<ol>
<li>Add labels to dev, test &amp; stage environments.</li>
<li>Add labels to prod env's.</li>
<li>Deploy (1)</li>
<li>Deploy (2)</li>
<li>Delete old labels &amp; update the services of dev to use the new labels. Then test &amp; deploy. (<strong>currently on this step</strong>)</li>
<li>Repeat (5) for stage.</li>
<li>Repeat (5) for prod.</li>
</ol>

<p>When using <code>$ kubectl apply</code> to update the resources I've added the "app" label to/replaced "service" label with "app" labels to, I run into the following error:</p>

<blockquote>
  <p>Error from server (Invalid): error when applying patch:
  {<em>longAssPatchWhichIWon'tIncludeButYaGetThePoint</em>} to: &amp;{0xc421b02f00
  0xc420803650 default provisioning
  manifests/prod/provisioning-deployment.yaml 0xc 42000c6f8 3942200
  false} for: "manifests/prod/provisioning-deployment.yaml":
  Deployment.apps "provisioning" is invalid: s
  pec.template.metadata.labels: Invalid value:
  map[string]string{"app":"provisioning", "component" :"marketplace"}:
  <code>selector</code> does not match template <code>labels</code></p>
</blockquote>

<p>I need some insights on why it's throwing this error.</p>

<p>It seems you are in trouble. Check this section: <a href="https://kubernetes.io/docs/concepts/workloads/controllers/deployment/#label-selector-updates" rel="noreferrer">Label selector updates</a></p>

<blockquote>
  <p>Note: In API version <code>apps/v1</code>, a Deployment’s label selector is immutable after it gets created.</p>
</blockquote>

<p>So, this line say you can not update <code>selector</code> once deployment is created. Selector can not be changed for any API version except <code>apps/v1beta1</code> and <code>extension/v1beta1</code>. Ref: <a href="https://github.com/kubernetes/kubernetes/blob/dad6741530b0c69587a71a6c08544b4c9412fa01/test/integration/deployment/deployment_test.go#L226" rel="noreferrer">TestDeploymentSelectorImmutability</a>.</p>

<p>One possible workaround might be to keep the old labels and adding new labels along with old ones. This way, you don't have to update <code>selector</code>. Deployment will select pods using old labels but your dashboard can select using new labels. This might not meet your requirement but I don't see any better way.</p>
