{"action":"created","issue":{"url":"https://api.github.com/repos/deis/controller/issues/989","repository_url":"https://api.github.com/repos/deis/controller","labels_url":"https://api.github.com/repos/deis/controller/issues/989/labels{/name}","comments_url":"https://api.github.com/repos/deis/controller/issues/989/comments","events_url":"https://api.github.com/repos/deis/controller/issues/989/events","html_url":"https://github.com/deis/controller/issues/989","id":171929174,"number":989,"title":"Deployment errors can lead to wrong pods being deployed","user":{"login":"felixbuenemann","id":909587,"avatar_url":"https://avatars.githubusercontent.com/u/909587?v=3","gravatar_id":"","url":"https://api.github.com/users/felixbuenemann","html_url":"https://github.com/felixbuenemann","followers_url":"https://api.github.com/users/felixbuenemann/followers","following_url":"https://api.github.com/users/felixbuenemann/following{/other_user}","gists_url":"https://api.github.com/users/felixbuenemann/gists{/gist_id}","starred_url":"https://api.github.com/users/felixbuenemann/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/felixbuenemann/subscriptions","organizations_url":"https://api.github.com/users/felixbuenemann/orgs","repos_url":"https://api.github.com/users/felixbuenemann/repos","events_url":"https://api.github.com/users/felixbuenemann/events{/privacy}","received_events_url":"https://api.github.com/users/felixbuenemann/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/deis/controller/labels/bug","name":"bug","color":"fc2929"},{"url":"https://api.github.com/repos/deis/controller/labels/priority%201","name":"priority 1","color":"e99695"}],"state":"open","locked":false,"assignee":{"login":"kmala","id":12552530,"avatar_url":"https://avatars.githubusercontent.com/u/12552530?v=3","gravatar_id":"","url":"https://api.github.com/users/kmala","html_url":"https://github.com/kmala","followers_url":"https://api.github.com/users/kmala/followers","following_url":"https://api.github.com/users/kmala/following{/other_user}","gists_url":"https://api.github.com/users/kmala/gists{/gist_id}","starred_url":"https://api.github.com/users/kmala/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kmala/subscriptions","organizations_url":"https://api.github.com/users/kmala/orgs","repos_url":"https://api.github.com/users/kmala/repos","events_url":"https://api.github.com/users/kmala/events{/privacy}","received_events_url":"https://api.github.com/users/kmala/received_events","type":"User","site_admin":false},"assignees":[{"login":"kmala","id":12552530,"avatar_url":"https://avatars.githubusercontent.com/u/12552530?v=3","gravatar_id":"","url":"https://api.github.com/users/kmala","html_url":"https://github.com/kmala","followers_url":"https://api.github.com/users/kmala/followers","following_url":"https://api.github.com/users/kmala/following{/other_user}","gists_url":"https://api.github.com/users/kmala/gists{/gist_id}","starred_url":"https://api.github.com/users/kmala/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kmala/subscriptions","organizations_url":"https://api.github.com/users/kmala/orgs","repos_url":"https://api.github.com/users/kmala/repos","events_url":"https://api.github.com/users/kmala/events{/privacy}","received_events_url":"https://api.github.com/users/kmala/received_events","type":"User","site_admin":false}],"milestone":{"url":"https://api.github.com/repos/deis/controller/milestones/16","html_url":"https://github.com/deis/controller/milestones/v2.5","labels_url":"https://api.github.com/repos/deis/controller/milestones/16/labels","id":1921992,"number":16,"title":"v2.5","description":null,"creator":{"login":"deis-admin","id":12703824,"avatar_url":"https://avatars.githubusercontent.com/u/12703824?v=3","gravatar_id":"","url":"https://api.github.com/users/deis-admin","html_url":"https://github.com/deis-admin","followers_url":"https://api.github.com/users/deis-admin/followers","following_url":"https://api.github.com/users/deis-admin/following{/other_user}","gists_url":"https://api.github.com/users/deis-admin/gists{/gist_id}","starred_url":"https://api.github.com/users/deis-admin/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/deis-admin/subscriptions","organizations_url":"https://api.github.com/users/deis-admin/orgs","repos_url":"https://api.github.com/users/deis-admin/repos","events_url":"https://api.github.com/users/deis-admin/events{/privacy}","received_events_url":"https://api.github.com/users/deis-admin/received_events","type":"User","site_admin":false},"open_issues":2,"closed_issues":38,"state":"open","created_at":"2016-08-03T21:37:32Z","updated_at":"2016-09-06T18:09:22Z","due_on":"2016-09-06T00:00:00Z","closed_at":null},"comments":21,"created_at":"2016-08-18T15:26:05Z","updated_at":"2016-09-06T18:23:25Z","closed_at":null,"body":"Due to the problem fixed by #946 deployments on our deis v2.3.0 cluster were failing most of the time, but not always.\r\n\r\nBecause version numbers are sequentially inremented and a failed deploy does not skip that bversion number, but re-uses it for the next deploy, it happened that a new deployment referenced pods from a previous failed deployment with the same version number.\r\n\r\nExample:\r\n\r\n| Release | Description | Comment |\r\n| --- | --- | --- |\r\n| v49 | buenemann deployed 2aead2a | successfull release |\r\n| v50 | gitlab deployed 7c1d5e02 | this release failed with a 409 Conflict error when writing the secrets and is not listed in `deis releases` |\r\n| v50 | gitlab deployed 4e0a908 | this deployment went through, but actually used the pod with image `localhost:5555/cms:git-7c1d5e02` from the previous failed release, which was also tagged with `version=v50` so it fit the selectors of the new deployment |\r\n| v51 | buenemann deployed 4e0a908 | successful deploy of 4e0a908 after upgrading to v2.4.0 |\r\n\r\nThis ended up with the following replica set on v50:\r\n\r\n```yaml\r\napiVersion: extensions/v1beta1\r\nkind: ReplicaSet\r\nmetadata: \r\n  annotations: \r\n    deployment.kubernetes.io/revision: \"3\"\r\n    kubernetes.io/change-cause: gitlab deployed 7c1d5e0\r\n  creationTimestamp: 2016-08-18T11:58:43Z\r\n  generation: 1\r\n  labels: \r\n    app: cms\r\n    heritage: deis\r\n    pod-template-hash: \"3228967599\"\r\n    type: cmd\r\n    version: v50\r\n  name: cms-cmd-3228967599\r\n  namespace: cms\r\n  resourceVersion: \"5041061\"\r\n  selfLink: /apis/extensions/v1beta1/namespaces/cms/replicasets/cms-cmd-3228967599\r\n  uid: 1c3d409c-653b-11e6-bedf-0274740f37d1\r\nspec: \r\n  replicas: 1\r\n  selector: \r\n    matchLabels: \r\n      app: cms\r\n      heritage: deis\r\n      pod-template-hash: \"3228967599\"\r\n      type: cmd\r\n  template: \r\n    metadata: \r\n      creationTimestamp: null\r\n      labels: \r\n        app: cms\r\n        heritage: deis\r\n        pod-template-hash: \"3228967599\"\r\n        type: cmd\r\n        version: v50\r\n      name: cms-cmd\r\n    spec: \r\n      containers: \r\n        - env: \r\n            - name: DATABASE_URL\r\n              valueFrom: \r\n                secretKeyRef: \r\n                  key: database-url\r\n                  name: cms-v50-env\r\n            - name: DEIS_APP\r\n              valueFrom: \r\n                secretKeyRef: \r\n                  key: deis-app\r\n                  name: cms-v50-env\r\n            - name: DEIS_KUBERNETES_DEPLOYMENTS\r\n              valueFrom: \r\n                secretKeyRef: \r\n                  key: deis-kubernetes-deployments\r\n                  name: cms-v50-env\r\n            - name: INSTALL_TOOL_PASSWORD\r\n              valueFrom: \r\n                secretKeyRef: \r\n                  key: install-tool-password\r\n                  name: cms-v50-env\r\n            - name: PORT\r\n              valueFrom: \r\n                secretKeyRef: \r\n                  key: port\r\n                  name: cms-v50-env\r\n            - name: SSH_KEY\r\n              valueFrom: \r\n                secretKeyRef: \r\n                  key: ssh-key\r\n                  name: cms-v50-env\r\n            - name: TRUSTED_HOSTS_PATTERN\r\n              valueFrom: \r\n                secretKeyRef: \r\n                  key: trusted-hosts-pattern\r\n                  name: cms-v50-env\r\n            - name: WORKFLOW_RELEASE\r\n              valueFrom: \r\n                secretKeyRef: \r\n                  key: workflow-release\r\n                  name: cms-v50-env\r\n          image: localhost:5555/cms:git-7c1d5e02\r\n          imagePullPolicy: Always\r\n          name: cms-cmd\r\n          readinessProbe: \r\n            failureThreshold: 1\r\n            initialDelaySeconds: 5\r\n            periodSeconds: 5\r\n            successThreshold: 1\r\n            tcpSocket: \r\n              port: 80\r\n            timeoutSeconds: 5\r\n          resources: {}\r\n          terminationMessagePath: /dev/termination-log\r\n      dnsPolicy: ClusterFirst\r\n      restartPolicy: Always\r\n      securityContext: {}\r\n      terminationGracePeriodSeconds: 30\r\nstatus: \r\n  fullyLabeledReplicas: 1\r\n  observedGeneration: 1\r\n  replicas: 1\r\n```\r\n\r\nController Log on deis-controller v2.3.0:\r\n\r\nFailed v50 release (gitlab deployed 7c1d5e0):\r\n\r\n```\r\n10.2.50.8 \"POST /v2/hooks/build HTTP/1.1\" 200 26 \"deis-builder\"\r\n10.2.50.8 \"GET /v2/hooks/key/bf:67:1d:7d:0a:c3:64:79:7b:e3:9d:9e:ad:ff:f6:4a HTTP/1.1\" 200 131 \"deis-builder\"\r\n10.2.50.8 \"POST /v2/hooks/push HTTP/1.1\" 201 357 \"deis-builder\"\r\n10.2.50.8 \"POST /v2/hooks/config HTTP/1.1\" 200 2775 \"deis-builder\"\r\nINFO [cms]: build cms-7f7d7bf created\r\nINFO:api.models.app:[cms]: build cms-7f7d7bf created\r\nINFO [cms]: gitlab deployed 7c1d5e0\r\nINFO:api.models.app:[cms]: gitlab deployed 7c1d5e0\r\nINFO Pulling Docker image localhost:5555/cms:git-7c1d5e02\r\nINFO:registry.dockerclient:Pulling Docker image localhost:5555/cms:git-7c1d5e02\r\nINFO [cms]: adding 5s on to the original 120s timeout to account for the initial delay specified in the liveness / readiness probe\r\nINFO:scheduler:[cms]: adding 5s on to the original 120s timeout to account for the initial delay specified in the liveness / readiness probe\r\nINFO [cms]: This deployments overall timeout is 125s - batch timout is 125s and there are 1 batches to deploy with a total of 1 pods\r\nINFO:scheduler:[cms]: This deployments overall timeout is 125s - batch timout is 125s and there are 1 batches to deploy with a total of 1 pods\r\nERROR [cms]: (app::deploy): There was a problem while deploying v50 of cms-web. Going back to the previous release\r\nERROR:api.models.app:[cms]: (app::deploy): There was a problem while deploying v50 of cms-web. Going back to the previous release\r\nERROR:root:(app::deploy): There was a problem while deploying v50 of cms-web. Going back to the previous release\r\nTraceback (most recent call last):\r\n  File \"/app/scheduler/__init__.py\", line 560, in set_application_config\r\n    self.get_secret(namespace, secret_name)\r\n  File \"/app/scheduler/__init__.py\", line 1206, in get_secret\r\n    'get Secret \"{}\" in Namespace \"{}\"', name, namespace\r\nscheduler.KubeHTTPException: ('failed to get Secret \"cms-v50-env\" in Namespace \"cms\": 404 Not Found\\n{\\'reason\\': \\'NotFound\\', \\'details\\': {\\'name\\': \\'cms-v50-env\\', \\'kind\\': \\'secrets\\'}, \\'metadata\\': {}, \\'code\\': 404, \\'status\\': \\'Failure\\', \\'kind\\': \\'Status\\', \\'message\\': \\'secrets \"cms-v50-env\" not found\\', \\'apiVersion\\': \\'v1\\'}', 'cms-v50-env', 'cms')\r\nDuring handling of the above exception, another exception occurred:\r\nTraceback (most recent call last):\r\n  File \"/app/scheduler/__init__.py\", line 155, in deploy_deployment\r\n    **kwargs)\r\n  File \"/app/scheduler/__init__.py\", line 1692, in update_deployment\r\n    **kwargs)\r\n  File \"/app/scheduler/__init__.py\", line 1854, in _build_deployment_manifest\r\n    manifest['spec']['template'] = self._build_pod_manifest(namespace, name, image, **kwargs)\r\n  File \"/app/scheduler/__init__.py\", line 461, in _build_pod_manifest\r\n    self._set_container(namespace, container_name, container, **kwargs)\r\n  File \"/app/scheduler/__init__.py\", line 587, in _set_container\r\n    self.set_application_config(namespace, env, kwargs.get('version'))\r\n  File \"/app/scheduler/__init__.py\", line 562, in set_application_config\r\n    self.create_secret(namespace, secret_name, secrets_env, labels=labels)\r\n  File \"/app/scheduler/__init__.py\", line 1269, in create_secret\r\n    'failed to create Secret \"{}\" in Namespace \"{}\"', name, namespace\r\nscheduler.KubeHTTPException: ('failed to failed to create Secret \"cms-v50-env\" in Namespace \"cms\": 409 Conflict\\n{\\'reason\\': \\'AlreadyExists\\', \\'details\\': {\\'name\\': \\'cms-v50-env\\', \\'kind\\': \\'secrets\\'}, \\'metadata\\': {}, \\'code\\': 409, \\'status\\': \\'Failure\\', \\'kind\\': \\'Status\\', \\'message\\': \\'secrets \"cms-v50-env\" already exists\\', \\'apiVersion\\': \\'v1\\'}', 'cms-v50-env', 'cms')\r\nThe above exception was the direct cause of the following exception:\r\nTraceback (most recent call last):\r\n  File \"/app/api/models/app.py\", line 560, in deploy\r\n    async_run(tasks)\r\n  File \"/app/api/utils.py\", line 161, in async_run\r\n    raise error\r\n  File \"/usr/lib/python3.5/concurrent/futures/thread.py\", line 55, in run\r\n    result = self.fn(*self.args, **self.kwargs)\r\n  File \"/app/scheduler/__init__.py\", line 114, in deploy\r\n    self.deploy_deployment(namespace, name, image, entrypoint, command, **kwargs)\r\n  File \"/app/scheduler/__init__.py\", line 169, in deploy_deployment\r\n    ) from e\r\nscheduler.KubeException: There was a problem while deploying v50 of cms-web. Going back to the previous release\r\nThe above exception was the direct cause of the following exception:\r\nTraceback (most recent call last):\r\n  File \"/app/api/models/build.py\", line 64, in create\r\n    self.app.deploy(new_release)\r\n  File \"/app/api/models/app.py\", line 564, in deploy\r\n    raise ServiceUnavailable(err) from e\r\napi.exceptions.ServiceUnavailable: (app::deploy): There was a problem while deploying v50 of cms-web. Going back to the previous release\r\nThe above exception was the direct cause of the following exception:\r\nTraceback (most recent call last):\r\n  File \"/usr/local/lib/python3.5/dist-packages/rest_framework/views.py\", line 463, in dispatch\r\n    response = handler(request, *args, **kwargs)\r\n  File \"/app/api/views.py\", line 474, in create\r\n    super(BuildHookViewSet, self).create(request, *args, **kwargs)\r\n  File \"/usr/local/lib/python3.5/dist-packages/rest_framework/mixins.py\", line 21, in create\r\n    self.perform_create(serializer)\r\n  File \"/app/api/viewsets.py\", line 21, in perform_create\r\n    self.post_save(obj)\r\n  File \"/app/api/views.py\", line 480, in post_save\r\n    build.create(self.user)\r\n  File \"/app/api/models/build.py\", line 71, in create\r\n    raise DeisException(str(e)) from e\r\napi.exceptions.DeisException: (app::deploy): There was a problem while deploying v50 of cms-web. Going back to the previous release\r\n10.2.50.8 \"POST /v2/hooks/build HTTP/1.1\" 400 114 \"deis-builder\"\r\n```\r\n\r\nSuccessful v50 release (gitlab deployed 4e0a908):\r\n\r\n```\r\n10.2.50.8 \"GET /v2/hooks/key/bf:67:1d:7d:0a:c3:64:79:7b:e3:9d:9e:ad:ff:f6:4a HTTP/1.1\" 200 131 \"deis-builder\"\r\n10.2.50.8 \"POST /v2/hooks/push HTTP/1.1\" 201 357 \"deis-builder\"\r\n10.2.50.8 \"POST /v2/hooks/config HTTP/1.1\" 200 2775 \"deis-builder\"\r\nINFO [cms]: build cms-db04ef1 created\r\nINFO:api.models.app:[cms]: build cms-db04ef1 created\r\nINFO [cms]: gitlab deployed 4e0a908\r\nINFO:api.models.app:[cms]: gitlab deployed 4e0a908\r\nINFO Pulling Docker image localhost:5555/cms:git-4e0a9088\r\nINFO:registry.dockerclient:Pulling Docker image localhost:5555/cms:git-4e0a9088\r\nINFO [cms]: Deployment cms-web with release v50 already exists. Stopping deploy\r\nINFO:scheduler:[cms]: Deployment cms-web with release v50 already exists. Stopping deploy\r\nINFO [cms]: Deployment cms-cmd with release v50 already exists. Stopping deploy\r\nINFO:scheduler:[cms]: Deployment cms-cmd with release v50 already exists. Stopping deploy\r\n10.2.50.8 \"POST /v2/hooks/build HTTP/1.1\" 200 26 \"deis-builder\"\r\n10.2.50.13 \"GET /v2/apps/cms/releases/?limit=100 HTTP/1.1\" 200 14609 \"Deis Client v2.3.0\"\r\n10.2.50.13 \"GET /v2/apps/cms/releases/?limit=100 HTTP/1.1\" 200 14609 \"Deis Client v2.3.0\"\r\n10.2.50.8 \"GET /v2/hooks/key/b4:8b:80:d0:e8:d4:12:14:7b:6e:5c:ad:c2:e9:74:43 HTTP/1.1\" 200 134 \"deis-builder\"\r\n```\r\n\r\nThe above log shows that it re-used the previous failed v50 release.\r\n\r\nOutput from `deis releases`:\r\n```\r\nv51\t2016-08-18T14:33:26Z\tbuenemann deployed 4e0a908\r\nv50\t2016-08-18T12:12:43Z\tgitlab deployed 4e0a908\r\nv49\t2016-08-18T11:27:29Z\tbuenemann deployed 2aead2a\r\nv48\t2016-08-18T11:06:15Z\tbuenemann added DEIS_KUBERNETES_DEPLOYMENTS\r\n```\r\n\r\nGit history:\r\n```\r\n4e0a9088175a2664602f243937fc557466106247\r\nbf362e11b53675bb3d4cdbaa14478dd267f10f37\r\n7c1d5e02c55c3869a1b940bacfb8668789b5b2cc\r\n2aead2aaeebb54c685a53e81f211fd3deaa202ac\r\n```\r\n"},"comment":{"url":"https://api.github.com/repos/deis/controller/issues/comments/245042690","html_url":"https://github.com/deis/controller/issues/989#issuecomment-245042690","issue_url":"https://api.github.com/repos/deis/controller/issues/989","id":245042690,"user":{"login":"kmala","id":12552530,"avatar_url":"https://avatars.githubusercontent.com/u/12552530?v=3","gravatar_id":"","url":"https://api.github.com/users/kmala","html_url":"https://github.com/kmala","followers_url":"https://api.github.com/users/kmala/followers","following_url":"https://api.github.com/users/kmala/following{/other_user}","gists_url":"https://api.github.com/users/kmala/gists{/gist_id}","starred_url":"https://api.github.com/users/kmala/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kmala/subscriptions","organizations_url":"https://api.github.com/users/kmala/orgs","repos_url":"https://api.github.com/users/kmala/repos","events_url":"https://api.github.com/users/kmala/events{/privacy}","received_events_url":"https://api.github.com/users/kmala/received_events","type":"User","site_admin":false},"created_at":"2016-09-06T18:23:25Z","updated_at":"2016-09-06T18:23:25Z","body":"yes we are evaluating on using a different release numbers and have status as failed deploy and also make sure that user can't rollback to this specific release.\r\nthe only time when this issue occurs is when the rollback didn't complete properly and a new deploy happens in the mean time."}}
{"actor":{"display_login":"kmala"}}