{"action":"created","issue":{"url":"https://api.github.com/repos/weaveworks/flux/issues/257","repository_url":"https://api.github.com/repos/weaveworks/flux","labels_url":"https://api.github.com/repos/weaveworks/flux/issues/257/labels{/name}","comments_url":"https://api.github.com/repos/weaveworks/flux/issues/257/comments","events_url":"https://api.github.com/repos/weaveworks/flux/issues/257/events","html_url":"https://github.com/weaveworks/flux/issues/257","id":192276617,"number":257,"title":"Unusual experience rolling out authfe","user":{"login":"jml","id":125674,"avatar_url":"https://avatars.githubusercontent.com/u/125674?v=3","gravatar_id":"","url":"https://api.github.com/users/jml","html_url":"https://github.com/jml","followers_url":"https://api.github.com/users/jml/followers","following_url":"https://api.github.com/users/jml/following{/other_user}","gists_url":"https://api.github.com/users/jml/gists{/gist_id}","starred_url":"https://api.github.com/users/jml/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jml/subscriptions","organizations_url":"https://api.github.com/users/jml/orgs","repos_url":"https://api.github.com/users/jml/repos","events_url":"https://api.github.com/users/jml/events{/privacy}","received_events_url":"https://api.github.com/users/jml/received_events","type":"User","site_admin":false},"labels":[{"id":404383402,"url":"https://api.github.com/repos/weaveworks/flux/labels/bug","name":"bug","color":"ee0701","default":true}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":2,"created_at":"2016-11-29T13:05:53Z","updated_at":"2016-11-30T13:17:54Z","closed_at":null,"body":"Apologies in advance that this isn't a well-defined bug. I just wanted to capture an unusual experience rolling out authfe w/ fluxctl in case it helps.\r\n\r\nDoing the daily prod roll out, a copy/paste error meant that I accidentally typed `fluxctl release --all --update-image quay.io/weaveworks/authfe:quay.io/weaveworks/authfe` and hit enter.\r\n\r\nI hit Ctrl-C as quickly as I could. I got some sort of output, which unfortunately I no longer have access to. \r\n\r\nI then ran the correct command:\r\n\r\n```\r\n$ fluxctl release --all --update-image quay.io/weaveworks/authfe:master-8de2e90\r\n```\r\n\r\nWhich emitted 3 lines of output (again, I don't have it), and hanged.\r\n\r\nI went to our `compare-images` page showed production running the revision `quay.io/weaveworks/authfe`.\r\n\r\nDuring this time, I could still access admin pages which were behind the authfe.\r\n\r\nDue to time pressure (I can't leave prod broken for long; others were waiting for deploy to clear), I attempted to resolve the problem by running `wcloud deploy quay.io/weaveworks/authfe:master-8de2e90`, which worked.\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/weaveworks/flux/issues/comments/263870967","html_url":"https://github.com/weaveworks/flux/issues/257#issuecomment-263870967","issue_url":"https://api.github.com/repos/weaveworks/flux/issues/257","id":263870967,"user":{"login":"squaremo","id":62033,"avatar_url":"https://avatars.githubusercontent.com/u/62033?v=3","gravatar_id":"","url":"https://api.github.com/users/squaremo","html_url":"https://github.com/squaremo","followers_url":"https://api.github.com/users/squaremo/followers","following_url":"https://api.github.com/users/squaremo/following{/other_user}","gists_url":"https://api.github.com/users/squaremo/gists{/gist_id}","starred_url":"https://api.github.com/users/squaremo/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/squaremo/subscriptions","organizations_url":"https://api.github.com/users/squaremo/orgs","repos_url":"https://api.github.com/users/squaremo/repos","events_url":"https://api.github.com/users/squaremo/events{/privacy}","received_events_url":"https://api.github.com/users/squaremo/received_events","type":"User","site_admin":false},"created_at":"2016-11-30T13:17:54Z","updated_at":"2016-11-30T13:17:54Z","body":"I think there's two things going on. The first is that releasing an image without a tag leads to bad things. In general we should stop people deploying an image with no tag, along with #173 perhaps.\r\n\r\nThe second is that rolling authfe out means disrupting websocket connections from daemons (including the one running in prod). Broadly, the system is tolerant to this, since the daemons will reconnect. But the operations underway will look broken."}}
{"actor":{"display_login":"squaremo"}}