{"action":"opened","issue":{"url":"https://api.github.com/repos/hail-is/hail/issues/4463","repository_url":"https://api.github.com/repos/hail-is/hail","labels_url":"https://api.github.com/repos/hail-is/hail/issues/4463/labels{/name}","comments_url":"https://api.github.com/repos/hail-is/hail/issues/4463/comments","events_url":"https://api.github.com/repos/hail-is/hail/issues/4463/events","html_url":"https://github.com/hail-is/hail/issues/4463","id":364292890,"node_id":"MDU6SXNzdWUzNjQyOTI4OTA=","number":4463,"title":"site deploy not working","user":{"login":"cseed","id":1244990,"node_id":"MDQ6VXNlcjEyNDQ5OTA=","avatar_url":"https://avatars3.githubusercontent.com/u/1244990?v=4","gravatar_id":"","url":"https://api.github.com/users/cseed","html_url":"https://github.com/cseed","followers_url":"https://api.github.com/users/cseed/followers","following_url":"https://api.github.com/users/cseed/following{/other_user}","gists_url":"https://api.github.com/users/cseed/gists{/gist_id}","starred_url":"https://api.github.com/users/cseed/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/cseed/subscriptions","organizations_url":"https://api.github.com/users/cseed/orgs","repos_url":"https://api.github.com/users/cseed/repos","events_url":"https://api.github.com/users/cseed/events{/privacy}","received_events_url":"https://api.github.com/users/cseed/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2018-09-27T03:39:18Z","updated_at":"2018-09-27T03:39:18Z","closed_at":null,"author_association":"COLLABORATOR","body":"The pods currently look like this:\r\n\r\n```\r\nsite-deployment-5b5697d6bb-2p5rk        1/1       Running             0          6h\r\nsite-deployment-69f686bf7f-266kg        0/1       ContainerCreating   0          2h\r\n```\r\n\r\nThe problem is that site mounts a volume RWO with the certs.  The problem is the second pod can't launch for seamless upgrade because it can't mount the volume.  There is a further discussion here: https://github.com/kubernetes/kubernetes/issues/26567\r\n\r\nShort-term fix: we could delete the pod and recreate on upgrade, which would lead to a short window of downtime.\r\n\r\nLong-term fix: Normally the certs are only read by nginx, but need to be written by the certbot renew cron job.  We could keep the certs in a volume.  We could put a copy of the certs in a secret which can be mounted by multiple pods (e.g. site including when it is upgraded).  Then we run certbot periodically in its own pod and update the original certs stored in the volume.  After it runs, we create a new secret (or upgrade it if we can) with the new certs and then do a seamless upgrade on site."}}
{"actor":{"display_login":"cseed"}}