{"action":"created","issue":{"url":"https://api.github.com/repos/deis/router/issues/212","repository_url":"https://api.github.com/repos/deis/router","labels_url":"https://api.github.com/repos/deis/router/issues/212/labels{/name}","comments_url":"https://api.github.com/repos/deis/router/issues/212/comments","events_url":"https://api.github.com/repos/deis/router/issues/212/events","html_url":"https://github.com/deis/router/issues/212","id":163534342,"number":212,"title":"With many (custom) domains the router might be too slow to startup","user":{"login":"nathansamson","id":93735,"avatar_url":"https://avatars.githubusercontent.com/u/93735?v=3","gravatar_id":"","url":"https://api.github.com/users/nathansamson","html_url":"https://github.com/nathansamson","followers_url":"https://api.github.com/users/nathansamson/followers","following_url":"https://api.github.com/users/nathansamson/following{/other_user}","gists_url":"https://api.github.com/users/nathansamson/gists{/gist_id}","starred_url":"https://api.github.com/users/nathansamson/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/nathansamson/subscriptions","organizations_url":"https://api.github.com/users/nathansamson/orgs","repos_url":"https://api.github.com/users/nathansamson/repos","events_url":"https://api.github.com/users/nathansamson/events{/privacy}","received_events_url":"https://api.github.com/users/nathansamson/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/deis/router/labels/refactor","name":"refactor","color":"E31919"}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":3,"created_at":"2016-07-02T20:25:36Z","updated_at":"2016-09-16T00:07:02Z","closed_at":null,"body":"Scenario:\r\nHave a few applications; but many custom domains. (100s of them, I have around 300, the specific amount that triggers the problem will be dependant on the load and CPU/memory size of the node)\r\n\r\nRestart the router replica (eg during an upgrade). One way to achieve this is to run\r\n`kubectl --namespace deis scale rc --replicas=0; kubectl --namespace deis scale rc --replicas=1`\r\n\r\nFollow the pods status; see it never coming up; after 20s it will go into Restarts 1\r\nAfter some more time it restarts again and so on.\r\n\r\nIt seems the initial configuration is taking so long to build that nginx never comes up, and the K8s system decides to restart it. \r\n\r\nmanually removing some domans from the annotations of an application made the routers start again\r\n\r\n\r\n\r\nI worked around this issue by changing the router-rc manifest\r\n\r\n~~~\r\n        livenessProbe:\r\n          httpGet:\r\n            path: /healthz\r\n            port: 9090\r\n          initialDelaySeconds: 60\r\n          timeoutSeconds: 1\r\n        readinessProbe:\r\n          httpGet:\r\n            path: /healthz\r\n            port: 9090\r\n          initialDelaySeconds: 60\r\n          timeoutSeconds: 1\r\n~~~\r\n\r\nNote I can not figure out why it initially restarted after 20s, because the initialDelaySeconds was also 1.\r\n\r\n\r\nNote that I am not sure what the correct solution would be; a few solutions spring to mind\r\n\r\n* Make it faster to generate the config. Obviously this only increases the problem treshold, so unless it is like 10 times faster (which is unlikely this is possible) this would not help much\r\n* Make the default config a working config; obviously this has the disadvantage the router is in an up state while in fact it isn't....\r\n* Increase the healthcheck timeout, but again this only changes the treshold a bit...\r\n"},"comment":{"url":"https://api.github.com/repos/deis/router/issues/comments/247488175","html_url":"https://github.com/deis/router/issues/212#issuecomment-247488175","issue_url":"https://api.github.com/repos/deis/router/issues/212","id":247488175,"user":{"login":"helgi","id":70530,"avatar_url":"https://avatars.githubusercontent.com/u/70530?v=3","gravatar_id":"","url":"https://api.github.com/users/helgi","html_url":"https://github.com/helgi","followers_url":"https://api.github.com/users/helgi/followers","following_url":"https://api.github.com/users/helgi/following{/other_user}","gists_url":"https://api.github.com/users/helgi/gists{/gist_id}","starred_url":"https://api.github.com/users/helgi/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/helgi/subscriptions","organizations_url":"https://api.github.com/users/helgi/orgs","repos_url":"https://api.github.com/users/helgi/repos","events_url":"https://api.github.com/users/helgi/events{/privacy}","received_events_url":"https://api.github.com/users/helgi/received_events","type":"User","site_admin":false},"created_at":"2016-09-16T00:07:02Z","updated_at":"2016-09-16T00:07:02Z","body":"DNS resolver being slow? https://stackoverflow.com/questions/35949182/nginx-starts-and-stops-too-slowly-15-seconds-delay"}}
{"actor":{"display_login":"helgi"}}