{"action":"created","issue":{"url":"https://api.github.com/repos/vfarcic/docker-flow-proxy/issues/437","repository_url":"https://api.github.com/repos/vfarcic/docker-flow-proxy","labels_url":"https://api.github.com/repos/vfarcic/docker-flow-proxy/issues/437/labels{/name}","comments_url":"https://api.github.com/repos/vfarcic/docker-flow-proxy/issues/437/comments","events_url":"https://api.github.com/repos/vfarcic/docker-flow-proxy/issues/437/events","html_url":"https://github.com/vfarcic/docker-flow-proxy/issues/437","id":299502794,"number":437,"title":"Ways to improve HA of DFP","user":{"login":"Vad1mo","id":1492007,"avatar_url":"https://avatars3.githubusercontent.com/u/1492007?v=4","gravatar_id":"","url":"https://api.github.com/users/Vad1mo","html_url":"https://github.com/Vad1mo","followers_url":"https://api.github.com/users/Vad1mo/followers","following_url":"https://api.github.com/users/Vad1mo/following{/other_user}","gists_url":"https://api.github.com/users/Vad1mo/gists{/gist_id}","starred_url":"https://api.github.com/users/Vad1mo/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Vad1mo/subscriptions","organizations_url":"https://api.github.com/users/Vad1mo/orgs","repos_url":"https://api.github.com/users/Vad1mo/repos","events_url":"https://api.github.com/users/Vad1mo/events{/privacy}","received_events_url":"https://api.github.com/users/Vad1mo/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":7,"created_at":"2018-02-22T20:48:48Z","updated_at":"2018-02-27T00:45:39Z","closed_at":null,"author_association":"NONE","body":"We are running DFP on 3 nodes in a 6 node cluster.  Right now we have about 20-30 services running. The 30 days avg. service availability is ~90% \r\n\r\nThis is has various reasons. One is DFP, DFSL, LetsEncrypt (not the only). Most prominently we see **503** errors even if the service behind DFP is fine. #221\r\n\r\nWe have noticed a few patterns that cause problems.\r\n\r\n1. For example or CI server is rescheduled on a new node, this causes a service disruption. Because our Jenkins isn't in a HA setup it should cause only a disruption of 30-60 seconds, however it often \r\ntakes 15 minutes and more to recover. But more often the service is screwed up and 1/3 of the DFPs responds with 503. Then the only solution is to redeploy DFP.\r\n\r\n2. If DFP is restarted it doesn't get a state and serves 1/3 of the traffic with 503\r\n\r\nI think it would help if DFP health checks would be smarter, the main process could be started with the --check parameter and do extensive checks. \r\n\r\n1. Verify DFP has a state not older then 1 minute\r\n2. The Targets are reachable (hard or soft mode) \r\n\r\nhttps://blog.sixeyed.com/docker-healthchecks-why-not-to-use-curl-or-iwr/\r\n\r\n\r\nIts difficult to pinpoint the problem to a particular problem or but in the code, however I opened this issue to gather feedback and provide more evidence. We really would like to got productive with DFP,Swarm but since months we don't have the confidence to do this step. "},"comment":{"url":"https://api.github.com/repos/vfarcic/docker-flow-proxy/issues/comments/368705107","html_url":"https://github.com/vfarcic/docker-flow-proxy/issues/437#issuecomment-368705107","issue_url":"https://api.github.com/repos/vfarcic/docker-flow-proxy/issues/437","id":368705107,"user":{"login":"thomasjpfan","id":5402633,"avatar_url":"https://avatars2.githubusercontent.com/u/5402633?v=4","gravatar_id":"","url":"https://api.github.com/users/thomasjpfan","html_url":"https://github.com/thomasjpfan","followers_url":"https://api.github.com/users/thomasjpfan/followers","following_url":"https://api.github.com/users/thomasjpfan/following{/other_user}","gists_url":"https://api.github.com/users/thomasjpfan/gists{/gist_id}","starred_url":"https://api.github.com/users/thomasjpfan/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/thomasjpfan/subscriptions","organizations_url":"https://api.github.com/users/thomasjpfan/orgs","repos_url":"https://api.github.com/users/thomasjpfan/repos","events_url":"https://api.github.com/users/thomasjpfan/events{/privacy}","received_events_url":"https://api.github.com/users/thomasjpfan/received_events","type":"User","site_admin":false},"created_at":"2018-02-27T00:45:39Z","updated_at":"2018-02-27T00:45:39Z","author_association":"NONE","body":"I'm looking into it. A solution maybe to add the flag, `-g`, to `tini` as stated in their [docs](https://github.com/krallin/tini#process-group-killing). `tini` by default only kills immediate child process. Adding `-g` would kill the child process group:\r\n\r\n```\r\nENTRYPOINT [\"/sbin/tini\", \"-g\" ,\"--\"]\r\n```\r\n\r\nDFP spawns child processes for haproxy, and for configuration verification.\r\n\r\nSide node: I placed this issue on this thread, since this can cause DFP to not start up properly, thus leading to the OP problem of only 1/3 of the replicas running properly. I have this this happen on running servers as well if i'm quickly iterating over different configuration options."}}
{"actor":{"display_login":"thomasjpfan"}}