{"action":"created","issue":{"url":"https://api.github.com/repos/stripe/sequins/issues/163","repository_url":"https://api.github.com/repos/stripe/sequins","labels_url":"https://api.github.com/repos/stripe/sequins/issues/163/labels{/name}","comments_url":"https://api.github.com/repos/stripe/sequins/issues/163/comments","events_url":"https://api.github.com/repos/stripe/sequins/issues/163/events","html_url":"https://github.com/stripe/sequins/pull/163","id":301894788,"number":163,"title":"Serialize cleaning up znodes for old db versions and avoid a race condition","user":{"login":"ying-stripe","id":31416070,"avatar_url":"https://avatars2.githubusercontent.com/u/31416070?v=4","gravatar_id":"","url":"https://api.github.com/users/ying-stripe","html_url":"https://github.com/ying-stripe","followers_url":"https://api.github.com/users/ying-stripe/followers","following_url":"https://api.github.com/users/ying-stripe/following{/other_user}","gists_url":"https://api.github.com/users/ying-stripe/gists{/gist_id}","starred_url":"https://api.github.com/users/ying-stripe/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ying-stripe/subscriptions","organizations_url":"https://api.github.com/users/ying-stripe/orgs","repos_url":"https://api.github.com/users/ying-stripe/repos","events_url":"https://api.github.com/users/ying-stripe/events{/privacy}","received_events_url":"https://api.github.com/users/ying-stripe/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":{"login":"bartle-stripe","id":23070319,"avatar_url":"https://avatars2.githubusercontent.com/u/23070319?v=4","gravatar_id":"","url":"https://api.github.com/users/bartle-stripe","html_url":"https://github.com/bartle-stripe","followers_url":"https://api.github.com/users/bartle-stripe/followers","following_url":"https://api.github.com/users/bartle-stripe/following{/other_user}","gists_url":"https://api.github.com/users/bartle-stripe/gists{/gist_id}","starred_url":"https://api.github.com/users/bartle-stripe/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bartle-stripe/subscriptions","organizations_url":"https://api.github.com/users/bartle-stripe/orgs","repos_url":"https://api.github.com/users/bartle-stripe/repos","events_url":"https://api.github.com/users/bartle-stripe/events{/privacy}","received_events_url":"https://api.github.com/users/bartle-stripe/received_events","type":"User","site_admin":false},"assignees":[{"login":"bartle-stripe","id":23070319,"avatar_url":"https://avatars2.githubusercontent.com/u/23070319?v=4","gravatar_id":"","url":"https://api.github.com/users/bartle-stripe","html_url":"https://github.com/bartle-stripe","followers_url":"https://api.github.com/users/bartle-stripe/followers","following_url":"https://api.github.com/users/bartle-stripe/following{/other_user}","gists_url":"https://api.github.com/users/bartle-stripe/gists{/gist_id}","starred_url":"https://api.github.com/users/bartle-stripe/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bartle-stripe/subscriptions","organizations_url":"https://api.github.com/users/bartle-stripe/orgs","repos_url":"https://api.github.com/users/bartle-stripe/repos","events_url":"https://api.github.com/users/bartle-stripe/events{/privacy}","received_events_url":"https://api.github.com/users/bartle-stripe/received_events","type":"User","site_admin":false},{"login":"vasi-stripe","id":28717751,"avatar_url":"https://avatars0.githubusercontent.com/u/28717751?v=4","gravatar_id":"","url":"https://api.github.com/users/vasi-stripe","html_url":"https://github.com/vasi-stripe","followers_url":"https://api.github.com/users/vasi-stripe/followers","following_url":"https://api.github.com/users/vasi-stripe/following{/other_user}","gists_url":"https://api.github.com/users/vasi-stripe/gists{/gist_id}","starred_url":"https://api.github.com/users/vasi-stripe/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/vasi-stripe/subscriptions","organizations_url":"https://api.github.com/users/vasi-stripe/orgs","repos_url":"https://api.github.com/users/vasi-stripe/repos","events_url":"https://api.github.com/users/vasi-stripe/events{/privacy}","received_events_url":"https://api.github.com/users/vasi-stripe/received_events","type":"User","site_admin":false}],"milestone":null,"comments":0,"created_at":"2018-03-02T20:28:38Z","updated_at":"2018-03-02T20:38:21Z","closed_at":null,"author_association":"CONTRIBUTOR","pull_request":{"url":"https://api.github.com/repos/stripe/sequins/pulls/163","html_url":"https://github.com/stripe/sequins/pull/163","diff_url":"https://github.com/stripe/sequins/pull/163.diff","patch_url":"https://github.com/stripe/sequins/pull/163.patch"},"body":"We found in the sequins log most zookeeper errors happen because of 'node does not exist'. TriggerCleanup() is used to cleanup the old znodes for db versions that sequins nodes are no longer serving. But it takes a naive approach to deletes znodes. It just traverses the directory tree and delete the non-ephemeral znodes that can be deleted.  So there is a race condition when a znode for a new db version was just created and no partition of the db version has been downloaded by any sequins node (therefore no ephemeral nodes created in the db version znode). TriggerCleanup() can delete the db version znode right after it is created, causing all sequins nodes watching the db version znode to get 'node does not exist' error and they have to reconnect to zookeeper. The reconnecting to zookeeper is very expensive in sequins. It also causes unnecessary load on zookeeper server. We should avoid this race condition. \r\n\r\nThere is another problem with TriggerCleanup(). It launches a large number of goroutines to attempt to delete znodes in parallel. We found this behavior causes scheduling delay for other goroutines. It causes the 504 errors because the goroutines serving http requests are timed out. We also need to change this behavior.  \r\n\r\nr? @bartle-stripe @vasi-stripe  "},"comment":{"url":"https://api.github.com/repos/stripe/sequins/issues/comments/370045597","html_url":"https://github.com/stripe/sequins/pull/163#issuecomment-370045597","issue_url":"https://api.github.com/repos/stripe/sequins/issues/163","id":370045597,"user":{"login":"bartle-stripe","id":23070319,"avatar_url":"https://avatars2.githubusercontent.com/u/23070319?v=4","gravatar_id":"","url":"https://api.github.com/users/bartle-stripe","html_url":"https://github.com/bartle-stripe","followers_url":"https://api.github.com/users/bartle-stripe/followers","following_url":"https://api.github.com/users/bartle-stripe/following{/other_user}","gists_url":"https://api.github.com/users/bartle-stripe/gists{/gist_id}","starred_url":"https://api.github.com/users/bartle-stripe/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bartle-stripe/subscriptions","organizations_url":"https://api.github.com/users/bartle-stripe/orgs","repos_url":"https://api.github.com/users/bartle-stripe/repos","events_url":"https://api.github.com/users/bartle-stripe/events{/privacy}","received_events_url":"https://api.github.com/users/bartle-stripe/received_events","type":"User","site_admin":false},"created_at":"2018-03-02T20:38:21Z","updated_at":"2018-03-02T20:38:21Z","author_association":"CONTRIBUTOR","body":"lgtm"}}
{"actor":{"display_login":"bartle-stripe"}}