{"push_id":1082795043,"size":1,"distinct_size":1,"ref":"refs/heads/master","head":"892a9a1f07d9407b6e0ac4c1e079e779af763ec7","before":"d0b5dac36b8703afaaacb7b5f32f89df019591a3","commits":[{"sha":"892a9a1f07d9407b6e0ac4c1e079e779af763ec7","author":{"email":"46f1a0bd5592a2f9244ca321b129902a06b53e03@epriestley.com","name":"epriestley"},"message":"Make cluster repositories more resistant to freezing\n\nSummary:\nRef T10860. This allows us to recover if the connection to the database is lost during a push.\n\nIf we lose the connection to the master database during a push, we would previously freeze the repository. This is very safe, but not very operator-friendly since you have to go manually unfreeze it.\n\nWe don't need to be quite this aggressive about freezing things. The repository state is still consistent after we've \"upgraded\" the lock by setting `isWriting = 1`, so we're actually fine even if we lost the global lock.\n\nInstead of just freezing the repository immediately, sit there in a loop waiting for the master to come back up for a few minutes. If it recovers, we can release the lock and everything will be OK again.\n\nBasically, the changes are:\n\n  - If we can't release the lock at first, sit in a loop trying really hard to release it for a while.\n  - Add a unique lock identifier so we can be certain we're only releasing //our// lock no matter what else is going on.\n  - Do the version reads on the same connection holding the lock, so we can be sure we haven't lost the lock before we do that read.\n\nTest Plan:\n  - Added a `sleep(10)` after accepting the write but before releasing the lock so I could run `mysqld stop` and force this issue to occur.\n  - Pushed like this:\n\n```\n$ echo D >> record && git commit -am D && git push\n[master 707ecc3] D\n 1 file changed, 1 insertion(+)\n# Push received by \"local001.phacility.net\", forwarding to cluster host.\n# Waiting up to 120 second(s) for a cluster write lock...\n# Acquired write lock immediately.\n# Waiting up to 120 second(s) for a cluster read lock on \"local001.phacility.net\"...\n# Acquired read lock immediately.\n# Device \"local001.phacility.net\" is already a cluster leader and does not need to be synchronized.\n# Ready to receive on cluster host \"local001.phacility.net\".\nCounting objects: 3, done.\nDelta compression using up to 8 threads.\nCompressing objects: 100% (2/2), done.\nWriting objects: 100% (3/3), 254 bytes | 0 bytes/s, done.\nTotal 3 (delta 1), reused 0 (delta 0)\nBEGIN SLEEP\n```\n\n  - Here, I stopped `mysqld` from the CLI in another terminal window.\n\n```\nEND SLEEP\n# CRITICAL. Failed to release cluster write lock!\n# The connection to the master database was lost while receiving the write.\n# This process will spend 300 more second(s) attempting to recover, then give up.\n```\n\n  - Here, I started `mysqld` again.\n\n```\n# RECOVERED. Link to master database was restored.\n# Released cluster write lock.\nTo ssh://local@localvault.phacility.com/diffusion/26/locktopia.git\n   2cbf87c..707ecc3  master -> master\n```\n\nReviewers: chad\n\nReviewed By: chad\n\nManiphest Tasks: T10860\n\nDifferential Revision: https://secure.phabricator.com/D15792","distinct":true,"url":"https://api.github.com/repos/phacility/phabricator/commits/892a9a1f07d9407b6e0ac4c1e079e779af763ec7"}]}