{"action":"created","issue":{"url":"https://api.github.com/repos/elastic/elasticsearch/issues/18481","repository_url":"https://api.github.com/repos/elastic/elasticsearch","labels_url":"https://api.github.com/repos/elastic/elasticsearch/issues/18481/labels{/name}","comments_url":"https://api.github.com/repos/elastic/elasticsearch/issues/18481/comments","events_url":"https://api.github.com/repos/elastic/elasticsearch/issues/18481/events","html_url":"https://github.com/elastic/elasticsearch/pull/18481","id":155913528,"number":18481,"title":"Snapshotting and sync could cause a dead lock TranslogWriter","user":{"login":"bleskes","id":1006375,"avatar_url":"https://avatars.githubusercontent.com/u/1006375?v=3","gravatar_id":"","url":"https://api.github.com/users/bleskes","html_url":"https://github.com/bleskes","followers_url":"https://api.github.com/users/bleskes/followers","following_url":"https://api.github.com/users/bleskes/following{/other_user}","gists_url":"https://api.github.com/users/bleskes/gists{/gist_id}","starred_url":"https://api.github.com/users/bleskes/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bleskes/subscriptions","organizations_url":"https://api.github.com/users/bleskes/orgs","repos_url":"https://api.github.com/users/bleskes/repos","events_url":"https://api.github.com/users/bleskes/events{/privacy}","received_events_url":"https://api.github.com/users/bleskes/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/elastic/elasticsearch/labels/:Translog","name":":Translog","color":"009800"},{"url":"https://api.github.com/repos/elastic/elasticsearch/labels/bug","name":"bug","color":"9e2c2c"},{"url":"https://api.github.com/repos/elastic/elasticsearch/labels/v5.0.0","name":"v5.0.0","color":"dddddd"}],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":1,"created_at":"2016-05-20T08:41:31Z","updated_at":"2016-05-20T08:41:44Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/elastic/elasticsearch/pulls/18481","html_url":"https://github.com/elastic/elasticsearch/pull/18481","diff_url":"https://github.com/elastic/elasticsearch/pull/18481.diff","patch_url":"https://github.com/elastic/elasticsearch/pull/18481.patch"},"body":" #18360 introduced an extra lock in order to allow writes while syncing the translog. This caused a potential deadlock (see [1])  with snapshotting code where we first acquire the instance lock, followed by a sync (which acquires the syncLock). However, the sync logic acquires the syncLock first, followed by the instance lock.\r\n\r\nI considered solving this by not syncing the translog on snapshot - I think we can get away with just flushing it. That however will create subtleties around snapshoting and whether operations in them are persisted. I opted instead to have slightly uglier code with nest synchronized, where the scope of the change is contained to the TranslogWriter class alone. If people feel differently, I can take the other approach.\r\n\r\n[1]\r\n```\r\n  2> \"elasticsearch[node_s1][index][T#1]\" ID=8144 BLOCKED on org.elasticsearch.index.translog.TranslogWriter@7527aa29 owned by \"elasticsearch[node_s1][generic][T#2]\" ID=8079\r\n  2> \tat org.elasticsearch.index.translog.TranslogWriter.syncUpTo(TranslogWriter.java:225)\r\n  2> \t- blocked on org.elasticsearch.index.translog.TranslogWriter@7527aa29\r\n  2> \t- locked java.lang.Object@46936c8a\r\n\r\n  2> \"elasticsearch[node_s1][generic][T#2]\" ID=8079 BLOCKED on java.lang.Object@46936c8a owned by \"elasticsearch[node_s1][index][T#1]\" ID=8144\r\n  2> \tat org.elasticsearch.index.translog.TranslogWriter.syncUpTo(TranslogWriter.java:219)\r\n  2> \t- blocked on java.lang.Object@46936c8a\r\n  2> \tat org.elasticsearch.index.translog.TranslogWriter.sync(TranslogWriter.java:151)\r\n  2> \tat org.elasticsearch.index.translog.TranslogWriter.newSnapshot(TranslogWriter.java:201)\r\n  2> \t- locked org.elasticsearch.index.translog.TranslogWriter@7527aa29\r\n```\r\n"},"comment":{"url":"https://api.github.com/repos/elastic/elasticsearch/issues/comments/220549102","html_url":"https://github.com/elastic/elasticsearch/pull/18481#issuecomment-220549102","issue_url":"https://api.github.com/repos/elastic/elasticsearch/issues/18481","id":220549102,"user":{"login":"bleskes","id":1006375,"avatar_url":"https://avatars.githubusercontent.com/u/1006375?v=3","gravatar_id":"","url":"https://api.github.com/users/bleskes","html_url":"https://github.com/bleskes","followers_url":"https://api.github.com/users/bleskes/followers","following_url":"https://api.github.com/users/bleskes/following{/other_user}","gists_url":"https://api.github.com/users/bleskes/gists{/gist_id}","starred_url":"https://api.github.com/users/bleskes/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bleskes/subscriptions","organizations_url":"https://api.github.com/users/bleskes/orgs","repos_url":"https://api.github.com/users/bleskes/repos","events_url":"https://api.github.com/users/bleskes/events{/privacy}","received_events_url":"https://api.github.com/users/bleskes/received_events","type":"User","site_admin":false},"created_at":"2016-05-20T08:41:44Z","updated_at":"2016-05-20T08:41:44Z","body":"@s1monw can you take a look?"}}
{"actor":{"display_login":"bleskes"}}