{"action":"created","issue":{"url":"https://api.github.com/repos/tendermint/tendermint/issues/2955","repository_url":"https://api.github.com/repos/tendermint/tendermint","labels_url":"https://api.github.com/repos/tendermint/tendermint/issues/2955/labels{/name}","comments_url":"https://api.github.com/repos/tendermint/tendermint/issues/2955/comments","events_url":"https://api.github.com/repos/tendermint/tendermint/issues/2955/events","html_url":"https://github.com/tendermint/tendermint/issues/2955","id":386383609,"node_id":"MDU6SXNzdWUzODYzODM2MDk=","number":2955,"title":"Deadlock in addrbook","user":{"login":"ebuchman","id":2300911,"node_id":"MDQ6VXNlcjIzMDA5MTE=","avatar_url":"https://avatars3.githubusercontent.com/u/2300911?v=4","gravatar_id":"","url":"https://api.github.com/users/ebuchman","html_url":"https://github.com/ebuchman","followers_url":"https://api.github.com/users/ebuchman/followers","following_url":"https://api.github.com/users/ebuchman/following{/other_user}","gists_url":"https://api.github.com/users/ebuchman/gists{/gist_id}","starred_url":"https://api.github.com/users/ebuchman/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ebuchman/subscriptions","organizations_url":"https://api.github.com/users/ebuchman/orgs","repos_url":"https://api.github.com/users/ebuchman/repos","events_url":"https://api.github.com/users/ebuchman/events{/privacy}","received_events_url":"https://api.github.com/users/ebuchman/received_events","type":"User","site_admin":false},"labels":[{"id":99653654,"node_id":"MDU6TGFiZWw5OTY1MzY1NA==","url":"https://api.github.com/repos/tendermint/tendermint/labels/bug","name":"bug","color":"fad2fa","default":true},{"id":728824307,"node_id":"MDU6TGFiZWw3Mjg4MjQzMDc=","url":"https://api.github.com/repos/tendermint/tendermint/labels/p2p","name":"p2p","color":"fafad2","default":false}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":3,"created_at":"2018-11-30T23:08:48Z","updated_at":"2018-12-12T17:34:40Z","closed_at":null,"author_association":"CONTRIBUTOR","body":"Seeds halted on gaia-9002 testnet. Stopped responding to RPC and no new log messages. SIGABRT resulted in a large dump, with many routines blocked on accessing the AddrBook, suggesting a deadlock.\r\n\r\nIt also included this gem:\r\n\r\n```\r\nNov 30 22:21:27 ip-10-0-0-195 gaiad[3406]: goroutine 2056 [runnable]:\r\nNov 30 22:21:27 ip-10-0-0-195 gaiad[3406]: math/rand.(*Rand).Int31(0xc422610690, 0x6ce96401)\r\nNov 30 22:21:27 ip-10-0-0-195 gaiad[3406]: /usr/local/go/src/math/rand/rand.go:96 +0x47\r\nNov 30 22:21:27 ip-10-0-0-195 gaiad[3406]: math/rand.(*Rand).Int31n(0xc422610690, 0x40, 0xc400000001)\r\nNov 30 22:21:27 ip-10-0-0-195 gaiad[3406]: /usr/local/go/src/math/rand/rand.go:128 +0xb4\r\nNov 30 22:21:27 ip-10-0-0-195 gaiad[3406]: math/rand.(*Rand).Intn(0xc422610690, 0x40, 0x1)\r\nNov 30 22:21:27 ip-10-0-0-195 gaiad[3406]: /usr/local/go/src/math/rand/rand.go:169 +0x45\r\nNov 30 22:21:27 ip-10-0-0-195 gaiad[3406]: github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/libs/common.(*Rand).Intn(0xc420b20370, 0x40, 0x1)\r\nNov 30 22:21:27 ip-10-0-0-195 gaiad[3406]: /home/greg/go/src/github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/libs/common/random.go:276 +0x49\r\nNov 30 22:21:27 ip-10-0-0-195 gaiad[3406]: github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/p2p/pex.(*addrBook).GetSelectionWithBias(0xc4200ca700, 0x1e, 0x0, 0x0, 0x0)\r\nNov 30 22:21:27 ip-10-0-0-195 gaiad[3406]: /home/greg/go/src/github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/p2p/pex/addrbook.go:412 +0x2b9\r\n```\r\n\r\nSpecifically `/usr/local/go/src/math/rand/rand.go:169 +0x45` is a panic! it doesn’t seem to be getting caught, but because we’re not using defer, the mtx in our libs/common/rand isn’t being released, locking up the addrbook\r\n\r\nThe core of the problem is `sdk/vendor/github.com/tendermint/tendermint/p2p/pex/addrbook.go:412 +0x2b9`, where we must be passing a 0, which is invalid for the rand methods. for some reason the `len(a.bucketsOld)` is 0 there, and we’ll need to prevent that."},"comment":{"url":"https://api.github.com/repos/tendermint/tendermint/issues/comments/446674683","html_url":"https://github.com/tendermint/tendermint/issues/2955#issuecomment-446674683","issue_url":"https://api.github.com/repos/tendermint/tendermint/issues/2955","id":446674683,"node_id":"MDEyOklzc3VlQ29tbWVudDQ0NjY3NDY4Mw==","user":{"login":"melekes","id":1282182,"node_id":"MDQ6VXNlcjEyODIxODI=","avatar_url":"https://avatars2.githubusercontent.com/u/1282182?v=4","gravatar_id":"","url":"https://api.github.com/users/melekes","html_url":"https://github.com/melekes","followers_url":"https://api.github.com/users/melekes/followers","following_url":"https://api.github.com/users/melekes/following{/other_user}","gists_url":"https://api.github.com/users/melekes/gists{/gist_id}","starred_url":"https://api.github.com/users/melekes/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/melekes/subscriptions","organizations_url":"https://api.github.com/users/melekes/orgs","repos_url":"https://api.github.com/users/melekes/repos","events_url":"https://api.github.com/users/melekes/events{/privacy}","received_events_url":"https://api.github.com/users/melekes/received_events","type":"User","site_admin":false},"created_at":"2018-12-12T17:34:40Z","updated_at":"2018-12-12T17:34:40Z","author_association":"CONTRIBUTOR","body":"[sigabrt1.txt](https://github.com/tendermint/tendermint/files/2672878/sigabrt1.txt)\r\n\r\nsome interesting bits from the goroutines dump. 3 locked mutexes: cs state, cs reactor and addrbook."}}
{"actor":{"display_login":"melekes"}}