{"action":"closed","issue":{"url":"https://api.github.com/repos/tendermint/tendermint/issues/1890","repository_url":"https://api.github.com/repos/tendermint/tendermint","labels_url":"https://api.github.com/repos/tendermint/tendermint/issues/1890/labels{/name}","comments_url":"https://api.github.com/repos/tendermint/tendermint/issues/1890/comments","events_url":"https://api.github.com/repos/tendermint/tendermint/issues/1890/events","html_url":"https://github.com/tendermint/tendermint/issues/1890","id":337977560,"node_id":"MDU6SXNzdWUzMzc5Nzc1NjA=","number":1890,"title":"Tendermint can hang forever if abci app dies","user":{"login":"ethanfrey","id":5689864,"node_id":"MDQ6VXNlcjU2ODk4NjQ=","avatar_url":"https://avatars3.githubusercontent.com/u/5689864?v=4","gravatar_id":"","url":"https://api.github.com/users/ethanfrey","html_url":"https://github.com/ethanfrey","followers_url":"https://api.github.com/users/ethanfrey/followers","following_url":"https://api.github.com/users/ethanfrey/following{/other_user}","gists_url":"https://api.github.com/users/ethanfrey/gists{/gist_id}","starred_url":"https://api.github.com/users/ethanfrey/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ethanfrey/subscriptions","organizations_url":"https://api.github.com/users/ethanfrey/orgs","repos_url":"https://api.github.com/users/ethanfrey/repos","events_url":"https://api.github.com/users/ethanfrey/events{/privacy}","received_events_url":"https://api.github.com/users/ethanfrey/received_events","type":"User","site_admin":false},"labels":[{"id":422249655,"node_id":"MDU6TGFiZWw0MjIyNDk2NTU=","url":"https://api.github.com/repos/tendermint/tendermint/labels/abci","name":"abci","color":"fafad2","default":false},{"id":99653654,"node_id":"MDU6TGFiZWw5OTY1MzY1NA==","url":"https://api.github.com/repos/tendermint/tendermint/labels/bug","name":"bug","color":"fad2fa","default":true}],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":{"url":"https://api.github.com/repos/tendermint/tendermint/milestones/14","html_url":"https://github.com/tendermint/tendermint/milestone/14","labels_url":"https://api.github.com/repos/tendermint/tendermint/milestones/14/labels","id":3091634,"node_id":"MDk6TWlsZXN0b25lMzA5MTYzNA==","number":14,"title":"launch","description":null,"creator":{"login":"greg-szabo","id":16846635,"node_id":"MDQ6VXNlcjE2ODQ2NjM1","avatar_url":"https://avatars3.githubusercontent.com/u/16846635?v=4","gravatar_id":"","url":"https://api.github.com/users/greg-szabo","html_url":"https://github.com/greg-szabo","followers_url":"https://api.github.com/users/greg-szabo/followers","following_url":"https://api.github.com/users/greg-szabo/following{/other_user}","gists_url":"https://api.github.com/users/greg-szabo/gists{/gist_id}","starred_url":"https://api.github.com/users/greg-szabo/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/greg-szabo/subscriptions","organizations_url":"https://api.github.com/users/greg-szabo/orgs","repos_url":"https://api.github.com/users/greg-szabo/repos","events_url":"https://api.github.com/users/greg-szabo/events{/privacy}","received_events_url":"https://api.github.com/users/greg-szabo/received_events","type":"User","site_admin":false},"open_issues":94,"closed_issues":58,"state":"open","created_at":"2018-02-06T01:49:17Z","updated_at":"2018-07-11T06:11:17Z","due_on":null,"closed_at":null},"comments":4,"created_at":"2018-07-03T16:46:10Z","updated_at":"2018-07-11T06:11:17Z","closed_at":"2018-07-11T06:11:16Z","author_association":"CONTRIBUTOR","body":"<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.-->\r\n\r\n**Is this a BUG REPORT or FEATURE REQUEST?** (choose one):\r\n \r\nBUG REPORT\r\n<!--\r\nIf this is a BUG REPORT, please:\r\n  - Fill in as much of the template below as you can.\r\n\r\nIf this is a FEATURE REQUEST, please:\r\n  - Describe *in detail* the feature/behavior/change you'd like to see.\r\n\r\nIn both cases, be ready for followup questions, and please respond in a timely\r\nmanner. We might ask you to provide additional logs and data (tendermint & app)\r\nin a case of bug.\r\n-->\r\n\r\n**Tendermint version** (use `tendermint version` or `git rev-parse --verify HEAD` if installed from source):\r\n\r\n0.20 (verified with unit test on 0.22+develop)\r\n\r\n**ABCI app** (name for built-in, URL for self-written if it's publicly available):\r\n\r\n**Environment**:\r\n- **OS** (e.g. from /etc/os-release):\r\n- **Install tools**:\r\n- **Others**:\r\n\r\n\r\n**What happened**:\r\n\r\nSome tendermint nodes got stuck, in that they responded to rpc for status, etc, but they did not add more blocks not did they pass on any proposed transactions to other nodes. These were non-validating proxy nodes exposing rpc to the outside.\r\n\r\nDigging through the log files, I found this on the tendermint fullnode logs:\r\n\r\n(There were many such websocket messages, but this seems to be the first one with an error)\r\n```\r\nI[07-01|18:54:21.044] We need more addresses. Sending pexRequest to random peer module=p2p peer=\"Peer{MConn{10.24.2.65:46656} 5c6e058cafccbdc156557b4570a52bda9c23f7a0 out}\r\nI[07-01|18:54:51.041] We need more addresses. Sending pexRequest to random peer module=p2p peer=\"Peer{MConn{10.24.10.10:46656} 778cfa655a5d70a2c356947f90b19bfab83b8c7c out}\r\nE[07-01|18:54:51.948] Error closing connection                     module=rpc-server protocol=websocket remote=127.0.0.1:60496 err=\"close tcp 127.0.0.1:46657->127.0.0.1:60496: use of closed network connection\"\r\nE[07-01|18:54:55.743] Stopping abci.socketClient for error: EOF    module=abci-client connection=consensus\r\nE[07-01|18:54:55.743] Stopping abci.socketClient for error: EOF    module=abci-client connection=mempool\r\nE[07-01|18:54:55.743] Stopping abci.socketClient for error: EOF    module=abci-client connection=query\r\n...\r\nE[07-01|18:55:37.139] Error closing connection                     module=rpc-server protocol=websocket remote=127.0.0.1:44472 err=\"close tcp 127.0.0.1:46657->127.0.0.1:44472: use of closed network connection\" \r\nE[07-01|18:55:41.348] Connection failed @ sendRoutine              module=p2p peer=10.24.5.59:46656 conn=MConn{10.24.5.59:46656} err=\"pong timeout\"\r\nI[07-01|18:55:41.348] Stopping MConnection                         module=p2p peer=10.24.5.59:46656 impl=MConn{10.24.5.59:46656}  \r\n```\r\n\r\nI found this on the abci/bov side:\r\n\r\n```\r\nD[07-01|18:54:40.139] Commit synced                                module=bov height=139302 hash=955C604B97AFF17B6F0B39AD6F9E09D11824628E\r\nD[07-01|18:54:41.939] Commit synced                                module=bov height=139303 hash=955C604B97AFF17B6F0B39AD6F9E09D11824628E\r\nI[07-01|18:55:02.444] Starting ABCI app                            module=bov bind=unix:///socks/app.sock\r\nI[07-01|18:55:02.444] Starting ABCIServer                          module=abci-server impl=ABCIServer    \r\n```\r\n\r\nTendermint-ABCI connection broke and thus tendermint could not process any blocks. Normal behavior for tendermint is to exit in such a condition (I tested this locally, using both tcp:// and unix:// connections). Thus if both containers restart, then they reconnect, reestablish connection, and continue.\r\n\r\nNote that \"Stopping abci.socketClient for error: EOF\" was **15 seconds** after the last block was processed by bov.\r\n\r\n**What you expected to happen**:\r\n\r\nWe were running the abci app with 32MB memory limit, which caused it to crash, but in all my experience tendermint commits suicide when it looses connection to the abci app. This is also reported behavior: https://tendermint.com/docs/running-in-production.html#what-happens-when-my-app-dies. Tendermint continuing with a broken abci connection is pathalogical.\r\n\r\n\r\n**How to reproduce it** (as minimally and precisely as possible):\r\n\r\nKill abci app while some messages are being processed. A bit tricky in production (I think it was due to the behavior of file sockets over mounted docker volumes), but I have a nice unit test for you all :)\r\n\r\n**Logs (you can paste a small part showing an error or link a pastebin, gist, etc. containing more of the log file)**:\r\n\r\n**Config (you can paste only the changes you've made)**:\r\n\r\n**`/dump_consensus_state` output for consensus bugs**\r\n\r\n**Anything else do we need to know**:\r\n\r\nPR coming up, along with an analysis of problematic abci code. Looking at this code, it seems due for a refactor, not up to the level of the rest of this repo."}}
{"actor":{"display_login":"melekes"}}