{"action":"created","issue":{"url":"https://api.github.com/repos/tendermint/tendermint/issues/2392","repository_url":"https://api.github.com/repos/tendermint/tendermint","labels_url":"https://api.github.com/repos/tendermint/tendermint/issues/2392/labels{/name}","comments_url":"https://api.github.com/repos/tendermint/tendermint/issues/2392/comments","events_url":"https://api.github.com/repos/tendermint/tendermint/issues/2392/events","html_url":"https://github.com/tendermint/tendermint/issues/2392","id":360098498,"node_id":"MDU6SXNzdWUzNjAwOTg0OTg=","number":2392,"title":"Testnet experiences connection issues under load","user":{"login":"philipstanislaus","id":6912756,"node_id":"MDQ6VXNlcjY5MTI3NTY=","avatar_url":"https://avatars1.githubusercontent.com/u/6912756?v=4","gravatar_id":"","url":"https://api.github.com/users/philipstanislaus","html_url":"https://github.com/philipstanislaus","followers_url":"https://api.github.com/users/philipstanislaus/followers","following_url":"https://api.github.com/users/philipstanislaus/following{/other_user}","gists_url":"https://api.github.com/users/philipstanislaus/gists{/gist_id}","starred_url":"https://api.github.com/users/philipstanislaus/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/philipstanislaus/subscriptions","organizations_url":"https://api.github.com/users/philipstanislaus/orgs","repos_url":"https://api.github.com/users/philipstanislaus/repos","events_url":"https://api.github.com/users/philipstanislaus/events{/privacy}","received_events_url":"https://api.github.com/users/philipstanislaus/received_events","type":"User","site_admin":false},"labels":[{"id":99653654,"node_id":"MDU6TGFiZWw5OTY1MzY1NA==","url":"https://api.github.com/repos/tendermint/tendermint/labels/bug","name":"bug","color":"fad2fa","default":true},{"id":728824307,"node_id":"MDU6TGFiZWw3Mjg4MjQzMDc=","url":"https://api.github.com/repos/tendermint/tendermint/labels/p2p","name":"p2p","color":"fafad2","default":false},{"id":823253846,"node_id":"MDU6TGFiZWw4MjMyNTM4NDY=","url":"https://api.github.com/repos/tendermint/tendermint/labels/user","name":"user","color":"d2d2fa","default":false}],"state":"open","locked":false,"assignee":{"login":"melekes","id":1282182,"node_id":"MDQ6VXNlcjEyODIxODI=","avatar_url":"https://avatars2.githubusercontent.com/u/1282182?v=4","gravatar_id":"","url":"https://api.github.com/users/melekes","html_url":"https://github.com/melekes","followers_url":"https://api.github.com/users/melekes/followers","following_url":"https://api.github.com/users/melekes/following{/other_user}","gists_url":"https://api.github.com/users/melekes/gists{/gist_id}","starred_url":"https://api.github.com/users/melekes/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/melekes/subscriptions","organizations_url":"https://api.github.com/users/melekes/orgs","repos_url":"https://api.github.com/users/melekes/repos","events_url":"https://api.github.com/users/melekes/events{/privacy}","received_events_url":"https://api.github.com/users/melekes/received_events","type":"User","site_admin":false},"assignees":[{"login":"melekes","id":1282182,"node_id":"MDQ6VXNlcjEyODIxODI=","avatar_url":"https://avatars2.githubusercontent.com/u/1282182?v=4","gravatar_id":"","url":"https://api.github.com/users/melekes","html_url":"https://github.com/melekes","followers_url":"https://api.github.com/users/melekes/followers","following_url":"https://api.github.com/users/melekes/following{/other_user}","gists_url":"https://api.github.com/users/melekes/gists{/gist_id}","starred_url":"https://api.github.com/users/melekes/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/melekes/subscriptions","organizations_url":"https://api.github.com/users/melekes/orgs","repos_url":"https://api.github.com/users/melekes/repos","events_url":"https://api.github.com/users/melekes/events{/privacy}","received_events_url":"https://api.github.com/users/melekes/received_events","type":"User","site_admin":false}],"milestone":null,"comments":5,"created_at":"2018-09-13T23:08:24Z","updated_at":"2018-10-25T12:31:34Z","closed_at":null,"author_association":"NONE","body":"Thank you for your outstanding work on Tendermint! \r\n\r\nOur team is building THORChain on top of Tendermint using Cosmos. In our initial load tests, we noticed that under heavy load nodes start to loose connections. After some time, nodes give up to reconnect and eventually block creation halts.\r\n\r\nWe are considering to commit resources to improve Tendermint's robustness – but first need to understand a bit better what is going on here.\r\n\r\n**Tendermint version**\r\n\r\n```\r\ntendermint_version=0.22.8-d542d2c3\r\namino_version=0.10.1\r\np2p_version=0.5.0\r\nconsensus_version=v1/0.2.2\r\nrpc_version=0.7.0/3\r\n```\r\n\r\n**ABCI app**\r\n\r\nhttps://github.com/thorchain/THORChain\r\n\r\n**Environment**:\r\n\r\n- **OS** (e.g. from /etc/os-release): \r\n```\r\nNAME=\"Amazon Linux\"\r\nVERSION=\"2\"\r\nID=\"amzn\"\r\nID_LIKE=\"centos rhel fedora\"\r\nVERSION_ID=\"2\"\r\nPRETTY_NAME=\"Amazon Linux 2\"\r\nANSI_COLOR=\"0;33\"\r\nCPE_NAME=\"cpe:2.3:o:amazon:amazon_linux:2\"\r\nHOME_URL=\"https://amazonlinux.com/\"\r\n```\r\n- **Install tools**: See Cosmos SDK\r\n- **Others**: EC2 `c5d.18xlarge` instances running on AWS, all connected using their internal IP addresses for low latency.\r\n\r\n**What happened**:\r\n\r\nOn 9/11/2018, 19:43 UTC the blockchain stopped producing blocks. Until that point, it produced 351,609 blocks with a total of 22,373,891 txs. Around 52 minutes earlier, one of the validators (node0) lost connection. Eventually, all other nodes gave up on reconnecting too. Right now, they are all stuck at different block heights:\r\n\r\n```\r\nnode0: 351588 from 2018-09-11T18:51:37.882205811Z\r\nnode1: 351609 from 2018-09-11T19:43:18.054503634Z\r\nnode2: 351609 from 2018-09-11T19:43:18.054503634Z\r\nnode3: 351609 from 2018-09-11T19:43:18.054503634Z\r\n```\r\n\r\nPlease find detailed logs of all 4 validators below.\r\n\r\nAnalyzing these logs, I found that there were many connection failures, e. g. node1 had the error `Connection failed @ sendRoutine` 61 times and the error `Send failed` 140 times in a 24 hour period.\r\n\r\n**Setup** \r\n\r\nWe have 4 validators running THORChain, an application based on Cosmos with slight tweaks. I have uploaded all the `config.toml` and `genesis.json` here: https://github.com/swishlabsco/tendermint-testnet-issues/tree/master/2018-09-11_testnet_disconnects_under_load/config . We use a [script](https://github.com/thorchain/THORChain/tree/master/cmd/thorchainspam) to produce a load of around 500 TPS. Running the scripts at 100 TPS does not seem to lead to issues – but that needs to be verified.\r\n\r\n**What you expected to happen**:\r\n\r\nNodes not do disconnect\r\n\r\n**Have you tried the latest version**: \r\n\r\nNo, we are using the latest Cosmos SDK (v0.23.1) which depends on Tendermint v0.22.8\r\n\r\n**How to reproduce it** (as minimally and precisely as possible):\r\n\r\nPut a lot of load on a testnet until connection issues occur. We use a spam script to create load, which you can find here: https://github.com/thorchain/THORChain/tree/master/cmd/thorchainspam\r\n\r\n**Logs (paste a small part showing an error (< 10 lines) or link a pastebin, gist, etc. containing more of the log file)**:\r\n\r\n(Logs have ~1 GB in total, so I uploaded them to AWS S3)\r\n\r\nNode0: https://s3.us-east-2.amazonaws.com/tendermint-testnet-issues/log-node0.txt\r\nNode1: https://s3.us-east-2.amazonaws.com/tendermint-testnet-issues/log-node1.txt\r\nNode2: https://s3.us-east-2.amazonaws.com/tendermint-testnet-issues/log-node2.txt\r\nNode3: https://s3.us-east-2.amazonaws.com/tendermint-testnet-issues/log-node3.txt\r\n\r\n**Config (you can paste only the changes you've made)**:\r\n\r\nhttps://github.com/swishlabsco/tendermint-testnet-issues/tree/master/2018-09-11_testnet_disconnects_under_load/config\r\n\r\n**node command runtime flags**:\r\n\r\n`thorchaind start`\r\n\r\n**`/dump_consensus_state` output for consensus bugs**\r\n\r\nhttps://github.com/swishlabsco/tendermint-testnet-issues/tree/master/2018-09-11_testnet_disconnects_under_load/dump_consensus_state"},"comment":{"url":"https://api.github.com/repos/tendermint/tendermint/issues/comments/433032058","html_url":"https://github.com/tendermint/tendermint/issues/2392#issuecomment-433032058","issue_url":"https://api.github.com/repos/tendermint/tendermint/issues/2392","id":433032058,"node_id":"MDEyOklzc3VlQ29tbWVudDQzMzAzMjA1OA==","user":{"login":"melekes","id":1282182,"node_id":"MDQ6VXNlcjEyODIxODI=","avatar_url":"https://avatars2.githubusercontent.com/u/1282182?v=4","gravatar_id":"","url":"https://api.github.com/users/melekes","html_url":"https://github.com/melekes","followers_url":"https://api.github.com/users/melekes/followers","following_url":"https://api.github.com/users/melekes/following{/other_user}","gists_url":"https://api.github.com/users/melekes/gists{/gist_id}","starred_url":"https://api.github.com/users/melekes/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/melekes/subscriptions","organizations_url":"https://api.github.com/users/melekes/orgs","repos_url":"https://api.github.com/users/melekes/repos","events_url":"https://api.github.com/users/melekes/events{/privacy}","received_events_url":"https://api.github.com/users/melekes/received_events","type":"User","site_admin":false},"created_at":"2018-10-25T12:31:34Z","updated_at":"2018-10-25T12:31:34Z","author_association":"CONTRIBUTOR","body":"> Does (or could) Tendermint use a worker process to offload CPU intensive work like checkTx and deliverTx to other cores and keep one core free for these functions?\r\n\r\nGolang scales pretty well with the number of cores\r\n\r\n> Additionally, when I run /dump_consensus_state, I get no response from node0 and node3 (I eventually killed the request after a few hours)\r\n\r\nnext time please provide the output of `kill -6 <PID>`. https://github.com/tendermint/tendermint/issues/1772#issuecomment-410074054 might be related."}}
{"actor":{"display_login":"melekes"}}