{"action":"created","issue":{"url":"https://api.github.com/repos/tendermint/tendermint/issues/3002","repository_url":"https://api.github.com/repos/tendermint/tendermint","labels_url":"https://api.github.com/repos/tendermint/tendermint/issues/3002/labels{/name}","comments_url":"https://api.github.com/repos/tendermint/tendermint/issues/3002/comments","events_url":"https://api.github.com/repos/tendermint/tendermint/issues/3002/events","html_url":"https://github.com/tendermint/tendermint/issues/3002","id":389812563,"node_id":"MDU6SXNzdWUzODk4MTI1NjM=","number":3002,"title":"Tendermint throughput, can't achieve more than 2K. ","user":{"login":"tulioalberton","id":17565140,"node_id":"MDQ6VXNlcjE3NTY1MTQw","avatar_url":"https://avatars0.githubusercontent.com/u/17565140?v=4","gravatar_id":"","url":"https://api.github.com/users/tulioalberton","html_url":"https://github.com/tulioalberton","followers_url":"https://api.github.com/users/tulioalberton/followers","following_url":"https://api.github.com/users/tulioalberton/following{/other_user}","gists_url":"https://api.github.com/users/tulioalberton/gists{/gist_id}","starred_url":"https://api.github.com/users/tulioalberton/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tulioalberton/subscriptions","organizations_url":"https://api.github.com/users/tulioalberton/orgs","repos_url":"https://api.github.com/users/tulioalberton/repos","events_url":"https://api.github.com/users/tulioalberton/events{/privacy}","received_events_url":"https://api.github.com/users/tulioalberton/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2018-12-11T15:16:50Z","updated_at":"2018-12-11T15:24:46Z","closed_at":null,"author_association":"NONE","body":"Hello guys, I hope this message finds you well.\r\n\r\nI could not find another way to ask this question, but could you inform me about the configurations to achieve the throughput of 10K operations/second?\r\n\r\n\"speed: Tendermint blocks can commit to finality in the order of 1 second. Tendermint can handle transaction volume at the rate of 10,000 transactions per second for 250byte transactions. The bottleneck is in the application.\"\r\n\r\nI am using four (4) persistent_peers but I can't reach the 10K throughput. \r\nI barely achieve 2K, I am getting this throughput through tm-monitor.\r\n\r\nI tried many variants of tm-bench command (with r=1000, r=10K, c=100, etc. ) through 6 machines:\r\n./tm-bench -v -c 50 -T 60 -r 100 -s 300 192.168.2.29:26657 -broadcast-tx-method commit\r\nEven send the tx to all endpoints, I can't reach more than 2k.\r\n\r\nI believe that the batch is getting full because of the logs says during several outputs:\r\n\" I[1126-12-01|16:34:31.285] Executed block module=state height=992 validTxs=**36514** invalidTxs=0\r\nI[1126-12-01|16:34:31.299] Committed state module=state height=992 txs=**36514** appHash=F8CF380000000000\"\r\n\r\nThanks in advance!!!\r\n\r\n##########################################################\r\n################ Detailed machine configuration ################\r\n\r\nI have four nodes with the configuration below running the persistent_peers:\r\n\r\nLinux s9 3.13.0-32-generic #57-Ubuntu SMP Tue Jul 15 03:51:08 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux\r\nArchitecture:          x86_64\r\nCPU op-mode(s):        32-bit, 64-bit\r\nByte Order:            Little Endian\r\nCPU(s):                16\r\nOn-line CPU(s) list:   0-15\r\nThread(s) per core:    2\r\nCore(s) per socket:    4\r\nSocket(s):             2\r\nNUMA node(s):          2\r\nVendor ID:             GenuineIntel\r\nCPU family:            6\r\nModel:                 26\r\nStepping:              5\r\nCPU MHz:               2261.099\r\nBogoMIPS:              4521.86\r\nVirtualization:        VT-x\r\nL1d cache:             32K\r\nL1i cache:             32K\r\nL2 cache:              256K\r\nL3 cache:              8192K\r\nNUMA node0 CPU(s):     0,2,4,6,8,10,12,14\r\nNUMA node1 CPU(s):     1,3,5,7,9,11,13,15\r\n\r\nEach one has this config.toml:\r\n#### This is a TOML config file.\r\n#### For more information, see https://github.com/toml-lang/toml\r\n\r\n### main base config options #####\r\n\r\n#### TCP or UNIX socket address of the ABCI application,\r\n#### or the name of an ABCI application compiled in with the Tendermint binary\r\nproxy_app = \"tcp://192.168.2.29:26658\"\r\n\r\n#### A custom human readable name for this node\r\nmoniker = \"s9\"\r\n\r\n#### If this node is many blocks behind the tip of the chain, FastSync\r\n#### allows them to catchup quickly by downloading blocks in parallel\r\n#### and verifying their commits\r\nfast_sync = true\r\n\r\n#### Database backend: leveldb | memdb | cleveldb\r\ndb_backend = \"leveldb\"\r\n\r\n#### Database directory\r\ndb_dir = \"data\"\r\n\r\n#### Output level for logging, including package level options\r\nlog_level = \"main:info,state:info,*:error\"\r\n\r\n#### Output format: 'plain' (colored text) or 'json'\r\nlog_format = \"plain\"\r\n\r\n### additional base config options #####\r\n\r\n#### Path to the JSON file containing the initial validator set and other meta data\r\ngenesis_file = \"config/genesis.json\"\r\n\r\n#### Path to the JSON file containing the private key to use as a validator in the consensus protocol\r\npriv_validator_file = \"config/priv_validator.json\"\r\n\r\n#### TCP or UNIX socket address for Tendermint to listen on for\r\n#### connections from an external PrivValidator process\r\npriv_validator_laddr = \"\"\r\n\r\n#### Path to the JSON file containing the private key to use for node authentication in the p2p protocol\r\nnode_key_file = \"config/node_key.json\"\r\n\r\n#### Mechanism to connect to the ABCI application: socket | grpc\r\nabci = \"socket\"\r\n\r\n#### TCP or UNIX socket address for the profiling server to listen on\r\nprof_laddr = \"\"\r\n\r\n#### If true, query the ABCI app on connecting to a new peer\r\n#### so the app can decide if we should keep the connection or not\r\nfilter_peers = false\r\n\r\n### advanced configuration options #####\r\n\r\n#### rpc server configuration options #####\r\n[rpc]\r\n\r\n#### TCP or UNIX socket address for the RPC server to listen on\r\nladdr = \"tcp://192.168.2.29:26657\"\r\n\r\n#### A list of origins a cross-domain request can be executed from\r\n#### Default value '[]' disables cors support\r\n#### Use '[\"*\"]' to allow any origin\r\ncors_allowed_origins = \"[*]\"\r\n\r\n#### A list of methods the client is allowed to use with cross-domain requests\r\ncors_allowed_methods = \"[HEAD GET POST]\"\r\n\r\n#### A list of non simple headers the client is allowed to use with cross-domain requests\r\ncors_allowed_headers = \"[Origin Accept Content-Type X-Requested-With X-Server-Time]\"\r\n\r\n#### TCP or UNIX socket address for the gRPC server to listen on\r\n#### NOTE: This server only supports /broadcast_tx_commit\r\ngrpc_laddr = \"\"\r\n\r\n#### Maximum number of simultaneous connections.\r\n#### Does not include RPC (HTTP&WebSocket) connections. See max_open_connections\r\n#### If you want to accept more significant number than the default, make sure\r\n#### you increase your OS limits.\r\n#### 0 - unlimited.\r\n#### Should be < {ulimit -Sn} - {MaxNumInboundPeers} - {MaxNumOutboundPeers} - {N of wal, db and other open files}\r\n#### 1024 - 40 - 10 - 50 = 924 = ~900\r\ngrpc_max_open_connections = 1024\r\n\r\n#### Activate unsafe RPC commands like /dial_seeds and /unsafe_flush_mempool\r\nunsafe = false\r\n\r\n#### Maximum number of simultaneous connections (including WebSocket).\r\n#### Does not include gRPC connections. See grpc_max_open_connections\r\n#### If you want to accept more significant number than the default, make sure\r\n#### you increase your OS limits.\r\n#### 0 - unlimited.\r\n#### Should be < {ulimit -Sn} - {MaxNumInboundPeers} - {MaxNumOutboundPeers} - {N of wal, db and other open files}\r\n#### 1024 - 40 - 10 - 50 = 924 = ~900\r\nmax_open_connections = 8192\r\n\r\n### peer to peer configuration options #####\r\n[p2p]\r\n\r\n#### Address to listen for incoming connections\r\nladdr = \"tcp://192.168.2.29:26656\"\r\n\r\n#### Address to advertise to peers for them to dial\r\n#### If empty, will use the same port as the laddr,\r\n#### and will introspect on the listener or use UPnP\r\n#### to figure out the address.\r\nexternal_address = \"\"\r\n\r\n#### Comma separated list of seed nodes to connect to\r\nseeds = \"\"\r\n\r\n#### Comma separated list of nodes to keep persistent connections to\r\npersistent_peers = \"33d312fe0853b9882227dc4238f46ae697ae47cf@192.168.2.30:26656,19cc59c4cfd17cf300cdef05cd86a9b6b3fbda0d@192.168.2.31:26656,87c0a035944c168af9b55d0f13eb22e724224fd2@192.168.2.32:26656\"\r\n\r\n#### UPNP port forwarding\r\nupnp = false\r\n\r\n#### Path to address book\r\naddr_book_file = \"config/addrbook.json\"\r\n\r\n#### Set true for strict address routability rules\r\n#### Set false for private or local networks\r\naddr_book_strict = false\r\n\r\n#### Maximum number of inbound peers\r\nmax_num_inbound_peers = 500\r\n\r\n#### Maximum number of outbound peers to connect to, excluding persistent peers\r\nmax_num_outbound_peers = 500\r\n\r\n#### Time to wait before flushing messages out on the connection\r\nflush_throttle_timeout = \"100ms\"\r\n\r\n#### Maximum size of a message packet payload, in bytes\r\nmax_packet_msg_payload_size = 1024\r\n\r\n#### Rate at which packets can be sent, in bytes/second\r\nsend_rate = 5120000\r\n\r\n#### Rate at which packets can be received, in bytes/second\r\nrecv_rate = 5120000\r\n\r\n#### Set true to enable the peer-exchange reactor\r\npex = true\r\n\r\n#### Seed mode, in which node constantly crawls the network and looks for\r\n#### peers. If another node asks it for addresses, it responds and disconnects.\r\n#### Does not work if the peer-exchange reactor is disabled.\r\nseed_mode = false\r\n\r\n#### Comma separated list of peer IDs to keep private (will not be gossiped to other peers)\r\nprivate_peer_ids = \"\"\r\n\r\n#### Toggle to disable guard against peers connecting from the same ip.\r\nallow_duplicate_ip = true\r\n\r\n#### Peer connection configuration.\r\nhandshake_timeout = \"20s\"\r\ndial_timeout = \"10s\"\r\n\r\n### mempool configuration options #####\r\n[mempool]\r\n\r\nrecheck = true\r\nbroadcast = true\r\nwal_dir = \"\"\r\n\r\n#### size of the mempool\r\nsize = 500000\r\n\r\n#### size of the cache (used to filter transactions we saw earlier)\r\ncache_size = 200000\r\n\r\n### consensus configuration options #####\r\n[consensus]\r\n\r\nwal_file = \"data/cs.wal/wal\"\r\n\r\ntimeout_propose = \"15s\"\r\ntimeout_propose_delta = \"5000ms\"\r\ntimeout_prevote = \"15s\"\r\ntimeout_prevote_delta = \"5000ms\"\r\ntimeout_precommit = \"15s\"\r\ntimeout_precommit_delta = \"5000ms\"\r\ntimeout_commit = \"30s\"\r\n\r\n#### Make progress as soon as we have all the precommits (as if TimeoutCommit = 0)\r\nskip_timeout_commit = false\r\n\r\n#### EmptyBlocks mode and possible interval between empty blocks\r\ncreate_empty_blocks = true\r\ncreate_empty_blocks_interval = \"1s\"\r\n\r\n#### Reactor sleep duration parameters\r\npeer_gossip_sleep_duration = \"200ms\"\r\npeer_query_maj23_sleep_duration = \"4s\"\r\n\r\n#### Block time parameters. Corresponds to the minimum time increment between consecutive blocks.\r\nblocktime_iota = \"1s\"\r\n\r\n### transactions indexer configuration options #####\r\n[tx_index]\r\n\r\n#### What indexer to use for transactions\r\n#### Options:\r\n####   1) \"null\" (default)\r\n####   2) \"kv\" - the simplest possible indexer, backed by key-value storage (defaults to levelDB; see DBBackend).\r\nindexer = \"kv\"\r\n\r\n#### Comma-separated list of tags to index (by default the only tag is \"tx.hash\")\r\n#### You can also index transactions by height by adding \"tx.height\" tag here.\r\n#### It's recommended to index only a subset of tags due to possible memory\r\n#### bloat. This is, of course, depends on the indexer's DB and the volume of\r\n#### transactions.\r\nindex_tags = \"\"\r\n\r\n#### When set to true, tells indexer to index all tags (predefined tags:\r\n#### \"tx.hash\", \"tx.height\" and all tags from DeliverTx responses).\r\n#### Note this may be not desirable (see the comment above). IndexTags has a\r\n#### precedence over IndexAllTags (i.e. when given both, IndexTags will be\r\n#### indexed).\r\nindex_all_tags = false\r\n\r\n### instrumentation configuration options #####\r\n[instrumentation]\r\n\r\n#### When true, Prometheus metrics are served under /metrics on\r\n#### PrometheusListenAddr.\r\n#### Check out the documentation for the list of available metrics.\r\nprometheus = false\r\n\r\n#### Address to listen for Prometheus collector(s) connections\r\nprometheus_listen_addr = \":26660\"\r\n\r\n#### Maximum number of simultaneous connections.\r\n#### If you want to accept more significant number than the default, make sure\r\n#### you increase your OS limits.\r\n#### 0 - unlimited.\r\nmax_open_connections = 3\r\n\r\n#### Instrumentation namespace\r\nnamespace = \"tendermint\"\r\n\r\n\r\n\r\n\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/tendermint/tendermint/issues/comments/446241761","html_url":"https://github.com/tendermint/tendermint/issues/3002#issuecomment-446241761","issue_url":"https://api.github.com/repos/tendermint/tendermint/issues/3002","id":446241761,"node_id":"MDEyOklzc3VlQ29tbWVudDQ0NjI0MTc2MQ==","user":{"login":"tulioalberton","id":17565140,"node_id":"MDQ6VXNlcjE3NTY1MTQw","avatar_url":"https://avatars0.githubusercontent.com/u/17565140?v=4","gravatar_id":"","url":"https://api.github.com/users/tulioalberton","html_url":"https://github.com/tulioalberton","followers_url":"https://api.github.com/users/tulioalberton/followers","following_url":"https://api.github.com/users/tulioalberton/following{/other_user}","gists_url":"https://api.github.com/users/tulioalberton/gists{/gist_id}","starred_url":"https://api.github.com/users/tulioalberton/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tulioalberton/subscriptions","organizations_url":"https://api.github.com/users/tulioalberton/orgs","repos_url":"https://api.github.com/users/tulioalberton/repos","events_url":"https://api.github.com/users/tulioalberton/events{/privacy}","received_events_url":"https://api.github.com/users/tulioalberton/received_events","type":"User","site_admin":false},"created_at":"2018-12-11T15:24:46Z","updated_at":"2018-12-11T15:24:46Z","author_association":"NONE","body":"The config.toml here also, it is better to see it. \r\n\r\n[config.toml.zip](https://github.com/tendermint/tendermint/files/2668151/config.toml.zip)\r\n"}}
{"actor":{"display_login":"tulioalberton"}}