{"action":"created","issue":{"url":"https://api.github.com/repos/nats-io/gnatsd/issues/718","repository_url":"https://api.github.com/repos/nats-io/gnatsd","labels_url":"https://api.github.com/repos/nats-io/gnatsd/issues/718/labels{/name}","comments_url":"https://api.github.com/repos/nats-io/gnatsd/issues/718/comments","events_url":"https://api.github.com/repos/nats-io/gnatsd/issues/718/events","html_url":"https://github.com/nats-io/gnatsd/issues/718","id":348281022,"node_id":"MDU6SXNzdWUzNDgyODEwMjI=","number":718,"title":"Slow message throughput compared to TCP server","user":{"login":"swdee","id":8000937,"node_id":"MDQ6VXNlcjgwMDA5Mzc=","avatar_url":"https://avatars3.githubusercontent.com/u/8000937?v=4","gravatar_id":"","url":"https://api.github.com/users/swdee","html_url":"https://github.com/swdee","followers_url":"https://api.github.com/users/swdee/followers","following_url":"https://api.github.com/users/swdee/following{/other_user}","gists_url":"https://api.github.com/users/swdee/gists{/gist_id}","starred_url":"https://api.github.com/users/swdee/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/swdee/subscriptions","organizations_url":"https://api.github.com/users/swdee/orgs","repos_url":"https://api.github.com/users/swdee/repos","events_url":"https://api.github.com/users/swdee/events{/privacy}","received_events_url":"https://api.github.com/users/swdee/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":4,"created_at":"2018-08-07T11:31:11Z","updated_at":"2018-08-07T21:17:51Z","closed_at":null,"author_association":"NONE","body":"## Issue\r\n\r\nDuring our testing of NATS we found the message throughput to be slower than expected compared to messages received by a simple TCP server.  Whilst we would expect some overhead due to the functionality that NATS is providing we are only achieving around 1.8k messages per second through NATS.  When the same sized 32 byte messages are sent direct to a TCP server the throughput is around 9k/sec on the same hardware.\r\n\r\n### Software\r\ngnatsd: 1.2.0-beta4\r\ngolang: go1.10.1 linux/amd64\r\n\r\n## Method\r\n\r\nWith reference to our test Producer and Worker programs here; https://github.com/swdee/natsbench\r\n\r\n### Test 1\r\n\r\nWe configure three Linode 2048 instances in the same datacenter with each program running on its own instance, producing a message flow of;\r\n\r\nProducer -> NATS -> Worker\r\n\r\nThe Producer and Worker start up HTTP /metrics endpoints that are scraped by Prometheus and a Grafana dashboard is used to monitor the message through put of Producer and Worker.\r\n\r\nThe NATS server is started with\r\n```\r\n./gnatsd\r\n```\r\n\r\nWorker is started with\r\n```\r\n./worker2 -n nats://<nats-server-ip>:4222\r\n```\r\n\r\nThe Producer is started and sends 10k msgs/sec at 32 byte size using\r\n```\r\n./producer2 -d nats://<nats-server-ip>:4222 -r 10000\r\n```\r\n\r\n### Test 2\r\n\r\nUsing the same Linode instances we change the topology and replace NATS with HAproxy to flow through with the Worker acting as a simple TCP server to receive the messages directly.\r\n\r\nProducer -> HAProxy -> Worker\r\n\r\nWorker is started and listens on port 8095\r\n```\r\n./worker2 -s :8095\r\n```\r\n\r\nThe Producer is started with same 10k msgs/sec rate at 32 byte size\r\n```\r\n./producer2 -d worker://<haproxy-server-ip>:8095 -r 10000\r\n```\r\n\r\n## Result\r\n\r\nThe attached graph shows throughput of Test 1 sending and receiving around 1.8k msg/sec when using NATS.\r\n\r\n![nats-thruput-graph](https://user-images.githubusercontent.com/8000937/43773639-77f47a22-9a9a-11e8-81e7-80d84d1261c3.png)\r\n\r\nTest 2 on the other hand achieves ~9k msg/sec\r\n\r\nWe did not expect NATS to have a throughput rate of approximately 20% of that when sending to a simple TCP server.  Is there some error we have made in the Producer/Worker code that explains the poor performance with NATS?"},"comment":{"url":"https://api.github.com/repos/nats-io/gnatsd/issues/comments/411205685","html_url":"https://github.com/nats-io/gnatsd/issues/718#issuecomment-411205685","issue_url":"https://api.github.com/repos/nats-io/gnatsd/issues/718","id":411205685,"node_id":"MDEyOklzc3VlQ29tbWVudDQxMTIwNTY4NQ==","user":{"login":"swdee","id":8000937,"node_id":"MDQ6VXNlcjgwMDA5Mzc=","avatar_url":"https://avatars3.githubusercontent.com/u/8000937?v=4","gravatar_id":"","url":"https://api.github.com/users/swdee","html_url":"https://github.com/swdee","followers_url":"https://api.github.com/users/swdee/followers","following_url":"https://api.github.com/users/swdee/following{/other_user}","gists_url":"https://api.github.com/users/swdee/gists{/gist_id}","starred_url":"https://api.github.com/users/swdee/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/swdee/subscriptions","organizations_url":"https://api.github.com/users/swdee/orgs","repos_url":"https://api.github.com/users/swdee/repos","events_url":"https://api.github.com/users/swdee/events{/privacy}","received_events_url":"https://api.github.com/users/swdee/received_events","type":"User","site_admin":false},"created_at":"2018-08-07T21:17:51Z","updated_at":"2018-08-07T21:17:51Z","author_association":"NONE","body":"Issue can be closed thanks."}}
{"actor":{"display_login":"swdee"}}