{"action":"created","issue":{"url":"https://api.github.com/repos/goadapp/goad/issues/28","repository_url":"https://api.github.com/repos/goadapp/goad","labels_url":"https://api.github.com/repos/goadapp/goad/issues/28/labels{/name}","comments_url":"https://api.github.com/repos/goadapp/goad/issues/28/comments","events_url":"https://api.github.com/repos/goadapp/goad/issues/28/events","html_url":"https://github.com/goadapp/goad/issues/28","id":154455398,"number":28,"title":"General questions","user":{"login":"jazzl0ver","id":2444175,"avatar_url":"https://avatars.githubusercontent.com/u/2444175?v=3","gravatar_id":"","url":"https://api.github.com/users/jazzl0ver","html_url":"https://github.com/jazzl0ver","followers_url":"https://api.github.com/users/jazzl0ver/followers","following_url":"https://api.github.com/users/jazzl0ver/following{/other_user}","gists_url":"https://api.github.com/users/jazzl0ver/gists{/gist_id}","starred_url":"https://api.github.com/users/jazzl0ver/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jazzl0ver/subscriptions","organizations_url":"https://api.github.com/users/jazzl0ver/orgs","repos_url":"https://api.github.com/users/jazzl0ver/repos","events_url":"https://api.github.com/users/jazzl0ver/events{/privacy}","received_events_url":"https://api.github.com/users/jazzl0ver/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":1,"created_at":"2016-05-12T11:01:46Z","updated_at":"2016-08-25T23:07:55Z","closed_at":null,"body":"Hello,\r\n\r\nCan you please answer the following questions:\r\n\r\n1 - Am I correct that TotErrors means connection errors happened during tcp handshake?\r\n\r\n2 - I have nginx 1.8 running on c4.4xlarge Amazon Linux instance (16 CPUs, bunch of memory), kernel is tuned like this:\r\n```\r\nnet.ipv4.tcp_syncookies = 0\r\nnet.ipv4.ip_local_port_range=4000 65535\r\nnet.ipv4.tcp_fin_timeout=2\r\nnet.ipv4.tcp_tw_recycle=1\r\nnet.ipv4.tcp_tw_reuse=1\r\nnet.core.somaxconn=1024\r\nnet.core.netdev_max_backlog=65535\r\nnet.ipv4.tcp_max_syn_backlog=65535\r\nnet.core.rmem_max = 16777216\r\nnet.core.wmem_max = 16777216\r\nnet.core.rmem_default = 8388608\r\nnet.core.wmem_default = 8388608\r\nnet.ipv4.tcp_rmem = 4096 8388608 16777216\r\nnet.ipv4.tcp_wmem = 4096 8388608 16777216\r\nfs.file-max = 1000000\r\nfs.nr_open = 1000000\r\n```\r\n\r\nnginx meaningful options are:\r\n```\r\nworker_processes 16;\r\nworker_cpu_affinity 0001 0010 0100 1000 01000 10000 010000 100000 0100000 1000000 01000000 10000000 010000000 100000000 0100000000 1000000000;\r\n\r\nevents {\r\n    worker_connections 655350;\r\n}\r\n```\r\n\r\ngoad is started with:\r\n```\r\n$ build/linux/x86-64/goad -c 25000 -n 2000000 -r us-east-1 -t 30 -u \"http://verygoodhost.dom/\"\r\n```\r\n\r\nThe results differ from time to time:\r\n```\r\n   TotReqs   TotBytes    AvgTime   AvgReq/s  AvgKbps/s\r\n   2000000      32 MB     0.023s  285493.59    4363.29\r\n   Slowest    Fastest   Timeouts  TotErrors\r\n   15.040s     0.000s          0      27467\r\nHTTPStatus   Requests\r\n       200    1972533\r\n```\r\n\r\n```\r\n   TotReqs   TotBytes    AvgTime   AvgReq/s  AvgKbps/s\r\n   2000000      30 MB     0.023s   40320.49     584.64\r\n   Slowest    Fastest   Timeouts  TotErrors\r\n   15.057s     0.000s          0      97712\r\nHTTPStatus   Requests\r\n       200    1902288\r\n```\r\n\r\n```\r\n   TotReqs   TotBytes    AvgTime   AvgReq/s  AvgKbps/s\r\n   1933334      29 MB     0.023s    8653.97     120.74\r\n   Slowest    Fastest   Timeouts  TotErrors\r\n    7.218s     0.000s          0     129231\r\nHTTPStatus   Requests\r\n       200    1804103\r\n```\r\n\r\n```\r\n   TotReqs   TotBytes    AvgTime   AvgReq/s  AvgKbps/s\r\n   2000000      30 MB     0.023s  571338.56    7865.94\r\n   Slowest    Fastest   Timeouts  TotErrors\r\n   15.075s     0.000s          0      97252\r\nHTTPStatus   Requests\r\n       200    1902748\r\n```\r\nCan you please explain why that might happen?\r\n\r\n3 - When I set concurrency to 25000 I expect there will be 25000 opened sockets on the target server. But it seems to be a wrong assumption:\r\n```\r\n# netstat -an|wc -l\r\n6352\r\n# netstat -an|wc -l\r\n3795\r\n# netstat -an|wc -l\r\n2104\r\n# netstat -an|wc -l\r\n3620\r\n# netstat -an|wc -l\r\n3832\r\n# netstat -an|wc -l\r\n4477\r\n# netstat -an|wc -l\r\n2654\r\n# netstat -an|wc -l\r\n742\r\n```\r\nNetstat ran almost every second or two while the test was running. Nginx's  internal status page also does not show 25000 connections. Why?\r\n\r\n4 - Do you have any recommendations for tuning the system in a better way?"},"comment":{"url":"https://api.github.com/repos/goadapp/goad/issues/comments/242573909","html_url":"https://github.com/goadapp/goad/issues/28#issuecomment-242573909","issue_url":"https://api.github.com/repos/goadapp/goad/issues/28","id":242573909,"user":{"login":"danielmotaleite","id":11890049,"avatar_url":"https://avatars.githubusercontent.com/u/11890049?v=3","gravatar_id":"","url":"https://api.github.com/users/danielmotaleite","html_url":"https://github.com/danielmotaleite","followers_url":"https://api.github.com/users/danielmotaleite/followers","following_url":"https://api.github.com/users/danielmotaleite/following{/other_user}","gists_url":"https://api.github.com/users/danielmotaleite/gists{/gist_id}","starred_url":"https://api.github.com/users/danielmotaleite/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/danielmotaleite/subscriptions","organizations_url":"https://api.github.com/users/danielmotaleite/orgs","repos_url":"https://api.github.com/users/danielmotaleite/repos","events_url":"https://api.github.com/users/danielmotaleite/events{/privacy}","received_events_url":"https://api.github.com/users/danielmotaleite/received_events","type":"User","site_admin":false},"created_at":"2016-08-25T23:07:55Z","updated_at":"2016-08-25T23:07:55Z","body":"1- yes, that one is connection errors.\r\nIf after connecting, the server returns 503, it will not be counted here but goes to the HTTPStatus table below\r\n\r\n2- aws ec2 are \"shared\" resources...  sometimes they are faster, other slower... Disk IO and network IO sometime can vary a lot.\r\nyou can try a similar test on real hardware with the same parameters and how it behaves.\r\nAlso, sometime is better to have several smaller machines than just one big one (all eggs in single basket vs several baskets), so try also that and see if you get better results or more consistent ones\r\nFinally, you might be also hitting lambda limits, try to use more zones to spread the lambda requests and avoid being throttled. I also have a pull request that tries to spread better the requests, it might help if you are hitting limits\r\n\r\n3- your result show that you have 0s response time, so probably your connections are being replied so fast that netstat can only see those active connections, the other ones are being open or are already closed. you can try to create a iptables rule that logs all new connections and you can check that way how many of then are open.\r\nTry to request a small static file (i use favicon.ico) and then a bigger file (watch out for volume of traffic) and you should see a difference in the number of open ports. Also, try to play with the keepalive values,it will keep the connection open longer.\r\n\r\nFinally, have you any other system in between? like a CDN, load balancer? those can open less backend connections and reuse then, so less connections for the server to manage.\r\n\r\n4- it is always hard to give this kind of tips without knowing the systems, but nginx is fast and looking at your machine, it looks too big...  unless it is a loadbalancer, it seems weird.\r\n\r\nTry to do several smaller goad tests and start to decrease the number of workers and repeat the tests and see how it behaves.... i suspect it will not make huge difference... and they start to increase the concurrency and again play with the workers.  "}}
{"actor":{"display_login":"danielmotaleite"}}