{"action":"created","issue":{"url":"https://api.github.com/repos/nghttp2/nghttp2/issues/662","repository_url":"https://api.github.com/repos/nghttp2/nghttp2","labels_url":"https://api.github.com/repos/nghttp2/nghttp2/issues/662/labels{/name}","comments_url":"https://api.github.com/repos/nghttp2/nghttp2/issues/662/comments","events_url":"https://api.github.com/repos/nghttp2/nghttp2/issues/662/events","html_url":"https://github.com/nghttp2/nghttp2/issues/662","id":171959139,"number":662,"title":"[WARN] shrpx_connect_blocker.cc:93 Could not connect X times in a row; sleep for T seconds","user":{"login":"joeshae","id":6749515,"avatar_url":"https://avatars.githubusercontent.com/u/6749515?v=3","gravatar_id":"","url":"https://api.github.com/users/joeshae","html_url":"https://github.com/joeshae","followers_url":"https://api.github.com/users/joeshae/followers","following_url":"https://api.github.com/users/joeshae/following{/other_user}","gists_url":"https://api.github.com/users/joeshae/gists{/gist_id}","starred_url":"https://api.github.com/users/joeshae/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/joeshae/subscriptions","organizations_url":"https://api.github.com/users/joeshae/orgs","repos_url":"https://api.github.com/users/joeshae/repos","events_url":"https://api.github.com/users/joeshae/events{/privacy}","received_events_url":"https://api.github.com/users/joeshae/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/nghttp2/nghttp2/labels/nghttpx","name":"nghttpx","color":"c7def8"}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":23,"created_at":"2016-08-18T17:42:18Z","updated_at":"2016-08-23T16:01:24Z","closed_at":null,"body":"I'm using nghttpx v1.13.0 as a HTTP/2 proxy, and the backend is squid v3.5.20.\r\nBoth of nghttpx and squid are on the same dedicated server, running the latest stable CentOS 7.\r\n\r\nBut there are a lot of the following WARNINGS in /var/log/nghttpx/error.log even when the system is in light load:\r\n\r\n>\r\n...\r\n19/Aug/2016:00:30:38 +0800 PID11918 [WARN] shrpx_connect_blocker.cc:93 Could not connect 1 times in a row; sleep for 1.55938 seconds\r\n19/Aug/2016:00:31:00 +0800 PID11918 [WARN] shrpx_connect_blocker.cc:93 Could not connect 1 times in a row; sleep for 1.39127 seconds\r\n19/Aug/2016:00:31:19 +0800 PID11918 [WARN] shrpx_connect_blocker.cc:93 Could not connect 1 times in a row; sleep for 1.28279 seconds\r\n19/Aug/2016:00:32:25 +0800 PID11918 [WARN] shrpx_connect_blocker.cc:93 Could not connect 1 times in a row; sleep for 1.85783 seconds\r\n19/Aug/2016:00:33:42 +0800 PID11918 [WARN] shrpx_connect_blocker.cc:93 Could not connect 1 times in a row; sleep for 1.89725 seconds\r\n19/Aug/2016:00:34:51 +0800 PID11918 [WARN] shrpx_connect_blocker.cc:93 Could not connect 1 times in a row; sleep for 1.33974 seconds\r\n19/Aug/2016:00:35:37 +0800 PID11918 [WARN] shrpx_connect_blocker.cc:93 Could not connect 1 times in a row; sleep for 1.31277 seconds\r\n19/Aug/2016:00:41:40 +0800 PID11918 [WARN] shrpx_connect_blocker.cc:93 Could not connect 1 times in a row; sleep for 1.90595 seconds\r\n19/Aug/2016:00:43:04 +0800 PID11918 [WARN] shrpx_connect_blocker.cc:93 Could not connect 1 times in a row; sleep for 1.76135 seconds\r\n19/Aug/2016:00:44:39 +0800 PID11918 [NOTICE] Renew new ticket keys\r\n19/Aug/2016:00:45:54 +0800 PID11918 [WARN] shrpx_connect_blocker.cc:93 Could not connect 1 times in a row; sleep for 1.41557 seconds\r\n19/Aug/2016:00:47:05 +0800 PID11918 [WARN] shrpx_connect_blocker.cc:93 Could not connect 1 times in a row; sleep for 1.84914 seconds\r\n19/Aug/2016:00:48:10 +0800 PID11918 [WARN] shrpx_connect_blocker.cc:93 Could not connect 1 times in a row; sleep for 1.62921 seconds\r\n19/Aug/2016:00:49:24 +0800 PID11918 [WARN] shrpx_connect_blocker.cc:93 Could not connect 1 times in a row; sleep for 1.86594 seconds\r\n19/Aug/2016:00:50:52 +0800 PID11918 [WARN] shrpx_connect_blocker.cc:93 Could not connect 1 times in a row; sleep for 1.85862 seconds\r\n19/Aug/2016:00:53:27 +0800 PID14292 [NOTICE] Renew new ticket keys\r\n19/Aug/2016:00:56:46 +0800 PID11918 [WARN] shrpx_connect_blocker.cc:93 Could not connect 1 times in a row; sleep for 1.46331 seconds\r\n19/Aug/2016:00:58:55 +0800 PID11918 [WARN] shrpx_connect_blocker.cc:93 Could not connect 1 times in a row; sleep for 1.87852 seconds\r\n19/Aug/2016:00:59:44 +0800 PID11918 [WARN] shrpx_connect_blocker.cc:93 Could not connect 1 times in a row; sleep for 1.73629 seconds\r\n19/Aug/2016:01:00:49 +0800 PID11918 [WARN] shrpx_connect_blocker.cc:93 Could not connect 1 times in a row; sleep for 1.35262 seconds\r\n19/Aug/2016:01:00:49 +0800 PID11918 [WARN] shrpx_connect_blocker.cc:93 Could not connect 1 times in a row; sleep for 1.43397 seconds\r\n19/Aug/2016:01:01:57 +0800 PID11918 [WARN] shrpx_connect_blocker.cc:93 Could not connect 1 times in a row; sleep for 1.57779 seconds\r\n19/Aug/2016:01:02:25 +0800 PID11918 [WARN] shrpx_connect_blocker.cc:93 Could not connect 1 times in a row; sleep for 1.69779 seconds\r\n19/Aug/2016:01:02:56 +0800 PID11918 [WARN] shrpx_connect_blocker.cc:93 Could not connect 1 times in a row; sleep for 1.86793 seconds\r\n19/Aug/2016:01:04:31 +0800 PID11918 [WARN] shrpx_connect_blocker.cc:93 Could not connect 1 times in a row; sleep for 1.81543 seconds\r\n19/Aug/2016:01:04:44 +0800 PID11918 [WARN] shrpx_connect_blocker.cc:93 Could not connect 1 times in a row; sleep for 1.73382 seconds\r\n19/Aug/2016:01:06:21 +0800 PID11918 [WARN] shrpx_connect_blocker.cc:93 Could not connect 1 times in a row; sleep for 1.38318 seconds\r\n19/Aug/2016:01:06:36 +0800 PID11918 [WARN] shrpx_connect_blocker.cc:93 Could not connect 1 times in a row; sleep for 1.76642 seconds\r\n19/Aug/2016:01:07:38 +0800 PID6821 [NOTICE] Renew new ticket keys\r\n19/Aug/2016:01:07:38 +0800 PID6809 [NOTICE] Renew new ticket keys\r\n19/Aug/2016:01:07:38 +0800 PID6832 [NOTICE] Renew new ticket keys\r\n19/Aug/2016:01:08:07 +0800 PID11918 [WARN] shrpx_connect_blocker.cc:93 Could not connect 1 times in a row; sleep for 1.69277 seconds\r\n19/Aug/2016:01:10:22 +0800 PID11918 [WARN] shrpx_connect_blocker.cc:93 Could not connect 1 times in a row; sleep for 1.82191 seconds\r\n19/Aug/2016:01:11:43 +0800 PID11918 [WARN] shrpx_connect_blocker.cc:93 Could not connect 1 times in a row; sleep for 1.64596 seconds\r\n19/Aug/2016:01:14:48 +0800 PID11918 [WARN] shrpx_connect_blocker.cc:93 Could not connect 1 times in a row; sleep for 1.40443 seconds\r\n19/Aug/2016:01:18:12 +0800 PID11918 [WARN] shrpx_connect_blocker.cc:93 Could not connect 1 times in a row; sleep for 1.58199 seconds\r\n...\r\n\r\n\r\nBut there are no WARNING or ERROR messages in squid logs.\r\nActually, squid seems to be running very well when I try connecting to it directly without nghttpx.\r\n\r\n\r\nThis is my nghttpx conf:\r\n>\r\nprivate-key-file=/usr/etc/foobar.com.key\r\ncertificate-file=/usr/etc/foobar.com.crt\r\n>\r\nfrontend=0.0.0.0,443\r\nbackend=127.0.0.1,143\r\nbackend=127.0.0.1,144\r\nbackend=127.0.0.1,145\r\nbackend=127.0.0.1,146\r\nbackend=127.0.0.1,147\r\nbackend=127.0.0.1,148\r\nbackend=127.0.0.1,149\r\nbackend=127.0.0.1,150\r\n>\r\nworkers=8\r\ndaemon=yes\r\nhttp2-proxy=yes\r\n>\r\n\\# I've tried spdy/3.1,h2,h2-16,h2-14,http/1.1 and h2,h2-16,h2-14,spdy/3.1,http/1.1, but the warning messages are still there.\r\nnpn-list=spdy/3.1,h2,h2-16,h2-14,http/1.1\r\n>\r\nadd-x-forwarded-for=yes\r\nno-via=yes\r\n>\r\naccesslog-file=/var/log/nghttpx/access.log\r\naccesslog-format=[$time_iso8601] $remote_addr:$remote_port->:$server_port $body_bytes_sent $request_time $status $alpn \"$request\" \"$http_user_agent\"\r\n>\r\nerrorlog-file=/var/log/nghttpx/error.log\r\n\r\n\r\nThis is /etc/sysctl.conf:\r\n>\r\nfs.file-max=655350\r\n>\r\nnet.core.wmem_default=8388608\r\nnet.core.rmem_default=8388608\r\nnet.core.rmem_max=16777216\r\nnet.core.wmem_max=16777216\r\nnet.core.optmem_max=204800\r\nnet.core.somaxconn=65535\r\nnet.core.netdev_max_backlog=655350\r\n>\r\nnet.ipv4.tcp_max_orphans=655350\r\nnet.ipv4.tcp_max_syn_backlog=655350\r\nnet.ipv4.tcp_timestamps=0\r\nnet.ipv4.tcp_tw_reuse=1\r\nnet.ipv4.tcp_max_tw_buckets=655350\r\nnet.ipv4.tcp_syncookies=1\r\nnet.ipv4.tcp_syn_retries=2\r\nnet.ipv4.tcp_synack_retries=2\r\n>\r\nnet.ipv4.ip_local_port_range=10000 60000\r\nnet.ipv4.ip_local_reserved_ports=20000-30000\r\nnet.ipv4.ip_forward=1\r\n>\r\nnet.ipv4.conf.lo.arp_announce=2\r\nnet.ipv4.conf.all.rp_filter=0\r\nnet.ipv4.conf.default.rp_filter=0\r\nnet.ipv4.conf.default.arp_announce=2\r\nnet.ipv4.conf.all.arp_announce=2\r\n>\r\nnet.ipv4.tcp_fin_timeout=2\r\nnet.ipv4.tcp_keepalive_time=120\r\nnet.ipv4.tcp_sack=0\r\nnet.ipv4.tcp_rmem=4096  87380  4194304\r\nnet.ipv4.tcp_wmem=4096  16384  4194304\r\nnet.ipv4.tcp_mem=196608  262144  393216\r\n>\r\nnet.ipv4.udp_wmem_min=4096\r\nnet.ipv4.udp_rmem_min=4096\r\nnet.ipv4.udp_mem=772704  1030272  1545408\r\n>\r\nnet.ipv4.neigh.default.gc_stale_time=120\r\n>\r\nnet.ipv4.igmp_max_memberships = 20\r\n>\r\nnet.ipv6.conf.all.disable_ipv6=1\r\nnet.ipv6.conf.default.disable_ipv6=1\r\nnet.ipv6.conf.lo.disable_ipv6=1\r\n>\r\nnet.nf_conntrack_max=25000000\r\nnet.netfilter.nf_conntrack_max=25000000\r\n\r\n\r\nAnd this is /etc/security/limits.conf:\r\n\r\n>\r\n\\* hard nofile 409600\r\n\\* soft nofile 409600\r\n\\* hard noproc 409600\r\n\\* soft noproc 409600\r\n\r\n\r\nI wanna know what's wrong with nghttpx about the warnings, and how to fix it?\r\nThank you!\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/nghttp2/nghttp2/issues/comments/241783652","html_url":"https://github.com/nghttp2/nghttp2/issues/662#issuecomment-241783652","issue_url":"https://api.github.com/repos/nghttp2/nghttp2/issues/662","id":241783652,"user":{"login":"joeshae","id":6749515,"avatar_url":"https://avatars.githubusercontent.com/u/6749515?v=3","gravatar_id":"","url":"https://api.github.com/users/joeshae","html_url":"https://github.com/joeshae","followers_url":"https://api.github.com/users/joeshae/followers","following_url":"https://api.github.com/users/joeshae/following{/other_user}","gists_url":"https://api.github.com/users/joeshae/gists{/gist_id}","starred_url":"https://api.github.com/users/joeshae/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/joeshae/subscriptions","organizations_url":"https://api.github.com/users/joeshae/orgs","repos_url":"https://api.github.com/users/joeshae/repos","events_url":"https://api.github.com/users/joeshae/events{/privacy}","received_events_url":"https://api.github.com/users/joeshae/received_events","type":"User","site_admin":false},"created_at":"2016-08-23T16:01:24Z","updated_at":"2016-08-23T16:01:24Z","body":"CPU usage in product environment:\r\nnghttpx ( compiled with openssl-1.0.2h ): 20% - 40%\r\nsquid (8 workers == 8 processes ): 20% - 30%"}}
{"actor":{"display_login":"joeshae"}}