<p>I'm experiencing intermittent failed to response when make an outbound connection such as RPC call, it is logged by my application (Java) like this : </p>

<p><code>org.apache.http.NoHttpResponseException: RPC_SERVER.com:443 failed to respond !</code></p>

<p><strong>Outbound connection flow</strong></p>

<p>Kubernetes Node -> ELB for internal NGINX -> internal NGINX ->[Upstream To]-> ELB RPC server -> RPC server instance</p>

<p>This problem is not occurred on usual EC2 (AWS). </p>

<p>I'm able to reproduce on my localhost by doing this</p>

<ol>
<li>Run main application which act as client in port 9200</li>
<li>Run RPC server in port 9205</li>
<li>Client will make a connection to server using port 9202</li>
<li>Run <code>$ socat TCP4-LISTEN:9202,reuseaddr TCP4:localhost:9205</code> that will listen on port 9202 and then forward it to 9205 (RPC Server)</li>
<li>Add rules on iptables using <code>$ sudo iptables -A INPUT -p tcp --dport 9202 -j DROP</code></li>
<li>Trigger a RPC calling, and it will return the same error message as I desrcibe before</li>
</ol>

<p><strong>Hypothesis</strong></p>

<p>Caused by NAT on kubernetes, as far as I know, NAT is using <code>conntrack</code>, <code>conntrack</code> and break the TCP connection if it was idle for some period of time, client will assume the connection is still established although it isn't. (CMIIW)</p>

<p>I also have tried scaling <code>kube-dns</code> into 10 replica, and the problem still occurred. </p>

<p><strong>Node Specification</strong></p>

<p>Use calico as network plugin</p>

<p><code>$ sysctl -a | grep conntrack</code></p>

<pre><code>net.netfilter.nf_conntrack_acct = 0
net.netfilter.nf_conntrack_buckets = 65536
net.netfilter.nf_conntrack_checksum = 1
net.netfilter.nf_conntrack_count = 1585
net.netfilter.nf_conntrack_events = 1
net.netfilter.nf_conntrack_expect_max = 1024
net.netfilter.nf_conntrack_generic_timeout = 600
net.netfilter.nf_conntrack_helper = 1
net.netfilter.nf_conntrack_icmp_timeout = 30
net.netfilter.nf_conntrack_log_invalid = 0
net.netfilter.nf_conntrack_max = 262144
net.netfilter.nf_conntrack_tcp_be_liberal = 0
net.netfilter.nf_conntrack_tcp_loose = 1
net.netfilter.nf_conntrack_tcp_max_retrans = 3
net.netfilter.nf_conntrack_tcp_timeout_close = 10
net.netfilter.nf_conntrack_tcp_timeout_close_wait = 3600
net.netfilter.nf_conntrack_tcp_timeout_established = 86400
net.netfilter.nf_conntrack_tcp_timeout_fin_wait = 120
net.netfilter.nf_conntrack_tcp_timeout_last_ack = 30
net.netfilter.nf_conntrack_tcp_timeout_max_retrans = 300
net.netfilter.nf_conntrack_tcp_timeout_syn_recv = 60
net.netfilter.nf_conntrack_tcp_timeout_syn_sent = 120
net.netfilter.nf_conntrack_tcp_timeout_time_wait = 120
net.netfilter.nf_conntrack_tcp_timeout_unacknowledged = 300
net.netfilter.nf_conntrack_timestamp = 0
net.netfilter.nf_conntrack_udp_timeout = 30
net.netfilter.nf_conntrack_udp_timeout_stream = 180
net.nf_conntrack_max = 262144
</code></pre>

<p><strong>Kubelet config</strong> </p>

<pre><code>[Service]
Restart=always
Environment="KUBELET_KUBECONFIG_ARGS=--kubeconfig=/etc/kubernetes/kubelet.conf --require-kubeconfig=true"
Environment="KUBELET_SYSTEM_PODS_ARGS=--pod-manifest-path=/etc/kubernetes/manifests --allow-privileged=true"
Environment="KUBELET_NETWORK_ARGS=--network-plugin=cni --cni-conf-dir=/etc/cni/net.d --cni-bin-dir=/opt/cni/bin"
Environment="KUBELET_DNS_ARGS=--cluster-dns=10.96.0.10 --cluster-domain=cluster.local"
Environment="KUBELET_AUTHZ_ARGS=--authorization-mode=Webhook --client-ca-file=/etc/kubernetes/pki/ca.crt"
Environment="KUBELET_CADVISOR_ARGS=--cadvisor-port=0"
Environment="KUBELET_CLOUD_ARGS=--cloud-provider=aws"
ExecStart=
ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_SYSTEM_PODS_ARGS $KUBELET_NETWORK_ARGS $KUBELET_DNS_ARGS $KUBELET_AUTHZ_ARGS $KUBELET_CADVISOR_ARGS $KUBELET_EXTRA_ARGS $KUBELET_CLOUD_ARGS
</code></pre>

<p><strong>Kubectl version</strong> </p>

<pre><code>Client Version: version.Info{Major:"1", Minor:"7", GitVersion:"v1.7.5", GitCommit:"17d7182a7ccbb167074be7a87f0a68bd00d58d97", GitTreeState:"clean", BuildDate:"2017-08-31T09:14:02Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"7", GitVersion:"v1.7.7", GitCommit:"8e1552342355496b62754e61ad5f802a0f3f1fa7", GitTreeState:"clean", BuildDate:"2017-09-28T23:56:03Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}
</code></pre>

<p><strong>Kube-proxy Log</strong></p>

<pre><code>W1004 05:34:17.400700       8 server.go:190] WARNING: all flags other than --config, --write-config-to, and --cleanup-iptables are deprecated. Please begin using a config file ASAP.
I1004 05:34:17.405871       8 server.go:478] Using iptables Proxier.
W1004 05:34:17.414111       8 server.go:787] Failed to retrieve node info: nodes "ip-172-30-1-20" not found
W1004 05:34:17.414174       8 proxier.go:483] invalid nodeIP, initializing kube-proxy with 127.0.0.1 as nodeIP
I1004 05:34:17.414288       8 server.go:513] Tearing down userspace rules.
I1004 05:34:17.443472       8 conntrack.go:98] Set sysctl 'net/netfilter/nf_conntrack_max' to 262144
I1004 05:34:17.443518       8 conntrack.go:52] Setting nf_conntrack_max to 262144
I1004 05:34:17.443555       8 conntrack.go:98] Set sysctl 'net/netfilter/nf_conntrack_tcp_timeout_established' to 86400
I1004 05:34:17.443584       8 conntrack.go:98] Set sysctl 'net/netfilter/nf_conntrack_tcp_timeout_close_wait' to 3600
I1004 05:34:17.443851       8 config.go:102] Starting endpoints config controller
I1004 05:34:17.443888       8 config.go:202] Starting service config controller
I1004 05:34:17.443890       8 controller_utils.go:994] Waiting for caches to sync for endpoints config controller
I1004 05:34:17.443916       8 controller_utils.go:994] Waiting for caches to sync for service config controller
I1004 05:34:17.544155       8 controller_utils.go:1001] Caches are synced for service config controller
I1004 05:34:17.544155       8 controller_utils.go:1001] Caches are synced for endpoints config controller
</code></pre>

<p><code>$ lsb_release -s -d</code>
Ubuntu 16.04.3 LTS</p>

<p>Check the value of <code>sysctl net.netfilter.nf_conntrack_tcp_timeout_close_wait</code> inside the pod that contains your program. It is possible that the value on the node that you listed (3600) isn't the same as the value inside the pod.</p>

<p>If the value in the pod is too small (e.g. 60), and your Java client half-closes the TCP connection with a FIN when it finishes transmitting, but the response takes longer than the close_wait timeout to arrive, nf_conntrack will lose the connection state and your client program will not receive the response.</p>

<p>You may need to change the behavior of the client program to not use a TCP half-close, OR modify the value of <code>net.netfilter.nf_conntrack_tcp_timeout_close_wait</code> to be larger. See <a href="https://kubernetes.io/docs/tasks/administer-cluster/sysctl-cluster/" rel="nofollow noreferrer">https://kubernetes.io/docs/tasks/administer-cluster/sysctl-cluster/</a>.</p>
