<pre><code>[Interface:vetha13c9067] 05:57:57.851421 IP 10.28.0.7 &gt; 172.28.10.17: ICMP echo request, id 56, seq 1, length 64
[Interface:cbr0] 05:57:57.851421 IP 10.28.0.7 &gt; 172.28.10.17: ICMP echo request, id 56, seq 1, length 64
[Interface:eth0] 05:57:57.851614 IP 172.18.0.25 &gt; 172.28.10.17: ICMP echo request, id 56, seq 1, length 64
</code></pre>

<p>The above log is without any trouble <code>10.28.0.7</code> can communicate with 172.28.10.17 successfully</p>

<pre><code>[Interface:veth916b4093] 05:57:09.699334 IP 10.20.4.194 &gt; 172.28.10.17: ICMP echo request, id 28, seq 1, length 64
[Interface:cbr0] 05:57:09.699334 IP 10.20.4.194 &gt; 172.28.10.17: ICMP echo request, id 28, seq 1, length 64
[Interface:eth0] 05:57:09.699380 IP 10.20.4.194 &gt; 172.28.10.17: ICMP echo request, id 28, seq 1, length 64
</code></pre>

<p>And the above log is with some trouble, as you can see its eth0 interface request from still <code>10.20.4.194</code> not <code>172.18.0.0/16</code> in which its subnet range.</p>

<p><strong>why the request in my pod isn't from subnet range(172.18.0.0/16)?</strong> </p>

<p>below are my current configs:</p>

<pre><code>bash-5.0# ip a
1: lo: &lt;LOOPBACK,UP,LOWER_UP&gt; mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
3: eth0@if135: &lt;BROADCAST,MULTICAST,UP,LOWER_UP&gt; mtu 1460 qdisc noqueue state UP group default
    link/ether 76:f9:ea:bf:d7:1d brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 10.20.4.194/24 scope global eth0
       valid_lft forever preferred_lft forever
bash-5.0# netstat -rn
Kernel IP routing table
Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface
0.0.0.0         10.20.4.1       0.0.0.0         UG        0 0          0 eth0
10.20.4.0       0.0.0.0         255.255.255.0   U         0 0          0 eth0
bash-5.0# ip neigh sh
10.20.4.1 dev eth0 lladdr 92:7f:08:52:f9:d4 STALE
bash-5.0# cat /etc/resolv.conf
nameserver 10.85.0.10
search default.svc.cluster.local svc.cluster.local cluster.local c.buzzdata.internal google.internal
options ndots:5
bash-5.0# nslookup kubernetes
Server:     10.85.0.10
Address:    10.85.0.10#53

Name:   kubernetes.default.svc.cluster.local
Address: 10.85.0.1
</code></pre>

<pre><code>╰─ k describe po netshoot-container
Name:               netshoot-container
Namespace:          default
Priority:           0
PriorityClassName:  &lt;none&gt;
Node:               gke-search-cluster-pool-765be39a-gkt4/172.18.0.17
Start Time:         Mon, 16 Mar 2020 14:05:32 +0900
Labels:             run=netshoot-container
Annotations:        kubernetes.io/limit-ranger: LimitRanger plugin set: cpu request for container netshoot-container
Status:             Running
IP:                 10.20.4.194
Containers:
  netshoot-container:
    Container ID:  docker://0df9d4b262f926d7d89a42f58de672284a5cb5637ab951b752e0c8b34ded676a
    Image:         nicolaka/netshoot
    Image ID:      docker-pullable://nicolaka/netshoot@sha256:99d15e34efe1e3c791b0898e05be676084638811b1403fae59120da4109368d4
    Port:          &lt;none&gt;
    Host Port:     &lt;none&gt;
    Args:
      /bin/bash
    State:          Running
      Started:      Mon, 16 Mar 2020 14:05:36 +0900
    Ready:          True
    Restart Count:  0
    Requests:
      cpu:        100m
    Environment:  &lt;none&gt;
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-xxxx (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             True
  ContainersReady   True
  PodScheduled      True
Volumes:
  default-token-xxxx:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-xxxx
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  kubernetes.io/hostname=gke-xxxx-cluster-pool-xxxx-gkt4
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:          &lt;none&gt;
</code></pre>

<pre><code>Chain INPUT (policy DROP)
target     prot opt source               destination         
KUBE-SERVICES  all  --  anywhere             anywhere             ctstate NEW /* kubernetes service portals */
KUBE-EXTERNAL-SERVICES  all  --  anywhere             anywhere             ctstate NEW /* kubernetes externally-visible service portals */
KUBE-FIREWALL  all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere             state RELATED,ESTABLISHED
ACCEPT     all  --  anywhere             anywhere            
ACCEPT     icmp --  anywhere             anywhere            
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:ssh
ACCEPT     tcp  --  anywhere             anywhere            
ACCEPT     udp  --  anywhere             anywhere            
ACCEPT     icmp --  anywhere             anywhere            
ACCEPT     sctp --  anywhere             anywhere            

Chain FORWARD (policy DROP)
target     prot opt source               destination         
KUBE-FORWARD  all  --  anywhere             anywhere             /* kubernetes forwarding rules */
KUBE-SERVICES  all  --  anywhere             anywhere             ctstate NEW /* kubernetes service portals */
DOCKER-USER  all  --  anywhere             anywhere            
DOCKER-ISOLATION-STAGE-1  all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere             ctstate RELATED,ESTABLISHED
DOCKER     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere            
ACCEPT     tcp  --  anywhere             anywhere            
ACCEPT     udp  --  anywhere             anywhere            
ACCEPT     icmp --  anywhere             anywhere            
ACCEPT     sctp --  anywhere             anywhere            

Chain OUTPUT (policy DROP)
target     prot opt source               destination         
KUBE-SERVICES  all  --  anywhere             anywhere             ctstate NEW /* kubernetes service portals */
KUBE-FIREWALL  all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere             state NEW,RELATED,ESTABLISHED
ACCEPT     all  --  anywhere             anywhere            

Chain DOCKER (1 references)
target     prot opt source               destination         

Chain DOCKER-ISOLATION-STAGE-1 (1 references)
target     prot opt source               destination         
DOCKER-ISOLATION-STAGE-2  all  --  anywhere             anywhere            
RETURN     all  --  anywhere             anywhere            

Chain DOCKER-ISOLATION-STAGE-2 (1 references)
target     prot opt source               destination         
DROP       all  --  anywhere             anywhere            
RETURN     all  --  anywhere             anywhere            

Chain DOCKER-USER (1 references)
target     prot opt source               destination         
RETURN     all  --  anywhere             anywhere            

Chain KUBE-EXTERNAL-SERVICES (1 references)
target     prot opt source               destination         

Chain KUBE-FIREWALL (2 references)
target     prot opt source               destination         
DROP       all  --  anywhere             anywhere             /* kubernetes firewall for dropping marked packets */ mark match 0x8000/0x8000

Chain KUBE-FORWARD (1 references)
target     prot opt source               destination         
ACCEPT     all  --  anywhere             anywhere             /* kubernetes forwarding rules */ mark match 0x4000/0x4000
ACCEPT     all  --  10.20.0.0/14         anywhere             /* kubernetes forwarding conntrack pod source rule */ ctstate RELATED,ESTABLISHED
ACCEPT     all  --  anywhere             10.20.0.0/14         /* kubernetes forwarding conntrack pod destination rule */ ctstate RELATED,ESTABLISHED

Chain KUBE-SERVICES (3 references)
target     prot opt source               destination         
</code></pre>

0xF4D3C0D3
<p>I found a cause, that's because IP-MASQ is enabled.</p>

<p>If you're suffering from the same trouble that your pod can't communicate with private subnet like 172.x.x.x then check it out your node's iptable rules by <code>iptables -t nat -L</code></p>

<p>if you can see IP-MASQ rule and if there are <code>RETURN     all  --  anywhere             172.16.0.0/12        /* ip-masq: RFC 1918 reserved range is not subject to MASQUERADE */
</code> then below out!</p>

<p><a href="https://cloud.google.com/kubernetes-engine/docs/how-to/ip-masquerade-agent" rel="nofollow noreferrer">https://cloud.google.com/kubernetes-engine/docs/how-to/ip-masquerade-agent</a></p>

<p>modifying iptable manually can be one of the solutions, but I think it's better that follow the documentation.</p>
