<p>I have created two seperate GKE clusters on K8s 1.14.10.</p>

<blockquote>
  <p><a href="https://stackoverflow.com/questions/58309735/vpn-access-to-in-house-network-not-working-after-gke-cluster-upgrade-to-1-14-6">VPN access to in-house network not working after GKE cluster upgrade to 1.14.6</a></p>
</blockquote>

<p>I have followed this and the IP masquerading agent documentation. 
I have tried to test this using  a client pod and server pod to exchange messages.
I'm using Internal node IP to send message and created a ClusterIP to expose the pods.</p>

<p>I have allowed requests for every instance in firewall rules for ingress and egress i.e <strong>0.0.0.0/0</strong>.
<a href="https://i.stack.imgur.com/VP7du.png" rel="nofollow noreferrer">Pic:This is the description of the cluster which I have created</a>
The config map of the IP masquerading agent stays the same as in the documentation.
I'm able to ping the other node from within the pod but curl request says connection refused and tcpdump shows no data.</p>

<p>Problem:
I need to communicate from cluster A to cluster B in gke 1.14 with ipmasquerading set to true. I either get connection refused or i/o timeout. I have tried using internal and external node IPs as well as using a loadbalancer.</p>

<p>You have provided quite general information and without details I cannot provide specific scenario answer. It might be related to how did you create clusters or other firewalls settings. Due to that I will provide correct steps to creation and configuration 2 clusters with firewall and <code>masquerade</code>. Maybe you will be able to find which step you missed or misconfigured.</p>
<p><strong>Clusters configuration (node,pods,svc) are on the bottom of the answer.</strong></p>
<p><strong>1. Create VPC and 2 clusters</strong></p>
<p>In docs it says about 2 different projects but you can do it in one project.
Good example of VPC creation and 2 clusters can be found in GKE docs. <a href="https://cloud.google.com/kubernetes-engine/docs/how-to/cluster-shared-vpc#creating_a_network_and_two_subnets" rel="nofollow noreferrer">Create VPC</a> and <a href="https://cloud.google.com/kubernetes-engine/docs/how-to/cluster-shared-vpc#creating_a_cluster_in_your_first_service_project" rel="nofollow noreferrer">Crate 2 clusters</a>. In cluster <code>Tier1</code> you can enable <code>NetworkPolicy</code> now instead of enabling it later.
After that you will need to create <a href="https://cloud.google.com/kubernetes-engine/docs/how-to/cluster-shared-vpc#console_8" rel="nofollow noreferrer">Firewall Rules</a>. You will also need to add <code>ICMP</code> protocol to firewall rule.</p>
<p>At this point you should be able to ping between nodes from 2 clusters.</p>
<p>For additional Firewall rules (allowing connection between pods, svc, etc) please check <a href="https://cloud.google.com/kubernetes-engine/docs/how-to/cluster-shared-vpc#creating_additional_firewall_rules" rel="nofollow noreferrer">this docs</a>.</p>
<p><strong>2. Enable <a href="https://cloud.google.com/kubernetes-engine/docs/how-to/ip-masquerade-agent" rel="nofollow noreferrer">IP masquerade agent</a></strong></p>
<p>As mentioned in docs, to run <code>IPMasquerade</code>:</p>
<blockquote>
<p>The ip-masq-agent DaemonSet is automatically installed as an add-on with --nomasq-all-reserved-ranges argument in a GKE cluster, if one or more of the following is true:</p>
</blockquote>
<blockquote>
<p>The cluster has a network policy.</p>
</blockquote>
<p><strong>OR</strong></p>
<blockquote>
<p>The Pod's CIDR range is not within 10.0.0.0/8.</p>
</blockquote>
<p>It mean that <code>tier-2-cluster</code> already have <code>ip-masq-agent</code> in <code>kube-system</code> namespace (because <code>The Pod's CIDR range is not within 10.0.0.0/8.</code>). And if you enabled <code>NetworkPolicy</code> during creation of <code>tier-1-cluster</code> it should be have also installed. If not, you will need to enable it using command:</p>
<p><code>$ gcloud container clusters update  tier-1-cluster --update-addons=NetworkPolicy=ENABLED --zone=us-central1-a</code></p>
<p>To verify if everything is ok you have to check if <code>Daemonset ip-masq-agent</code> pods were created. (Each pod for node).</p>
<pre><code>$ kubectl get ds ip-masq-agent -n kube-system
NAME            DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR                                 AGE
ip-masq-agent   3         3         3       3            3           beta.kubernetes.io/masq-agent-ds-ready=true   168m
</code></pre>
<p>If you will SSH to any of your nodes you will be able to see default <code>iptables</code> entries.</p>
<pre><code>$ sudo iptables -t nat -L IP-MASQ
Chain IP-MASQ (1 references)
target     prot opt source               destination         
RETURN     all  --  anywhere             169.254.0.0/16       /* ip-masq: local traffic is not subject to MASQUERADE */
RETURN     all  --  anywhere             10.0.0.0/8           /* ip-masq: RFC 1918 reserved range is not subject to MASQUERADE */
RETURN     all  --  anywhere             172.16.0.0/12        /* ip-masq: RFC 1918 reserved range is not subject to MASQUERADE */
RETURN     all  --  anywhere             192.168.0.0/16       /* ip-masq: RFC 1918 reserved range is not subject to MASQUERADE */
RETURN     all  --  anywhere             240.0.0.0/4          /* ip-masq: RFC 5735 reserved range is not subject to MASQUERADE */
RETURN     all  --  anywhere             192.0.2.0/24         /* ip-masq: RFC 5737 reserved range is not subject to MASQUERADE */
RETURN     all  --  anywhere             198.51.100.0/24      /* ip-masq: RFC 5737 reserved range is not subject to MASQUERADE */
RETURN     all  --  anywhere             203.0.113.0/24       /* ip-masq: RFC 5737 reserved range is not subject to MASQUERADE */
RETURN     all  --  anywhere             100.64.0.0/10        /* ip-masq: RFC 6598 reserved range is not subject to MASQUERADE */
RETURN     all  --  anywhere             198.18.0.0/15        /* ip-masq: RFC 6815 reserved range is not subject to MASQUERADE */
RETURN     all  --  anywhere             192.0.0.0/24         /* ip-masq: RFC 6890 reserved range is not subject to MASQUERADE */
RETURN     all  --  anywhere             192.88.99.0/24       /* ip-masq: RFC 7526 reserved range is not subject to MASQUERADE */
MASQUERADE  all  --  anywhere             anywhere             /* ip-masq: outbound traffic is subject to MASQUERADE (must be last in chain) */
</code></pre>
<p><strong>3. Deploy test application</strong></p>
<p>I've used Hello application from <a href="https://cloud.google.com/kubernetes-engine/docs/how-to/load-balance-ingress" rel="nofollow noreferrer">GKE docs</a> and deployed on both Clusters. In addition I have also deployed ubuntu image for tests.</p>
<p><strong>4. Apply proper configuration for IPMasquerade</strong>
This config need to be on the <code>source</code> cluster.</p>
<p>In short, if destination CIDR is in <code>nonMasqueradeCIDRs:</code>, it will show it internal IP, otherwise it will show NodeIP as source.</p>
<p>Save to file <code>config</code> below text:</p>
<pre><code>nonMasqueradeCIDRs:
  - 10.0.0.0/8
resyncInterval: 2s
masqLinkLocal: true
</code></pre>
<p>Create IPMasquarade ConfigMap</p>
<pre><code>$ kubectl create configmap ip-masq-agent --from-file config --namespace kube-system
</code></pre>
<p>It will overwrite <code>iptables</code> configuration</p>
<pre><code>$ sudo iptables -t nat -L IP-MASQ
Chain IP-MASQ (2 references)
target     prot opt source               destination         
RETURN     all  --  anywhere             10.0.0.0/8           /* ip-masq-agent: local traffic is not subject to MASQUERADE */
MASQUERADE  all  --  anywhere             anywhere             /* ip-masq-agent: outbound traffic is subject to MASQUERADE (must be last in chain) */
</code></pre>
<p><strong>5. Tests:</strong></p>
<p><strong>When IP is Masqueraded</strong></p>
<p>SSH to Node form <code>Tier2</code> cluster and run:</p>
<pre><code>sudo toolbox bash
apt-get update
apt install -y tcpdump
</code></pre>
<p>Now you should listen using command below. Port <code>32502</code> is <code>NodePort</code> service from <code>Tier 2</code> Cluster</p>
<pre><code>tcpdump -i eth0 -nn -s0 -v port 32502
</code></pre>
<p>In Cluster <code>Tier1</code> you need to enter ubuntu pod and curl <code>NodeIP:NodePort</code></p>
<pre><code>$ kubectl exec -ti ubuntu -- bin/bash 
</code></pre>
<p>You will need to install curl <code>apt-get install curl</code>.</p>
<p>curl NodeIP:NodePort (Node which is listening, NodePort from service from Cluster Tier 2).</p>
<p><strong>CLI:</strong></p>
<pre><code>root@ubuntu:/# curl 172.16.4.3:32502
Hello, world!
Version: 2.0.0
Hostname: hello-world-deployment-7f67f479f5-h4wdm
</code></pre>
<p>On Node you can see entry like:</p>
<pre><code>tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
12:53:30.321641 IP (tos 0x0, ttl 63, id 25373, offset 0, flags [DF], proto TCP (6), length 60)
    10.0.4.4.56018 &gt; 172.16.4.3.32502: Flags [S], cksum 0x8648 (correct), seq 3001889856
</code></pre>
<p><code>10.0.4.4</code> is <code>NodeIP</code> where Ubuntu pod is located.</p>
<p><strong>When IP was not Masqueraded</strong></p>
<p>Remove <code>ConfigMap</code> from Cluster Tier 1</p>
<pre><code>$ kubectl delete cm ip-masq-agent -n kube-system
</code></pre>
<p>Change in file <code>config</code> CIDR to <code>172.16.4.0/22</code> which is <code>Tier 2</code> nodes pool and reapply CM</p>
<pre><code>$ kubectl create configmap ip-masq-agent --from-file config --namespace kube-system
</code></pre>
<p>SSH to any node from Tier 1 to check if <code>iptables rules</code> were changed.</p>
<pre><code>sudo iptables -t nat -L IP-MASQ
Chain IP-MASQ (2 references)
target     prot opt source               destination         
RETURN     all  --  anywhere             172.16.4.0/22        /* ip-masq-agent: local traffic is not subject to MASQUERADE */
MASQUERADE  all  --  anywhere             anywhere             /* ip-masq-agent: outbound traffic is subject to MASQUERADE (must be last in chain) */
</code></pre>
<p>Now for test I have again used Ubuntu pod and curl the same ip like before.</p>
<pre><code>tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
13:16:50.316234 IP (tos 0x0, ttl 63, id 53160, offset 0, flags [DF], proto TCP (6), length 60)
    10.4.2.8.57876 &gt; 172.16.4.3.32502
</code></pre>
<p><code>10.4.2.8</code> is internal IP of Ubuntu pod.</p>
<p><strong>Configuration for Tests:</strong></p>
<p><strong>TIER1</strong></p>
<pre><code>NAME                                          READY   STATUS    RESTARTS   AGE   IP         NODE                                            NOMINATED NODE   READINESS GATES
pod/hello-world-deployment-7f67f479f5-b2qqz   1/1     Running   0          15m   10.4.1.8   gke-tier-1-cluster-default-pool-e006097b-5tnj   &lt;none&gt;           &lt;none&gt;
pod/hello-world-deployment-7f67f479f5-shqrt   1/1     Running   0          15m   10.4.2.5   gke-tier-1-cluster-default-pool-e006097b-lfvh   &lt;none&gt;           &lt;none&gt;
pod/hello-world-deployment-7f67f479f5-x7jvr   1/1     Running   0          15m   10.4.0.8   gke-tier-1-cluster-default-pool-e006097b-1wbf   &lt;none&gt;           &lt;none&gt;
ubuntu                                    1/1     Running   0          91s   10.4.2.8   gke-tier-1-cluster-default-pool-e006097b-lfvh   &lt;none&gt;           &lt;none&gt;

NAME                  TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)           AGE    SELECTOR
service/hello-world   NodePort    10.0.36.46   &lt;none&gt;        60000:31694/TCP   14m    department=world,greeting=hello
service/kubernetes    ClusterIP   10.0.32.1    &lt;none&gt;        443/TCP           115m   &lt;none&gt;

NAME                                                 STATUS   ROLES    AGE    VERSION           INTERNAL-IP   EXTERNAL-IP     OS-IMAGE                             KERNEL-VERSION   CONTAINER-RUNTIME
node/gke-tier-1-cluster-default-pool-e006097b-1wbf   Ready    &lt;none&gt;   115m   v1.14.10-gke.36   10.0.4.2      35.184.38.21    Container-Optimized OS from Google   4.14.138+        docker://18.9.7
node/gke-tier-1-cluster-default-pool-e006097b-5tnj   Ready    &lt;none&gt;   115m   v1.14.10-gke.36   10.0.4.3      35.184.207.20   Container-Optimized OS from Google   4.14.138+        docker://18.9.7
node/gke-tier-1-cluster-default-pool-e006097b-lfvh   Ready    &lt;none&gt;   115m   v1.14.10-gke.36   10.0.4.4      35.226.105.31   Container-Optimized OS from Google   4.14.138+        docker://18.9.7&lt;none&gt;   100m   v1.14.10-gke.36   10.0.4.4      35.226.105.31   Container-Optimized OS from Google   4.14.138+        docker://18.9.7
</code></pre>
<p><strong>TIER2</strong></p>
<pre><code>$ kubectl get pods,svc,nodes -o wide
NAME                                          READY   STATUS    RESTARTS   AGE   IP           NODE                                            NOMINATED NODE   READINESS GATES
pod/hello-world-deployment-7f67f479f5-92zvk   1/1     Running   0          12m   172.20.1.5   gke-tier-2-cluster-default-pool-57b1cc66-xqt5   &lt;none&gt;           &lt;none&gt;
pod/hello-world-deployment-7f67f479f5-h4wdm   1/1     Running   0          12m   172.20.1.6   gke-tier-2-cluster-default-pool-57b1cc66-xqt5   &lt;none&gt;           &lt;none&gt;
pod/hello-world-deployment-7f67f479f5-m85jn   1/1     Running   0          12m   172.20.1.7   gke-tier-2-cluster-default-pool-57b1cc66-xqt5   &lt;none&gt;           &lt;none&gt;

NAME                  TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)           AGE    SELECTOR
service/hello-world   NodePort    172.16.24.206   &lt;none&gt;        60000:32502/TCP   12m    department=world,greeting=hello
service/kubernetes    ClusterIP   172.16.16.1     &lt;none&gt;        443/TCP           113m   &lt;none&gt;

NAME                                                 STATUS   ROLES    AGE    VERSION           INTERNAL-IP   EXTERNAL-IP      OS-IMAGE                             KERNEL-VERSION   CONTAINER-RUNTIME
node/gke-tier-2-cluster-default-pool-57b1cc66-84ng   Ready    &lt;none&gt;   112m   v1.14.10-gke.36   172.16.4.2    35.184.118.151   Container-Optimized OS from Google   4.14.138+        docker://18.9.7

node/gke-tier-2-cluster-default-pool-57b1cc66-mlmn   Ready    &lt;none&gt;   112m   v1.14.10-gke.36   172.16.4.3    35.238.231.160   Container-Optimized OS from Google   4.14.138+        docker://18.9.7

node/gke-tier-2-cluster-default-pool-57b1cc66-xqt5   Ready    &lt;none&gt;   112m   v1.14.10-gke.36   172.16.4.4    35.202.94.194    Container-Optimized OS from Google   4.14.138+        docker://18.9.7
</code></pre>
