<p>I have a multi-node cluster setup. There are Kubernetes network policies defined for the pods in the cluster. I can access the services or pods using their clusterIP/podIP only from the node where the pod resides. For services with multiple pods, I cannot access the service from the node at all (I guess when the service directs the traffic to the pod with the resident node same as from where I am calling then the service will work).</p>
<p>Is this the expected behavior?
Is it a Kubernetes limitation or a security feature?
For debugging etc., we might need to access the services from the node. How can I achieve it?</p>

<p>No, it is not the expected behavior for Kubernetes. Pods should be accessible for all the nodes inside the same cluster through their internal IPs. <code>ClusterIP</code> service exposes the service on a cluster-internal IP and making it reachable from within the cluster - it  is basically set by default for all the service types, as stated in <a href="https://kubernetes.io/docs/concepts/services-networking/service/#publishing-services-service-types" rel="nofollow noreferrer">Kubernetes documentation</a>.</p>
<p>Services are <em>not</em> node-specific and they can point to a pod regardless of where it runs in the cluster at any given moment in time.  Also make sure that you are using the cluster-internal <code>port:</code> while trying to reach the services.  If you still can connect to the pod only from node where it is running, you might need to check if something is wrong with your networking  - e.g, check if UDP ports are blocked.</p>
<p><strong>EDIT</strong>: Concerning network policies - by default, a pod is non-isolated either for egress or ingress, i.e. if no <code>NetworkPolicy</code> resource is defined for the pod in Kubernetes, all traffic is allowed to/from this pod - so-called <code>default-allow</code> behavior. Basically, without network policies all pods are allowed to communicate with all other pods/services in the same cluster, as described above.
If one or more <code>NetworkPolicy</code> is applied to a particular pod, it will reject all traffic that is not explicitly allowed by that policies (meaning, <code>NetworkPolicy</code>that both selects the pod and has &quot;Ingress&quot;/&quot;Egress&quot; in its policyTypes) - <code>default-deny</code> behavior.</p>
<p>What is <a href="https://kubernetes.io/docs/concepts/services-networking/network-policies/" rel="nofollow noreferrer">more</a>:</p>
<blockquote>
<p>Network policies do not conflict; they are additive. If any policy or policies apply to a given pod for a given direction, the connections allowed in that direction from that pod is the union of what the applicable policies allow.</p>
</blockquote>
<p>So yes, it is expected behavior for Kubernetes <code>NetworkPolicy</code> -  when a pod is isolated for ingress/egress, the only allowed connections into/from the pod are those from the pod's node and those allowed by the connection list of  <code>NetworkPolicy</code> defined.
To be compatible with it, <a href="https://projectcalico.docs.tigera.io/security/calico-network-policy" rel="nofollow noreferrer">Calico network policy</a> follows the same behavior for Kubernetes pods.
<code>NetworkPolicy</code>  is applied  to pods within a particular namespace - either the same or different with the help of the <a href="https://kubernetes.io/docs/concepts/services-networking/network-policies/#behavior-of-to-and-from-selectors" rel="nofollow noreferrer">selectors</a>.</p>
<p>As for node specific policies - nodes can't be targeted  by their Kubernetes identities, instead CIDR notation should be used in form of <code>ipBlock</code> in pod/service <code>NetworkPolicy</code> -  particular <a href="https://kubernetes.io/docs/concepts/services-networking/network-policies/#behavior-of-to-and-from-selectors" rel="nofollow noreferrer">IP ranges</a> are selected to allow as ingress sources or egress destinations for pod/service.</p>
<p>Whitelisting Calico IP addresses for each node might seem to be a valid option in this case, please have a look at the similar issue described <a href="https://discuss.kubernetes.io/t/how-do-we-write-an-ingress-networkpolicy-object-in-kubernetes-so-that-calls-via-api-server-proxy-can-come-through/10142/4" rel="nofollow noreferrer">here</a>.</p>
