<p>Of all the concepts from Kubernetes, I find service working mechanism is the most difficult to understand </p>

<p>Here is what I imagine right now:</p>

<ul>
<li>kube-proxy in each node listen to any new service/endpoint in master API controller</li>
<li>If there is any new service/endpoint, it adds a rule to that node's iptables</li>
<li>For NodePort service, external client has to access new service through one of the node's ip and NodePort. The node will forward the request to the new service IP</li>
</ul>

<p>Is it correct? There are still a few things I'm still not clear:</p>

<ul>
<li>Are services lying within nodes? If so, can we ssh into nodes and inspect how services work? </li>
<li>Are service IPs virtual IPs and only accessible within nodes?</li>
</ul>

<p>Most of the diagrams that I see online draw services as crossing all nodes, which make it even more difficult to imagine</p>

<blockquote>
  <p>kube-proxy in each node listen to any new service/endpoint in master API controller</p>
</blockquote>

<p>Kubernetes uses etcd to share the current cluster configuration information across all nodes (including pods, services, deployments, etc.).</p>

<blockquote>
  <p>If there is any new service/endpoint, it adds a rule to that node's iptables</p>
</blockquote>

<p>Internally kubernetes has a so called Endpoint Controller that is responsible for modifying the DNS configuration of the virtual cluster network to make service endpoints available via DNS (and environment variables). </p>

<blockquote>
  <p>For NodePort service, external client has to access new service through one of the node's ip and NodePort. The node will forward the request to the new service IP</p>
</blockquote>

<p>Depending on the service type additional action is taken, e.g. to make a port available on the nodes through an automatically created clusterIP service for type nodePort. Or an external load balancer is created with the cloud provider, etc.</p>

<blockquote>
  <p>Are services lying within nodes? If so, can we ssh into nodes and inspect how services work?</p>
</blockquote>

<p>As explained, services are manifested in the cluster configuration, the endpoint controller as well as additional things, like the clusterIP services, load balancers, etc. I cannot see a need to ssh into nodes to inspect services. Typically interacting with the cluster api should be sufficient to investigate/update the service configuration.</p>

<blockquote>
  <p>Are service IPs virtual IPs and only accessible within nodes?</p>
</blockquote>

<p>Service IPs, like POD IPs are virtual and accessible from within the cluster network. There is a global allocation map in etcd that maintains the complete list that allows allocating unique new ones. For more information on the networking model read <a href="https://cloudnativelabs.github.io/post/2017-04-18-kubernetes-networking/" rel="nofollow noreferrer">this blog</a>.</p>

<p>For more detailed information see the docs for <a href="https://kubernetes.io/docs/concepts/overview/components/#etcd" rel="nofollow noreferrer">kubernetes components</a> and <a href="https://kubernetes.io/docs/concepts/services-networking/service/" rel="nofollow noreferrer">services</a>.</p>
