<p>We have had our database running on Kubernetes cluster (deployed to our private network) in Google cloud for a few months now.  Last week we noticed that for some reason the IP address of all underlying nodes (VMs) changed.  This caused an outage.  We have been using the NodePort configuration of Kubernetes for our service to access our database (<a href="https://kubernetes.io/docs/concepts/services-networking/service/#nodeport" rel="nofollow noreferrer">https://kubernetes.io/docs/concepts/services-networking/service/#nodeport</a>).
We understand that the IP address of the pods within the VMs are dynamic and will eventually  change, however we did not know that the IP address of the  actual nodes (VMs) may also change. Is this normal?  Does anyone know  what can cause a VM IP address change in a Kubernetes cluster?</p>

<p>From the documentation about <a href="https://cloud.google.com/compute/docs/ip-addresses/#ephemeraladdress" rel="nofollow noreferrer">Ephemeral IP Addresses</a> on GCP,</p>

<blockquote>
  <p>When you create an instance or forwarding rule without specifying an
  IP address, the resource is automatically assigned an ephemeral
  external IP address. Ephemeral external IP address are released from a
  resource if you delete the resource. For VM instances, if you stop the
  instance, the IP address is also released. Once you restart the
  instance, it is assigned a new ephemeral external IP address.</p>
</blockquote>

<p>You can assign static external IP addresses to instances, but as @Notauser mentioned, it is not recommended for Kubernetes nodes. This is because you may configure autoscaler for your instance groups and node sizes can be minimized or maximized.
Also, you need to reserve a static IP address for each node, which is not recommended. Moreover you will waste Static IP address resources and if the reserved static IP addresses are not used, you will still be charged for that.</p>

<p>Otherwise you can <a href="https://cloud.google.com/kubernetes-engine/docs/tutorials/http-balancer" rel="nofollow noreferrer">configure HTTP loadbalancer using ingress</a> and then reserve a static IP address for your load balancer. Instead of using NodePort you should use ClusterIP type services and create an ingress rule forwarding the traffic to those services.</p>
