<p>I have the following alert configured in prometheus:</p>

<pre><code>alert: ClockSkewDetected
expr: abs(node_timex_offset_seconds{job="node-exporter"})
  &gt; 0.03
for: 2m
labels:
  severity: warning
annotations:
  message: Clock skew detected on node-exporter {{ $labels.namespace }}/{{ $labels.pod }}. Ensure NTP is configured correctly on this host.
</code></pre>

<p>This alert is <a href="https://github.com/coreos/kube-prometheus/blob/773a9c73e9e1a817d079cf6db13603a1cb830c8b/jsonnet/kube-prometheus/alerts/node.libsonnet#L38-L50" rel="nofollow noreferrer">part of</a> the default <code>kube-prometheus</code> stack which I am using.</p>

<p>I find this alert fires for around 10 mins every day or two.</p>

<p>I'd like to know how to deal with this problem (the alert firing!). It's suggested in <a href="https://stackoverflow.com/questions/48792742/how-do-i-set-the-correct-time-in-my-google-kubernetes-engine-container-do-i-nee">this answer</a> that I shouldn't need to run NTP (via a daemonset I guess) myself on GKE.</p>

<p>I'm also keen to use the <code>kube-prometheus</code> defaults where possible - so I'm unsure about increasing the <code>0.03</code> value.</p>

<p>As pointed in the answer, <a href="https://cloud.google.com/compute/docs/instances/configure-ntp" rel="nofollow noreferrer">instances in GCP are preconfigured to have their own NTP server synced</a>, so there shouldn't be any need to use DaemonSets to manually configure them.</p>
<p>It might be the case that the clock is skewing on <a href="https://cloud.google.com/compute/docs/instances/live-migration" rel="nofollow noreferrer">live migrations</a> and it catches up automatically but not without triggering the alert. However, this theory only applies for non-preemptible instances.</p>
<p><a href="https://github.com/GoogleCloudPlatform/compute-image-packages/issues/634" rel="nofollow noreferrer">Some events on GCE instances are supposed to trigger the Clock Skew Daemon</a> that will eventually correct changes initiated by the user (or a process action on behalf of the user), so if this is happening in your nodes, that's another possibility.</p>
<p>Regardless of the aforementioned theories and since nodes are managed resources in GKE, I think you have a pretty solid case for the <a href="https://cloud.google.com/support/" rel="nofollow noreferrer">GKE support</a> to investigate as this might be an implementation detail.</p>
