<p>I'm planning to upgrade kubernetes cluster version from v1.7.10 to v1.8.12. but as this <a href="https://github.com/kubernetes/kubernetes/issues/53644" rel="nofollow noreferrer">issue</a> state that all containers will be restart because of the spec hash change. So, what's the suggested upgrade procedure ? do I need to drain the node before upgrading kubelet version, or just do an in-place upgrade and let all the containers restarting ? what's the difference ? </p>

<p>Also, since upgrading to v1.9.0 will also cause containers restarting, can I upgrade v1.7.10 directly to v1.10.3 ? In this way I can avoid two time-consuming upgrades to v1.8 and v1.9 at least. Is there any constraints provent me from doing this ? </p>

<p>Any suggestion will be appreciated.</p>

<p>After some testing and research, I come to some conclusion:</p>

<ol>
<li><p>Drain the node is not a must. At least, drain can't evict daemonsets. But drain a node is recommend way to upgrade kubernetes, since this can at most extent reduce the impact to applications that's deployed using deployment.</p>

<p>Also, sometimes, drain a node is neccesary. For instance, from kubernetes v1.10, the log files of all pods have changed from /var/log/pods/pod-id/container_id.log to /var/log/pods/pod-id/container/id.log, so when upgrade to v1.10, all pods have to restart to use the new log file, otherwise, you can't access their logs through 'kubectl logs' command. At this time, drain a node is helpful, and to those pods that can't be evicted, like daemonsets, we have to restart them manually.</p></li>
<li><p>Skip minor version upgrading is not supported, especially in a HA setup, with multiple masters, which can be proved somehow by <a href="https://cloud.google.com/kubernetes-engine/docs/how-to/upgrading-a-cluster" rel="nofollow noreferrer">GKE's supported upgrades policy</a>. Also, skip minor version upgrading implies some risk sometimes. </p>

<p>Again, as an example, upgrade to v1.10. In 1.10, objects in the apps API group began persisting in etcd in apps/v1 format, which can be handled very well by v1.9, but v1.8 can't. So, when you upgrade kubernetes from v1.8 to 1.10, in a HA setup, some masters have been upgraded, some haven't, will bring some weird problems, like deployment/daemonset can't be handled properly, more info refer to <a href="https://stackoverflow.com/questions/50595823/kubernetes-v1-8-12-cant-list-extensions-deployment">my another question</a>. So, such upgrade should be avoid as long as possible.</p></li>
</ol>
