<p>We ran some stateful applications (e.g. database) on AWS on-demand/reserved ec2 instances in the past, now we are considering moving those app to k8s statefulset with PVC.</p>

<p>My question is that is it recommended to run k8s statefulset on spot instance to reduce the cost? Since we can use kube-spot-termination-notice-handler to taint the node to move the pod to others before the spot instance terminated, it looks like it should be no problem as long as the statefulset has multiple replicas to prevent the service interrupted.</p>

<p>IMO, I would not recommend running a critical StatefulSet on Spot Instances. For example, a critical database. This is some of what would/could happen in these examples:</p>

<ul>
<li><p>Mysql master/slave/clustered. Any node going down would lead to unpredictable errors and/or downtime before recovering, or nodes coming back up (with different IP addresses!)</p></li>
<li><p>Cassandra. Any node going up/down would cause your cluster to rebalance. If you have these going up and down, then they will constantly be rebalancing! Not to mention the fact that if you had all your nodes in Spot Instances you have the chance of most of them going down.</p></li>
</ul>

<p>Spots are great for large one-time batch jobs and that they are not critically time bound. These can be anything data processing or for example, creating or updating an M/L model.</p>

<p>They are also great for stateless services, meaning an application that sits behind a load balancer and uses state store that is not in a spot instance (Mysql, Cassandra, CloudSQL, RDS, etc)</p>

<p>Spots are also great for test/dev environments, again not necessarily time-bound jobs/workloads.</p>
