The concept to save (i.e. checkpoint / dump) the state of a process, at a certain point in time, so that it may later be used to restore / restart the process (to the exact same state) has existed for many
The concept to save (i.e. checkpoint / dump) the state of a process, at a certain point in time, so that it may later be used to restore / restart the process (to the exact same state) has existed for many years. One of the most prominent motivations to develop and support checkpoint/restore functionality was to provide improved fault tolerance. For example, checkpoint/restore allows for processes to be restored from previously created checkpoints if, for one reason or another, these processes had been aborted.
Over the years there have been several different implementations of checkpoint/restore for Linux. Existing implementations of checkpoint/restore differ in terms of “what level” (of the operating system) they are operating; the lowest level approaches focus on implementing checkpoint/restore directly in the kernel while other “higher level” approaches implement checkpoint/restore completely in user-space. While it would be difficult to unearth each and every approach / implementation – it is likely fair to assume that the various permutations of checkpoint/restore have covered nearly all possible “levels” (i.e. from “completely in kernel” to “completely in user-space”).
Checkpoint/Restore in Linux
The closer to the kernel the checkpoint/restore is implemented – the more transparent it can be. What does “more transparent” mean? More transparent means that it has less requirements on the processes being checkpointed. For example, requirements can include things like: pre-loading special libraries (which could then be used to intercept system calls). It can also mean that an application has to be re-compiled to be able to be checkpointed. These kinds of pre-requisites make the usage of a checkpoint/restore implementation more difficult as it means that before starting some processes it must already be known if these processes will ever be checkpointed (…at some arbitrary point in “the future”).
Full transparency would likely be difficult to implement as it could require massive changes in the Linux kernel and such changes would have to be accepted by the Linux kernel community.
So… after many approaches to implement checkpoint/restore… a new approach started in 2011. This approach was named Checkpoint/Restore in Userspace (CRIU) and although it is named “in Userspace” it actually is both the user space and the kernel space. CRIU uses existing Linux kernel interfaces as far as possible; only extending existing interfaces / introducing new interfaces as needed. These required changes were (thankfully) all accepted by the Linux kernel community as most could also be used in other cases were more detailed information about a running process could be useful.
When CRIU is used to checkpoint a process it uses the existing and extended interfaces to the Linux kernel to collect as much information about the process. Using the PTRACE interface CRIU takes control over the process (CRIU actually always operates on a process tree; one process and all its child processes) and pauses that process. In the next step code is injected via PTRACE in the paused process. CRIU calls this code parasite-code. The parasite-code then runs from within the process’s address space and can access and dump/save/checkpoint the memory content of the process. Once all memory pages and all additional information have been collected (and possibly written to a directory) the process can either continue running or it can be aborted. Letting the process continue to run is something to expect in a fault tolerance scenario – to migrate a process to another system the process would be aborted.
To restore a process CRIU transforms the CRIU process doing the restore into the process to be restored. The is one of the places where CRIU uses a newly introduced Linux kernel interface. With CRIU a process can only be restored with the same process identifier (PID) the process had during checkpoint. To influence which PID the restored process gets, CRIU writes the PID of the to be restored process minus one to /proc/sys/kernel/ns_last_pid. CRIU then verifies if the newly created process actually has the desired PID. If not – the process restoration is aborted. Another example of steps performed during restore are file descriptors, whic