methods and apparatus are provided for non - deterministic incremental program replay using checkpoints and syndrome tracking . replay of a program proceeds by , for a given execution of the program , recording one or more checkpoints of the program , the one or more checkpoints containing program state information ; and a recorded list of values for one or more identified variables executing in one or more threads of the program . thereafter , during a replay execution of the program , the process continues by commencing execution from a particular one of the recorded checkpoints ; restoring the program state information associated with the particular one of the recorded checkpoints ; comparing an observed list of values to the recorded list of values for the one or more identified variables executing in each of the one or more threads ; and identifying a difference between the observed list of values and the recorded list of values a perturbation or suspend statement can optionally be introduced into the replay execution of the program .