For staring infrared sensors in geostationary orbit, the clutter caused by the high-frequency jitter and low-frequency drift of the sensor line-of-sight (LOS) is the impact of background features, sensor parameters, LOS motion characteristics, and background suppression algorithms. In this paper, the spectra of LOS jitter caused by cryocoolers and momentum wheels are analyzed, and the time-related factors such as the jitter spectrum, the detector integration time, the frame period, and the temporal differencing background suppression algorithm are considered comprehensively; they are combined into a background-independent jitter-equivalent angle model. A jitter-caused clutter model in the form of multiplying the background radiation intensity gradient statistics by the jitter-equivalent angle is established. This model has good versatility and high efficiency and is suitable for the quantitative evaluation of clutter and the iterative optimization of sensor design. Based on satellite ground vibration experiments and on-orbit measured image sequences, the jitter-caused clutter and drift-caused clutter models are verified. The relative deviation between the model calculation and the actual measurement results is less than 20%.