With the rapid development of computer vision, vision cameras have been used as noncontact sensors for structural displacement measurements. However, vision-based techniques are limited to short-term displacement measurements because of their degraded performance under varying illumination and inability to operate at night. To overcome these limitations, this study developed a continuous structural displacement estimation technique by combining measurements from an accelerometer with vision and infrared (IR) cameras collocated at the displacement estimation point of a target structure. The proposed technique enables continuous displacement estimation for both day and night, automatic optimization of the temperature range of an infrared camera to ensure a region of interest (ROI) with good matching features, and adaptive updating of the reference frame to achieve robust illumination–displacement estimation from vision/IR measurements. The performance of the proposed method was verified through lab-scale tests on a single-story building model. The displacements were estimated with a root-mean-square error of less than 2 mm compared with the laser-based ground truth. In addition, the applicability of the IR camera for displacement estimation under field conditions was validated using a pedestrian bridge test. The proposed technique eliminates the need for a stationary sensor installation location by the on-site installation of sensors and is therefore attractive for long-term continuous monitoring. However, it only estimates displacement at the sensor installation location, and cannot simultaneously estimate multi-point displacements which can be achieved by installing cameras off-site.