Active mid-infrared (MIR) imagers capable of retrieving three-dimensional (3D) structure and reflectivity information are highly attractive in a wide range of biomedical and industrial applications. However, infrared 3D imaging at low-light levels is still challenging due to the deficiency of sensitive and fast MIR sensors. Here we propose and implement a MIR time-of-flight imaging system that operates at single-photon sensitivity and femtosecond timing resolution. Specifically, back-scattered infrared photons from a scene are optically gated by delay-controlled ultrashort pump pulses through nonlinear frequency upconversion. The upconverted images with time stamps are then recorded by a silicon camera to facilitate the 3D reconstruction with high lateral and depth resolutions. Moreover, an effective numerical denoiser based on spatiotemporal correlation allows us to reveal the object profile and reflectivity under photon-starving conditions with a detected flux below 0.05 photons/pixel/second. The presented MIR 3D imager features high detection sensitivity, precise timing resolution, and wide-field operation, which may open new possibilities in life and material sciences.