Three-dimensional precise models of teeth are critical for a variety of dental procedures, including orthodontics, prosthodontics, and implantology. While X-ray-based imaging devices are commonly used to obtain anatomical information about teeth, optical devices offer a promising alternative for acquiring 3D data of teeth without exposing patients to harmful radiation. Previous research has not examined the optical interactions with all dental tissue compartments nor provided a thorough analysis of detected signals at various boundary conditions for both transmittance and reflectance modes. To address this gap, a GPU-based Monte Carlo (MC) method has been utilized to assess the feasibility of diffuse optical spectroscopy (DOS) systems operating at 633 nm and 1310 nm wavelengths for simulating light-tissue interactions in a 3D tooth model. The results show that the system’s sensitivity to detect pulp signals at both 633 nm and 1310 nm wavelengths is higher in the transmittance compared with that in the reflectance mode. Analyzing the recorded absorbance, reflectance, and transmittance data verified that surface reflection at boundaries can improve the detected signal, especially from the pulp region in both reflectance and transmittance DOS systems. These findings could ultimately lead to more accurate and effective dental diagnosis and treatment.