Acoustic sensors with ultrahigh sensitivity, broadband response, and high resolution are essential for high-precision nondestructive weak signal detection technology. In this paper, based on the size effect of an ultrahigh-quality (Q) calcium fluoride (CaF2) resonator, a weak acoustic signal is detected by the dispersive response regime in which an acoustic, elastic wave modulates the geometry and is converted to a resonance frequency shift. Through the structural design of the resonator, the sensitivity reaches 11.54 V/Pa at 10 kHz in the experiment. To our knowledge, the result is higher than that of other optical resonator acoustic sensors. We further detected a weak signal as low as 9.4 µPa/Hz1/2, which greatly improved the detection resolution. With a good directionality of 36.4 dB and a broadband frequency response range of 20 Hz–20 kHz, the CaF2resonator acoustic sensing system can not only acquire and reconstruct speech signals over a long distance but also accurately identify and separate multiple voices in noisy environments. This system shows high performance in weak sound detection, sound source localization, sleep monitoring, and many other voice interaction applications.