The mathematical objects employed in physical theories do not always behave well. Einstein’s theory of space and time allows for spacetime singularities and Van Hove singularities arise in condensed matter physics, while intensity, phase and polarization singularities pervade wave physics. Within dissipative systems governed by matrices, singularities occur at the exceptional points in parameter space whereby some eigenvalues and eigenvectors coalesce simultaneously. However, the nature of exceptional points arising in quantum systems described within an open quantum systems approach has been much less studied. Here we consider a quantum oscillator driven parametrically and subject to loss. This squeezed system exhibits an exceptional point in the dynamical equations describing its first and second moments, which acts as a borderland between two phases with distinctive physical consequences. In particular, we discuss how the populations, correlations, squeezed quadratures and optical spectra crucially depend on being above or below the exceptional point. We also remark upon the presence of a dissipative phase transition at a critical point, which is associated with the closing of the Liouvillian gap. Our results invite the experimental probing of quantum resonators under two-photon driving, and perhaps a reappraisal of exceptional and critical points within dissipative quantum systems more generally.