Phosphogypsum is an industrial by-product from the wet preparation of phosphoric acid. Phosphorus building gypsum (PBG) can be obtained from phosphogypsum after high-thermal dehydration. Improving the mechanical properties of PBG is of great significance to extending its application range. In this paper, PBG was modified by adding nano-CaCO3. Specifically, this study, conducted on 0.25–2% nano-CaCO3-doped PBG, tested effects on the fluidity, setting time, absolute dry flexural strength, absolute dry compressive strength, water absorption and softening coefficient of PBG, followed by its microscopic analysis with SEM and XRD. The experimental results showed that, with an increase in nano-CaCO3content, the fluidity and setting time of PBG-based mixes were decreased. When the content was 2%, the fluidity was 120 mm, which was 33% lower than that of the blank group; the initial setting time was 485 s, which was 38% lower than that in the blank group; the final setting time was 1321 s, which was reduced by 29%. Nano-CaCO3evidently improved the absolute dry flexural strength, absolute dry compressive strength, water absorption and softening coefficient of PBG to a certain extent. When the content was 1%, the strengthening effect reached the optimum, with the absolute dry flexural strength and absolute dry compressive strength being increased to 8.1 MPa and 20.5 MPa, respectively, which were 50% and 24% higher than those of the blank group; when the content was 1.5%, the water absorption was 0.22, which was 33% lower than that of the blank group; when the content approached 0.75%, the softening coefficient reached the peak of 0.63, which was 66% higher than that of the blank group. Doping with nano-CaCO3could significantly improve the performance of PBG, which provides a new scheme for its modification.