Sex differences in serum phosphate and calcium have been reported but the exact nature and underlying regulatory mechanisms remain unclear. We aimed to compare calcium and phosphate concentrations between sexes, and explore potential covariates to elucidate underlying mechanisms of sex differences in a prospective, population-based cohort study. Pooled data of subjects > 45 years from three independent cohorts of the Rotterdam Study (RS) were used: RS-I-3 (n = 3623), RS-II-1 (n = 2394), RS-III-1 (n = 3241), with separate analyses from an additional time point of the first cohort RS-I-1 (n = 2688). Compared to men, women had significantly higher total serum calcium and phosphate concentrations which was not explained by BMI, kidney function nor smoking. Adjustment for serum estradiol diminished sex differences in serum calcium while adjustment for serum testosterone diminished sex differences in serum phosphate. Adjustment for vitamin D and alkaline phosphatase did not change the association between sex and calcium or phosphate in RS-I-1. In the sex-combined group, both serum calcium and phosphate decreased with age with a significant interaction for sex differences for serum calcium but not phosphate. In sex-stratified analyses, serum estradiol but not testosterone was inversely associated with serum calcium in both sexes. Serum estradiol was inversely associated with serum phosphate in both sexes to a similar degree, while serum testosterone was inversely associated with serum phosphate in both sexes with an apparent stronger effect in men than in women. Premenopausal women had lower serum phosphate compared to postmenopausal women. Serum testosterone was inversely associated with serum phosphate in postmenopausal women only. In conclusion, women > 45 years have higher serum calcium and phosphate concentrations compared to men of similar age, not explained by vitamin D or alkaline phosphatase concentrations. Serum estradiol but not testosterone was inversely associated with serum calcium while serum testosterone was inversely associated with serum phosphate in both sexes. Serum testosterone may in part explain sex differences in serum phosphate while estradiol could partly explain sex differences in serum calcium.