Infertility is often associated with diverse psychiatric morbidities and quality of life impairments. Hence, this meta-analysis aimed to compare stress, depression, anxiety and quality of life (QoL) among infertile men and women. We retrieved the relevant articles from multiple databases. For the statistical analyses, we used Comprehensive Meta-Analysis software v. 3.7 (Biostat Inc., Englewood, NJ). Standardized mean differences (SMD) with 95% confidence intervals (CI) were calculated and displayed in forest plots. Among the 4123 articles identified, 35 studies met the inclusion criteria. Our results revealed that stress, depression, and anxiety were higher in infertile women compared to men. Similarly, infertile women presented a lower QoL than infertile men. Subgroup analysis revealed that the assessment tool used, study design, and geographical origin were a source of heterogeneity. This meta-analysis showed that psychological disturbances were higher in infertile women compared to men. Physicians need to consider this difference to enable couples to better understand and support each other.