We recently devised continuous “sex-scores” that sum up multiple quantitative traits, weighted by their respective sex-difference effect sizes, as an approach to estimating polyphenotypic “maleness/femaleness” within each binary sex. To identify the genetic architecture underlying these sex-scores, we conducted sex-specific genome-wide association studies (GWASs) in the UK Biobank cohort (females: n = 161,906; males: n = 141,980). As a control, we also conducted GWASs of sex-specific “sum-scores”, simply aggregating the same traits, without weighting by sex differences. Among GWAS-identified genes, while sum-score genes were enriched for genes differentially expressed in the liver in both sexes, sex-score genes were enriched for genes differentially expressed in the cervix and across brain tissues, particularly for females. We then considered single nucleotide polymorphisms with significantly different effects (sdSNPs) between the sexes for sex-scores and sum-scores, mapping to male-dominant and female-dominant genes. Here, we identified brain-related enrichment for sex-scores, especially for male-dominant genes; these findings were present but weaker for sum-scores. Genetic correlation analyses of sex-biased diseases indicated that both sex-scores and sum-scores were associated with cardiometabolic, immune, and psychiatric disorders.