Brown was born in Lexington, Kentucky, the daughter of businessman and former Governor of Kentucky John Y. Brown Jr. and former Miss America and businesswoman Phyllis George (1949-2020). George took maternity leave from her duties on CBS' NFL pregame show, "The NFL Today", to give birth to Pamela. Brown is the granddaughter of politician John Y. Brown Sr. and the half-sister of former Kentucky Secretary of State John Y. Brown, III. Her parents divorced in 1996 after 17 years of marriage, when Brown was 13.