Brown is openly lesbian and has had relationships with tennis player Martina Navratilova, actress-writer Fannie Flagg, socialite Judy Nelson, and politician Elaine Noble. Brown enjoys American fox hunting and is master of her Fox Hunt Club. She has also played polo, and started the women-only Blue Ridge Polo Club.