Brown was born in San Francisco, California. His father was Pat Brown who also served as Governor of California until losing re-election to Ronald Reagan. His mother was Bernice Layne Brown. He was raised in San Francisco. Brown studied at Yale University and at the University of California, Berkeley.