Brown originally wanted to attend Stanford University. His interviewer from Stanford was a faculty member at San Francisco State College and was surprised by Brown's ambition. Although Brown did not meet the qualifications for Stanford or San Francisco State, the professor facilitated Brown's admission to the latter school on probation. Brown adjusted to college studies after working especially hard to catch up in his first semester. He joined the Young Democrats and became friends with John L. Burton. Brown originally wanted to be a math instructor but campus politics changed his ambitions. He became active in his church and the San Francisco NAACP. Brown worked as a doorman, janitor and shoe salesman to pay for college. He is a member of Alpha Phi Alpha fraternity. He also joined the Reserve Officers' Training Corps (ROTC). Brown earned a bachelor's degree in political science from San Francisco State in 1955. He later said that his decision to attend law school was primarily to avoid being drafted. He quit the ROTC and joined the California Army National Guard's 126th Medical Battalion, where he was trained as a dental hygienist. Brown attended University of California, Hastings College of the Law, where he also worked as a janitor. He befriended future San Francisco Mayor George Moscone, for whom Brown later managed a campaign. Brown earned a J.D. in 1958 and was class president.