Brown was one of four Black Americans in the Assembly in 1965. The other three were Mervyn M. Dymally, F. Douglas Ferrell, Byron Rumford. He continued to be reelected to the Assembly until 1995. In the 1960s, Brown served as chair of the Legislative Representation Committee, a powerful position that helped him climb the Assembly ranks. He became the Democrats' Assembly whip in 1969. Brown also served on the Assembly Ways and Means Committee. In 1972, he delivered a speech at the Democratic National Convention. He lost his bid for the speakership in 1972. In 1975, Brown authored and lobbied the successful passing of the Consenting Adult Sex Bill that legalized homosexuality in California, thus earning the strong and lasting support of San Francisco's gay community. Similarly, he voted against AB 607, which banned same-sex marriage in 1977, further building his reputation as a supporter of the civil rights of gays and lesbians. During the 1970s, Brown continued to expand his legal practice, including the representation of several major real estate developers. He won the Speakership in 1980 with 28 Republican and 23 Democratic votes.