Brown was California's first Black American Speaker of the Assembly, and served in the office from 1981 to 1995. In 1990, he helped negotiate an end to a 64-day budget standoff. In 1994, Brown gained the vote of a few Republicans to maintain the Speakership when the Democrats lost control of the Assembly to the Republicans led by Jim Brulte. Brown regained control in 1995 by making a deal with Republican defectors Doris Allen and Brian Setencich, both of whom were elected Speaker by the Democratic minority. During their tenures, Brown was the "de facto" Speaker.