Brown was a debate champion as a member of the Lowell Forensic Society at San Francisco's Lowell High School, where he held twelve offices of student government; he graduated from Lowell in 1923. Rather than pursue an undergraduate degree, he instead worked in his father's cigar store, which doubled as a gambling shop. He studied law at night, while working part-time for attorney Milton Schmitt, receiving an LL.B. degree from San Francisco Law School in spring 1927. After passing the California bar exam the following fall, he began full-time employment in Schmitt's office.