Brown was born on March 20, 1934, in Mineola, a small segregated town in East Texas marked by racial tensions, to Minnie Collins Boyd and Lewis Brown. He was the fourth of five children. During Brown's childhood, mob violence periodically erupted in Mineola, keeping African-Americans from voting. His first job was a shoeshine boy in a whites-only barber shop. He later worked as a janitor, fry cook and field hand. He learned his strong work ethic at a young age from his grandmother. He graduated from Mineola Colored High School, which he later described as substandard, and left for San Francisco in August 1951 at the age of 17 to live with his uncle.