Brown was born in Brooklyn, New York, the middle child of three of Maude Margaret (Johnson) and Robert Bruce Brown. She was the granddaughter of politician Benjamin Gratz Brown. Her parents suffered from an unhappy marriage. She was initially raised in Brooklyn's Greenpoint neighborhood, and in 1923, attended Chateau Brilliantmont boarding school in Lausanne, Switzerland, while her parents were living in India and Canterbury, Connecticut. In 1925, she attended The Kew-Forest School. She began attending Dana Hall School in Wellesley, Massachusetts, in 1926, where she did well in athletics. After graduation in 1928, Brown went on to Hollins College in Roanoke, Virginia.