While at Hollins she was briefly engaged. She dated, for some time, an unknown "good, quiet man from Virginia", had a long-running affair with William Gaston, and had a summer romance with Preston Schoyer. In the summer of 1940, Brown began a long-term relationship with Blanche Oelrichs (nom de plume Michael Strange), poet/playwright, actress, and the former wife of John Barrymore. The relationship, which began as a mentoring one, eventually became romantic and included co-habiting at 10 Gracie Square in Manhattan beginning in 1943. As a studio, they used Cobble Court, a wooden house later moved to Charles Street. Oelrichs, who was 20 years Brown's senior, died in 1950.