Through his mother, Matilda Auchincloss (1824–1894), Brown was a cousin of businessman Hugh D. Auchincloss Sr. and first cousin once removed of stockbroker and Standard Oil heir Hugh D. Auchincloss Jr., stepfather of Jacqueline Kennedy Onassis and Lee Radziwill. Through his father, Horatio Silas Brown, he was descended from John Howland, a signatory of the Mayflower Compact, and Elizabeth Tilley, a fellow Mayflower passenger.
Through his mother, Matilda Auchincloss (1824–1894), Brown was related to Hugh D. Auchincloss Sr., a businessman, and Hugh D. Auchincloss Jr., a stockbroker and Standard Oil heir. He was also related to Jacqueline Kennedy Onassis and Lee Radziwill through his stepfather. Through his father, Horatio Silas Brown, he traced his lineage back to John Howland, a signatory of the Mayflower Compact, and Elizabeth Tilley, another passenger on the Mayflower.