Robert Henry Allerton was born on March 20, 1873, as the second child and only son to Samuel Waters Allerton (1828–1914) and Pamilla Thompson Allerton (1840–1880). Through an entirely paternal line, Robert Henry Allerton was descended from Isaac Allerton, an English Puritan who came to America on the Mayflower in 1620. Samuel Allerton was a self-made man who made his millions in land, livestock, banking, and other commercial enterprises. Pamilla Allerton died in 1880, five days before Robert's 7th birthday. Two years later, Samuel Allerton married Agnes Thompson, Pamilla's younger sister. Agnes Thompson Allerton (1858–1924) became mother, friend, and cultural mentor for her stepson/nephew. She kindled his interests in literature, music, gardening, and above all, visual arts.