Hubbard was born in Bloomington, Illinois, to Silas Hubbard and Juliana Frances Read on June 19, 1856. In the autumn of 1855, his parents had relocated to Bloomington from Buffalo, New York, where his father had a medical practice. Finding it difficult to settle in Bloomington—mainly due to the presence of several already established doctors—Silas moved his family to Hudson, Illinois the next year. Nicknamed "Bertie" by his family, Elbert had two older siblings: Charlie, who was largely bed-ridden after a fall when he was young, and Hannah Frances, nicknamed "Frank" like her mother. Charlie died at the age of nine, when Elbert was three-and-a-half years old. Elbert also had three younger sisters who were named Mary, Anna Miranda, and Honor.