Miller was born in Van Buren, Arkansas, but his parents migrated to Franklin, Texas when he was one year old, and he grew up there. His father Jacob Miller, born in Pennsylvania in 1801, was a stonemason, and helped build the first capitol building in Austin. Miller's mother was born Cynthia Basham. Just a few years after the move, Miller's father died, so his mother took the family to Evant, Texas to live with her parents. In 1869, when Miller was eight, his grandparents were found murdered in their home. Miller was arrested but never prosecuted for the crime.