Miller, a seventh-generation Texan, was born in Austin, Texas, to father Stewart Ransom "Randy" Miller, an attorney, and mother Ann Morwood (née Wilson Pugh). Miller's parents divorced when he was 17 years old. Miller, the oldest of three children, has a younger brother and sister. In 1952, Miller's paternal grandfather, Giles E. Miller, a young millionaire scion of a successful textile family, owned the first NFL football team in the South, the Dallas Texans. The Texans folded after seven games, marking the last time an NFL franchise would go bankrupt.