Vidor was born into a well-to-do family in Galveston, Texas, the son of Kate (née Wallis) and Charles Shelton Vidor, a lumber importer and mill owner. His grandfather, Károly Charles Vidor, was a refugee of the Hungarian Revolution of 1848, who settled in Galveston in the early 1850s. Vidor's mother, Kate Wallis, of Scotch-English descent, was a relative of the second wife of iconic frontiersman and politician Davy Crockett. The “King” in King Vidor is no sobriquet, but his given name in honor of his mother's favorite brother, King Wallis.