In 1938 he married fellow artist Hildegard Huber Sasse, and in 1942 the couple had a daughter, Ursula.
Born in 1902 in the Wuerttembergian village of Laudenbach (now part of Weikersheim), Huber first studied law, but gave that up to become an artist in Munich.