Fisher married Margaret Trusler on October 2, 1928. She gave birth to Thornton Fisher in February 1937. She was frequently apart from Vardis, and while Vardis disliked religion, Margaret was a Christian. In July 1936, Vardis met Opal Laurel Holmes, a competent researcher in his WPA project. They were mutually attracted to one another. Vardis divorced from Margaret in 1939. Fisher married Opal Laurel Holmes on April 16, 1940, and bought land near Hagerman, Idaho, where they built their own house.