Varney was born in Canning Town, Essex, England. He was married to Lilian from 1939 until her death in 2002. They had one daughter, Jeanne. He had a heart attack in 1965 and 1981, and a stroke in 1989.