He married Barbara Mavis Southern at St Edmund's Church, Manchester, in 1951, when he was 17 years old. Their daughter Stephanie was born later that year, and they later had two sons, Roger (born 1957) and Malcolm (born 1961), and another daughter, Deborah (born 1958). They divorced in 1969, and in June that year, Viollet married Helen B. Greeph; they were married until his death nearly 30 years later. Their daughter Rachel (born 1972) became the British number one ranked tennis player when she reached the second round of Wimbledon in 1996. During her tennis career, she won one ITF singles tournament and one ITF doubles tournament.