Duffy was born in Townsend, Montana. He was raised in Everett, Washington and in Eagle Point, Oregon. He studied at the University of Washington. He was married to Carlyn Rosser from 1974 until her death in 2017. They had two children.