 Duffy was born in Rochester, New York, one of nine children.He graduated from Georgetown University in 1901 and from Harvard Law School in 1904.He was admitted to the bar the same year and began practice in partnership with Republican James Breck Perkins, a member of Congress.