Duffy and his family landed in Philadelphia in August 1775. After a short stay in the Philadelphia area, he and his family moved further inland to Lancaster County. The family spent seven years in the Lancaster area before moving to Coxestown, now known as Susquehanna Township, just north of Harrisburg in Dauphin County. The Duffys owned a home along the banks of the Susquehanna River for about a year before it was flooded and accidentally burned in the winter of 1784. Duffy did not lot this stop him. He gathered his family and migrated further still into the Pennsylvania frontier. The Duffy family was joined by the Stewart family on their trip from Dauphin County to Lycoming County. Both families settled down to the east of what is now Jersey Shore. The Stewarts settled on the southern side of the West Branch Susquehanna River in Nippenose Township and the Duffys settled on the opposite shore in Piatt Township.