Duffy's widow Martha was left to manage the inn with the help of her children, whose numbers were soon reduced by the February 1807 death of their firstborn son James in an accident at a wedding party at Culbertson's Mill in Duboistown. Martha Duffy was left with her four remaining children. Her second daughter, Catharine, had married and moved with her husband to Meadville in Crawford County. The oldest daughter, Mary Ann, took over management of the inn. Bernard operated the family owned sawmill on Pine Creek, Margaret managed the Duffy family home, and Peter, the son, managed the family farm. These four Duffy children never married and spent the rest of their lives living together in the family home in Piatt Township.