Penn spent most of his time in the soon-to-be capital city of Philadelphia governing his settlement, leaving the manor house empty for long periods of time. From 1699 to 1701, he rented "The Slate Roof House" in Philadelphia as his second residence and city townhouse. He left the colony for England in 1701 to fend off a threatening French claim to his British grant, dying destitute in his home town or Ruscombe after having been defrauded by his English agent of rents and income due him. By 1736, one of Penn's sons remarked that the Pennsbury house "was very near falling, the roof open as well as the windows and the woodwork almost rotten." It remained in family hands until 1792.