William and his family left England in August 1641, moving to Antwerp; his parents had also left England and were living in the same area. He was allowed by Parliament to return to England with his wife for a time in 1646 and 1647; but in 1649 his estates were sequestered and he was forced to compound for recusancy and royalism. At his trial in 1680, he claimed to have performed many duties for King Charles II during the 1650s, travelling between England and the Low Countries, and visiting Rome, the Palatinate and Heidelberg; in this last he was arrested for claims of debt against the Arundel estate. Stafford was imprisoned in 1656 in the Netherlands, this time for his father's debts. There were many family quarrels over the Howard inheritance, especially between William and his elder brother's family, who pursued a series of lawsuits against William and his mother for money allegedly due to them.