Stafford's principal character flaw seems to have been his quarrelsome nature. During the Popish Plot he pointed out the absurdity of linking him with Lord Arundell as a co-conspirator, since it was well known that they had not been on speaking terms for 25 years. Over the years he quarreled with many of his Howard relations, including Henry Howard, 6th Duke of Norfolk, the head of the family, which was to prove unfortunate for him in 1680 when several of his Howard cousins sat as his judges to try him for treason. According to John Evelyn, an eye-witness, of his close relatives in the House of Lords who sat in judgement, only the Earl of Arundel voted Not Guilty, showing, as Evelyn rightly remarked, that Stafford was a man "not beloved by his family".