Castlehaven wrote his memoirs in 1681 in response to the hysteria of the Popish Plot. Like all Catholic peers he had been barred from the House of Lords (where he sat as Baron Audley) by the Test Act 1678, much to the regret of the Protestant peers who held him in high regard. He took his leave on 30 November 1678, with a speech expressing his duty to the Crown, and his concern for the peace and welfare of the Kingdom. His fellow peers, knowing him to be in financial difficulty, wrote to Charles II recommending Castlehaven to his bounty.