His last years were troubled by litigation and disputes regarding his East Anglian estates, in which he was helped by both John Paston, to whom he was related through Paston's wife, Margaret, and Sir William Yelverton, and by factional fighting at court which ultimately led to the so-called Wars of the Roses. Paston and Yelverton would go on to be two of the main protagonists in the battles over his property after his death. Fastolf was inclined to sympathise with Richard, Duke of York, whom he had known and served in France, but it would be an exaggeration to say that he ever became a 'Yorkist'. He was a widower throughout the last decade of his life, when he lived at Southwark and Caister, and he had no heir. He seems to have been a somewhat lonely figure, and made several attempts to draft a will, establishing a Chantry College at Caister Castle.