In 1538, James V summoned the 3rd Earl before the Privy Council for non-payment of his feudal dues, and in 1540 the Earl was banished to Inverness. Morton reached Brechin, in Angus, where he signed a deed resigning his lands to his kinsman Robert Douglas of Lochleven, who was then compelled to resign the lands in turn to James V. After James V's death in late 1542, George Douglas of Pittendreich and the Earl of Arran assisted Morton in reclaiming his lands, including Aberdour. In return their sons were to marry two of Morton's three daughters. Pittendreich's son James (1525–1581) married the heiress, Elizabeth, and succeeded to the earldom in 1553.