In Scotland Queen Margaret sought to regain the regency, but in vain. Young king James was kept a virtual prisoner by Albany, and queen Margaret was allowed to see her son only once between 1516 and end of Albany's regency. Margaret started to try get a divorce from Angus, also through Albany secretly. When Albany returned in November 1521 Margaret now sided with him against her husband. He came to Edinburgh Castle, where James V was kept, and in a public ceremony the keeper gave him the keys, which he passed to Margaret, who gave them back to Albany, symbolising that government of Scotland was in his hands. Thus Albany was able to keep an upper hand in regard to the ambitious Angus. The regent took the government into his own hands. Albany put Angus under charges of high treason in December 1521, and later sent him practically a prisoner to France. Angus's representative, Gavin Douglas, complained about Albany's government to Henry VIII.