It was initiated between 1420 and 1422 by Afonso, Count of Barcelos, the illegitimate son of John I of Portugal (and future Duke of Bragança), after his marriage to his second wife. His prodigeny would occupy the space until the Dukes of Braganza moved to Vila Viçosa, abandoning the palace. The 16th Century marked the beginning of period of ruin, which was aggravated during the 19th century, when the local population used the palace as a personal quarry. During the Estado Novo regime, a controversial restoration restored the Palace, while implying a grandeur that may not have existed. The "Palace of the Dukes" was classified as a National Monument () in 1910, and has been an official residence for the Presidency.