The residence was ordered to be built in the first quarter of the 15th century (likely 1420 to 1422) by Afonso, Count of Barcelos, the illegitimate son of John, and future Duke of Braganza, following his return to Portugal after a series of diplomatic missions to the Courts of France, Venice, Aragon and Castile. The construction of this building marked the concretization of his second marriage, when he took residence in the town. The project was given, as some authors have suggested, to French architect Antom. Yet, by 1442, it was still under construction when the Regent Peter came to Guimarães and stayed at the Palace, at the time conferring on his half-brother the title of 1st Duke of Braganza. The commission continued into 1461, when Afonso died: the property and titles passed on to his brother, Fernando, although the widow, Constança de Noronha, continued to live at the residence and received the land rents from the holdings in Guimarães.