Later the building became a possession of the Savoia-Acaja, a secondary branch of the House of Savoy; in the early 14th century, they enlarged it into a castle. A century later Ludovico of Acaja rebuilt it in square shape, with an inner court and a portico, and four cylindrical towers at each corner. The form of this edifice is still clearly recognisable from the back section of the palace. After the extinction of the Acajas, the edifice became a residence for guests of the house of Savoy.