The castle fell into disuse after the fall of the Second Mexican Empire in 1867. In 1876, a decree established it as an Astronomical, Meteorological and Magnetic Observatory on the site, which was opened in 1878 during the presidency of Sebastián Lerdo de Tejada. However, the observatory was only functional for five years until they decided to move it to the former residence of the Archbishop in Tacubaya. The reason was to allow the return of the Colegio Militar to the premises as well as transforming the building into the presidential residence.