As of 960, the was also repeatedly destroyed by fires, but it was systematically rebuilt and used as the official imperial residence until the late 12th century. During periods of rebuilding the following fires, the emperors frequently had to stay at their within the city. Often these secondary palaces were provided by the powerful family, which especially in the latter part of the Heian period exercised "de facto" control of politics by providing consorts to successive emperors. Thus the residences of the emperors' maternal grandparents started to usurp the residential role of the palace even before the end of the Heian period. The institution of rule by retired emperors or the from 1086 further added to the declining importance of the palace as retired emperors exercised power from their own residential palaces inside and outside the city.