Also, as the Takatori-jo Castle was a Yamajiro (a castle built on a mountain to take advantage of the mountain's topography) which was uncommon during the Edo period and inconvenient to access, the domain lords and vassals began to relocate in the end of Kanei era (1624 - 1643) and only Joban was placed in the castle.
また、高取城は江戸時代を通じてもめずらしい山城であったため、交通に不便で寛永末期から藩主・家臣団の移転が始まり、城には城番が置かれるだけであった。