The Yamakami Domain was successively headed by the Ando family, lord Shigenaga ANDO and lord Shigehiro ANDO, although the domain was abolished and became a tenryo (shogunal demesne) when Shigehiro was promoted to roju (member of shogun's council of elders) and transferred to the Bitchu Matsuyama Domain on June 12, 1695.
安藤氏はその後、安藤重長、安藤重博らが相次いで藩主となったが、重博が老中に栄進したため、元禄8年（1695年）5月1日に備中松山藩に移封され、ここに山上藩は廃藩、天領となった。