Thereafter, Kazuyoshi became a member of the Riryo (government official) faction, along with Nagataka FUKUHARA, Naomori KUMAGAI and others, and as such fell into conflict with Kiyomasa KATO, Masanori FUKUSHIMA and the others of the Budan (militarist) faction.
その後、福原長堯や熊谷直盛らと共に吏僚派として、加藤清正、福島正則ら武断派と対立することになる。