They declined due to the Uprising of Zenshu Uesugi during the generation of Norifuji's grandson, Zenshu UESUGI, and they were distanced from the main group in the government of Kanto, but many of Ujinori's children served the Shogunate and the bloodline continued to exist, and some of them even became Kanto shitsuji (stewards) to the Horigoe Kubo, and continuing to make their mark in the politics of Kanto even after Zenshu's time.
憲藤の孫の上杉禅秀の代に上杉禅秀の乱により衰退し、関東の政治の主流派からは遠ざかったが、氏憲の子の多くは幕府に仕え血統は存続し、一部は堀越公方の関東執事となるなど、禅秀以降も関東の政治に足跡を残した。