In 1576, however, Akinaga went into retirement because of his delicate health and transferred the family responsibilities to Nagazane, who became the head of the Irobe clan and served Kenshin UESUGI.
しかし、その顕長が病弱だったため、天正4年（1576年）に兄が隠居し、代わって家督を譲られて色部氏の当主となり、上杉謙信に仕えた。