Choei TAKANO, a scholar of the Western studies (rangakusha) of the late Edo Period, was his granduncle, a politician Etsusaburo SIINA, his nephew, the politician Yusuke TSURUMI, a son-in-law who married with his daughter, and a sociologist Kazuko TSURUMI, a philosopher Shunsuke TSURUMI, a theatrical director Seki SANO are his grandchildren.
江戸時代後期の蘭学者・高野長英は後藤の大叔父に当たり、甥に政治家の椎名悦三郎、娘婿に政治家の鶴見祐輔、孫に社会学者の鶴見和子、哲学者の鶴見俊輔、演出家の佐野碩をもつ。