TOGO was born in Nihonmatsu Umaba in Kajiyacho near Kagoshima Castle in Satsuma Province (Shimokajiyacho Katagiri), as the forth son of Satsuma feudal retainer Sanetomo TOGO and Mashuko (益子), the third daughter of Yomizaemon HORI (堀与三左衛門).
薩摩国鹿児島城下の加冶屋町二本松馬場（下加治屋町方限）に、薩摩藩士・東郷実友と堀与三左衛門の三女・益子の四男として生まれる。