Born on February 16, 1839 in the Kakegawa Domain, he is the fourth child of Hidetaka Koga, leader of the Kakegawa Clan in Totomi. In 1855, Gengo traveled to Edo and studied Rangaku with daimyo of the Sakura Domain, Guntaro Kimura. He returned to his province two years later in 1857, but in 1858, Gengo was aspired to study navigation and went back to Edo under the guidance of Kō Yatabori, who was the professor at the Gunkan Office in Tsukiji. Since 1855, the 2nd Dutch Navy Training Mission, led by Willem Huyssen van Kattendijke, came to the Nagasaki to set up a naval training center.