Jozan was born in Izumi-go in Mikawa Province (present day Izumicho, Anjo City, Aichi Prefecture) into a samurai family that had served the Tokugawa clan (Matsudaira clan) for successive generations.
三河国泉郷（現在の愛知県安城市和泉町）の代々徳川氏（松平氏）に仕える譜代武士の家に生まれた。