Manjiro was born in the Naka-no-hama in Tosa Province (now Tosashimizu in Kōchi Prefecture). He was the second son of a poor fisherman. He worked when he was young because his father died and his mother and older brother were sick. Because of this, he did not learn how to read and write very well.