Although Mutsu's father was a senior vassal serving in the Kishu Domain who accomplished financial reconstruction, he was defeated in a political strife within the domain and was overthrown when Mutsu was eight years old (1852); as the result, the family suffered hardship and poverty.
父は紀州藩に仕え財政再建をなした重臣であったが、宗光が8歳のとき（1852年）藩内の政争にやぶれて失脚したため、一家には困苦と窮乏の生活がおとずれた。