Yoshizawa excelled in music from an early age, adding a koto part to the jiuta 'Yashima' at the age of eleven; in 1852, he was given responsibility for visually impaired people in Owari (present-day Aichi Prefecture) by the domain, with a stipend for five persons, and became a celebrated musician in Nagoya, performing at various occasions for the Owari Tokugawa family, including playing Heikyoku at memorial services for their ancestors, playing the kokyu at the Dolls' Festival, and composing a song for the marriage ceremony of the head of the family that owned Matsuzakaya (present-day Matsuzakaya Department Stores).
11歳で地歌「屋島」に箏の手を付けるなど幼少より楽才秀で、1852年（嘉永5年）には藩の命で尾張の盲人支配頭となり、五人扶持を賜り、尾張徳川家の諸行事での演奏、例えば先祖供養の際の平曲演奏や、雛の節句での胡弓の演奏などを勤めたのをはじめ、松坂屋（現松坂屋百貨店）の当主の婚礼祝いの曲を作ったりと、名古屋の名士音楽家として活躍した。