His father Yoshikiyo, who was the second son of Yoshimitsu, was based in Takeda district in Naka County, Hitachi Province (Takeda, former Katsuta City, now Hitachinaka City, Ibaraki Prefecture) and bore the title Takeda kaja.
義光の次男であった父の義清は常陸国那珂郡武田郷（茨城県ひたちなか市、旧勝田市武田）を本拠とし武田冠者を称していた。