Tomomitsu was from the family of a lower-ranking government official serving Kurododokoro (the Chamberlain's Office) for generations, and called himself as the Iga clan since his appointment to Iga no kami (Governor of Iga Province).
蔵人所に代々使えた下級官人の出身で、朝光が伊賀守に任じられて以降、伊賀氏を称した。