Kaji was born in Minamiawaji on January 13, 1980. After graduating from Takigawa Daini High School, he joined J1 League club Cerezo Osaka in 1998. In 2000, he moved to J2 League club Oita Trinita. He played as regular right side back in 2 seasons. In 2002, he moved to J1 club FC Tokyo. The club won the champions in 2004 J.League Cup which is first major title in the club history. In 2006, he moved to Gamba Osaka. The club won the champions in the 2007 J.League Cup, 2008 and 2009 Emperor's Cup. In Asia, the club won the champions 2008 AFC Champions League and the 3rd place 2008 Club World Cup. He played 225 matches for the club in 9 seasons. In summer 2014. he moved to Major League Soccer club Chivas USA. However, the club was disbanded in 2014, he returned to Japan and joined J2 League club Fagiano Okayama in 2015. He retired end of 2017 season.