One of them, Mizuho Financial, has already admitted senior managers knew three years ago it had lent 200m yen ($2m; Â£1.28m) to Japanese criminal gangs, but did not take action.
An external inquiry cleared the bank of intentionally covering up the loans.
Sumitomo Mitsui is the third involved.
The yakuza gangs are not actually illegal. But like the Italian mafia or the Chinese triads, they are involved in unsavoury activities such as gambling, drugs and prostitution, as well as operating protection rackets.
On Monday, lawyers hired by Mizuho to look into the transactions said "many officials and board members were aware of, or were in a position to be aware of, the issue".
But the lawyers' report also said that Mizuho failed to recognise it as a problem, believing that the compliance division "was taking care of it".
The company said 54 former and current executives would be punished, including Mizuho bank chairman, Takashi Tsukamoto, who is resigning his position but will remain as head of the parent company.
Japan's Financial Services Agency (FSA) will inspect three major banks, including giant Mitsubishi, over possible transactions involving yakuza organised crime gangs.