The Premier League strugglers are searching for their third manager in under 12 months after sacking Bradley.
He lasted 11 games in 85 days and says fellow Americans Jason Levien and Steve Kaplan were "swayed by negativity".
"They haven't said a word. The chief executive [Chris Pearlman], he's not said a word," said James.
"The three Americans, you don't hear from them. Why?".
Wales manager Chris Coleman and Wales and Manchester United legend Ryan Giggs are among those to have been linked with succeeding Bradley.
Listen: The Bob Bradley interview
Listen again: Call Rob Phillips Swansea City special, with Leighton James
James says Bradley has been put in an unfair position.
"He is an honourable man left high and dry to answer every question that has been put to him," he told BBC Radio Wales Sport.
"I would ask a serious question to the two owners and the chief executive - where have you been in support of your manager?
"Because you've been deafening with their silence and that's what frightens me most of all.
"Never once in the difficult 11 games that Bob's had, never once has any of those people actually come out and supported Bob.
"I can't remember a comment by any of the three of them.
"He's their man, obviously, because nobody thought about him before he came in."
James, who scored 27 goals in 98 league appearances for the Swans from 1980-83, says long-standing chairman Huw Jenkins also bears responsibility for Swansea's situation.
"He's as culpable as the owners," he said.
Ex-Swansea City and Wales winger Leighton James says the silence from the club's owners after Bob Bradley's sacking, "frightens" him.