Media playback is not supported on this device
Britain's quartet of Nigel Levine, Delano Williams, Matthew Hudson-Smith and Rooney won their semi-final in two minutes 58.88 seconds.
However, Hudson-Smith, who ran the third leg, was ruled to have had part of his foot out of the takeover zone when he began running.
Team GB's appeal was rejected because video evidence was inconclusive, meaning the decision of the referee must be upheld.
"It's a track judge's opinion," said Rooney. "Unfortunately, there wasn't a Hawk-Eye on the line.
"In other sports, like tennis and football, they have Hawk-Eeye. We've basically got a track judge who has decided that that's what he has seen and there's no conclusive evidence to prove it either way.
"I've just seen it now and I'm even angrier. Matt has been very badly treated."
After the race, British Athletics said it was "hugely disappointed" and confident the team would have won a medal in the final.
Martyn Rooney tells BBC Sport he is even angrier after seeing a replay of Great Britain's controversial disqualification from the 4x400m relay for the first time on BBC television.