Rousey, 28, had won her previous 12 bouts and is considered the most feared female prize fighter in the world.
But fellow American Holm delivered a knockout head kick 59 seconds into the second round, plus several blows to the head, and the referee stopped the bout.
UFC pundit Joe Rogan called it the biggest upset in the sport's history.
It leaves multiple boxing world champion Holm, 34, undefeated in the mixed martial arts sport, with a 10-0 record.
The event drew a record crowd of more than 56,000 at the Etihad Stadium.
Holm said: "I had so much love and support. I thought, 'how could I not do this?'.
"There was lots of blood, sweat and tears, but it sure was worth it. I'm trying to take this in, but this is crazy."
Holly Holm stunned reigning champion Ronda Rousey to win the Ultimate Fighting Championship bantamweight title in Melbourne.