Staff and visitors were evacuated from the Metropolitan Remand Centre in Ravenhall after clashes involving hundreds of inmates began at midday.
Parts of the prison are in lockdown but the outer perimeter was not breached.
Victorian Corrections Commissioner Jan Shuard said it was one of Melbourne's biggest ever prison riots.
She said it might be linked to the smoking ban, due to come into effect on Wednesday, but the ban would go ahead regardless.
"Until I get a full debrief, I can't speculate about how it all came about," she told reporters.
But she said it was "very disappointing that it occurred today".
Other media reports quoted people in contact with prisoners who blamed the ban for the violence.
One man, David, told Melbourne's 3AW radio station an inmate had told him earlier in the week that "all hell was going to break loose" if their tobacco was removed.
Police and emergency services rushed to the Metropolitan Remand Centre in the afternoon afternoon after prisoners breached an inner perimeter.
There have been no reports so far of injuries but local media have reported police entering the prison and using a water cannon to subdue inmates.
Footage from the scene broadcast on Australian media showed prisoners in an outside yard with covered faces lighting fires and smashing windows.
Ms Shuard said several small fires started by the prisoners were quickly "contained".
Police secured the prison's perimeter while a police drone and police helicopter surveyed the scene from the air.
The riot, which broke out at about 12.30pm local time on Tuesday (02:30 GMT), was not a threat to public safety, said prison authorities.
The commissioner could not confirm reports the riot involved two rival prison gangs.
Earlier in the day, she said the state's prison system had been "very ready" for the smoking ban.
About 84% of Victorian prisoners are smokers, according to government figures, about five times the rate in the general community.
The government said it was introducing the ban to improve the health of prison staff and inmates.
Ms Shuard said the ban had been 18 months in the making.
"We've had a very long-term project in place to work with both our staff and the prisoners in preparing for [the ban]," Ms Shuard told the ABC on Tuesday.
Heavily armed police have been sent to a prison on the outskirts of Melbourne following a major riot reportedly linked to a smoking ban.