With most votes counted, Five Star's Virginia Raggi took about 36%, ahead of Roberto Giachetti of the centre-left Democratic Party (PD) on 25%.
The two will now go into a run-off vote on 19 June.
PD candidates came top in Milan and Turin but also face run-off votes.
"The wind is changing, this is the moment," Ms Raggi, a 37-year-old lawyer, told her supporters in Rome.
Five Star, founded by comedian Beppe Grillo in 2009, campaigned against the corruption that has plagued Italian politics for years.
In Rome the incumbent mayor, Ignazio Marino of the PD, resigned in October because of an expenses scandal. So the city has been without a mayor since then.
A much bigger scandal, involving alleged Mafia influence in Rome city hall, also appears to have fuelled Five Star's rise.
Five Star is looking to establish itself as the main opposition party or even pose a challenge to the government in general elections due in 2018.
Prime Minister Renzi has staked his political future on an October referendum in which he wants Italians to back far-reaching constitutional reforms.
The plan is to end Italy's tradition of "revolving-door" governments - that is, to inject stability into Italian politics after years of party infighting and legislative logjams.
Rome's next mayor will find a city mired in debt of more than â‚¬13bn (Â£10bn; $15bn) - twice its annual budget. Romans are frustrated by potholes, piles of rubbish and serious deficiencies in public transport and housing, the BBC's Julian Miglierini reports.
The anti-establishment Five Star Movement has a strong lead in the Rome mayoral race - a blow to Italy's centre-left Prime Minister Matteo Renzi in municipal elections.