Long-time Mugabe critic Morgan Tsvangirai and former Vice-President Joice Mujuru say they will work together in next year's election.
However, it is not yet clear which of them will be the presidential candidate.
Mr Mugabe, 93, has been in power since independence from Britain in 1980 and has said he will seek re-election.
"This is just the beginning of the building blocks towards establishing a broad alliance to confront Zanu-PF between now and the next election," Mr Tsvangirai told journalists.
The alliance is an important first step towards uniting a deeply divided opposition.
And if this alliance succeeds it will be the first time President Mugabe has faced a united opposition on this scale since coming into power in 1980.
At least a dozen parties are expected to be part of the coalition.
Former Vice-President Joice Mujuru who was fired from the ruling Zanu-PF party in 2014 was the first to sign a pact with Morgan Tsvangirai.  She says it follows six months of consultations.
A Movement for Democratic Change splinter group has now also come on board.
Divisions among the opposition have been blamed for previous electoral losses.
For the first time, Mr Tsvangirai apologised for this and accepted responsibility for the mistakes made in the past. His party has split four ways since it was formed in 1999.
Ms Mujuru's National People's Party recently splintered after less than a year.
But the real test for the opposition lies ahead. The parties still need to hammer out the terms of this alliance. In particular who will lead the coalition.
President Mugabe is a formidable opponent. He has been accused of stealing elections and using violence to stay in power.
Mr Mugabe has previously said he would not be losing any sleep over the proposed coalition.
Mr Tsvangirai has run against Mr Mugabe several times since he helped found the Movement for Democratic Change.
Each time he has said he was denied victory because of violence and rigging - charges denied by Mr Mugabe and his allies.
He became prime minister in a tension-filled coalition government with Mr Mugabe from 2009 until 2013.
Ms Mujuru was vice-president to Mr Mugabe for 10 years until she was fired in 2014.
Two of Zimbabwe's best known opposition figures have agreed to form an alliance against President Robert Mugabe.