The leaders reached the deal at a summit hosted by Nigeria's new president Muhammadu Buhari, who made tackling it a key campaign pledge.
Most of the troops will be from Nigeria with Chad, Cameroon, Niger and Benin also contributing.
Thousands of people have been killed and hundreds kidnapped by the group.
The decision that the force should be led by a Nigerian commander came after Mr Buhari rejected calls for leadership to rotate between the countries.
But the leaders agreed the force would be based in the Chadian capital, N'djamena, rather than Baga in northeast Nigeria.
Calling this meeting at short notice is a sign that ending the Boko Haram insurgency is the number one priority for Nigeria's new president, Muhammadu Buhari.
Since being sworn in less than two weeks ago he has flown to Chad, Niger and Germany to attend the G7 summit.
All of these meetings focused on building an international alliance to fight Boko Haram.
During the last two weeks more than 100 people have been killed in gun attacks and bomb blasts carried out by the jihadists.
They set a July deadline for the parties to contribute troops.
At the conference, Mr Buhari stressed the need for better sharing of intelligence to stop the flow of money and weapons across borders.
He said a campaign "on all fronts" was needed to bring the six-year insurgency to an end.
Chad, Cameroon and Niger have been helping Nigeria recapture territory from the insurgents, but a plan to create a large joint military force had stalled.
Five African countries have agreed to set up a joint military force led by Nigeria to fight the Islamist militant group Boko Haram.