The US and the UK were among the countries which sponsored a resolution which for the first time explicitly calls for an international probe.
Sri Lanka's army defeated separatist Tamil Tiger rebels after 26 years of bloody civil war in May 2009.
Both sides have been accused of committing war crimes.
But it is events in the final phase of the war that has come under scrutiny, with one UN report saying that as many as 40,000 Tamil civilians may have been killed, mostly by government shelling in those final months.
Sri Lanka has consistently denied such allegations and says it is being unfairly targeted. It also claims that Tamil rebels are attempting to regroup in the north of the country.
But the resolution calls for a "comprehensive investigation into alleged serious violations and abuses of human rights and related crimes by both parties in Sri Lanka".
"We reject this," Sri Lanka's President Mahinda Rajapaksa told the AFP news agency. "This resolution only hurts our reconciliation efforts. It does not help."
The BBC's Charles Haviland in Colombo says this is the first concrete step towards any kind of international inquiry on the conduct of the island's government and the defeated Tamil Tiger rebels.
The resolution was carried in a 23-12 vote.
That means that 23 member states of the 47-member Human Rights Council voted for this resolution, 12 voted against it while 12 abstained - including India which many had expected to vote against its neighbour.
The document also expresses "serious concern" at events said to be still going on in Sri Lanka - including the intimidation of civil society, disappearances and torture.
Earlier this month Sri Lankan detained two prominent human rights activists for 48 hours under anti-terrorism laws. While rights groups point to continuing abuses, the government just as vehemently denies such allegations.
Resolutions expressing alarm at the human rights situation in Sri Lanka have been passed before by the council, but the distinctiveness of this draft is that it asks the UN's Office of the High Commissioner for Human Rights to carry out an inquiry.
The UN Human Rights Council has voted for a resolution which paves the way for an inquiry into rights abuses at the close of Sri Lanka's civil war.