The International Court of Justice (ICJ) in The Hague said it could not examine Georgia's complaint because negotiations had not taken place.
Georgia said Russia and the rebels had used ethnic violence against Georgians in Abkhazia and South Ossetia.
Russia sent forces into the two regions in a brief war in August 2008.
Thousands of ethnic Georgians fled the regions during the conflict and many remain internally displaced in Georgia.
In a 10-6 vote on Friday the ICJ judges upheld Russia's argument that the court did not have jurisdiction to examine the Georgian complaint, because the two sides had not tried to resolve the dispute through negotiations.
Georgia said Russia had violated the International Convention on the Elimination of All Forms of Racial Discrimination (CERD).
The judges ruled that Georgia and Russia "did not engage in negotiations with respect to the latter's compliance with its substantive obligations under CERD".
The Georgian government said that despite Friday's ruling it would continue to pursue the case.
"The court has simply ruled that, due to a procedural technicality... the proceedings will not immediately lead to further consideration of the merits of the case against Russia," a government statement said.
"However, the court has left open the possibility that the case can proceed once the formal conditions for the exercise of jurisdiction by the court, as required by the 1965 convention, have been met."
In 2008 Western countries condemned Russia for penetrating deep into Georgia, beyond the conflict zones.
Russia later pulled its forces back to the two breakaway regions of Abkhazia and South Ossetia.
In September 2009 an EU-sponsored report said both sides had violated international law. It found that Georgia had attacked the Russian-backed South Ossetian rebels, triggering the war, after months of provocation.
The United Nations' top court has dismissed a case filed by Georgia that accuses Russia and separatist rebels of ethnic cleansing.