The airport spent £1bn building the five-mile line 20 years ago to connect the hub to the Great Western track.
The Office of Rail and Road (ORR) previously decided Heathrow could not charge trains for using the line, which it said would cost about £42m a year.
The airport applied for a judicial review claiming the ORR did not have the power to reach that decision.
The west London airport also claimed the ORR's ruling was irrational.
But, Mr Justice Ouseley said on Friday the challenge had failed on all grounds.
A Heathrow spokesman said: "We are disappointed with today's ruling and are considering our next steps.
"Both Heathrow and Network Rail agree that track access charges must be fair to encourage future private investment in the rail network."
It added it was "committed to increasing sustainable public transport to the airport".
An ORR spokeswoman said: "We welcome this judgment and we will now work with all the affected parties to enable Crossrail services to start running as scheduled into the airport."
Four Crossrail trains an hour will run between Paddington and Heathrow terminals 2 to 4, replacing Heathrow Connect trains from May 2018.
A spokesperson for the railway regulator said the airport had proposed each train should be charged historical build costs of £597 per use and an operational expenditure charge of £138 per use.
But Heathrow said the investment recovery charge would be £460.09 per train between 2019 and 2028.
Heathrow cannot charge Crossrail for using its track to travel to the airport, a High Court judge has ruled.