He had argued that he was being denied his rights to a family life, because his wife and children had not been granted UK visas.
The judges rejected this argument, saying they had not properly applied.
A UN-backed court convicted him of war crimes over his support for rebels who committed atrocities in Sierra Leone.
As well as complaining about not being able to see his family, Taylor argued he was being held "effectively in isolation" because he was "too much of a target and too vulnerable" to be kept with other inmates of the Frankland prison in the northern English city of Durham.
He further said he wanted to be on the African continent, where people would share a "cultural affinity" with him.
However, the court found that the rejection of his wife's visa was "due purely to his wife's failure to comply with the United Kingdom visa requirements and her ignoring the assistance offered to her [by the registrar's office] to re-apply".
It further stressed that prisoners did not have the right to choose their place of incarceration.
The Special Court for Sierra Leone trial was held at The Hague on the agreement that he was jailed elsewhere.
The overseas venue for the court case was chosen in case the trial sparked renewed unrest in West Africa.
An act of parliament was passed to allow for Taylor to serve his sentence in the UK, at the cost of the British government, following his conviction.
Taylor was sentenced in 2012 and arrived in the UK last October, having unsuccessfully challenged the decision to be detained there.
Charles Taylor: Preacher, warlord, president
Taylor was convicted on 11 charges including terrorism, rape, murder and the use of child soldiers by rebel groups in neighbouring Sierra Leone during the 1991-2002 civil war, in which some 50,000 people died.
The former Liberian leader was found to have supplied weapons to the Revolutionary United Front (RUF) rebels in exchange for so-called blood diamonds.
The rebels were notorious for hacking off the limbs of civilians to terrorise the population.
Taylor has always insisted he is innocent and that his only contact with the rebels was to urge them to stop fighting.
Ex-Liberian President Charles Taylor has been ordered to serve the rest of his jail term in the UK, after losing a request to be transferred to Rwanda.