Stephen Lansdown, who co-founded financial services firm Hargreaves Lansdown and owns Bristol City FC, relocated to Guernsey in 2010.
He told the BBC it "made sense" financially to move to an area with more favourable tax rates.
Mr Lansdown is number 974 on the Forbes list of the richest people in the world with an estimated fortune of Â£1bn.
He and fellow billionaire Peter Hargreaves started Hargreaves Lansdown in 1981 out of a house in Bristol.
Both Hargreaves Lansdown and Imperial Tobacco have their headquarters in Bristol and are in the FTSE 100.
The former Thornbury grammar school pupil said: "I paid all my taxes while I was in the UK and I paid an enormous amount of tax while I was there," he said.
"I don't live there any more, I don't have the benefits of the UK any more. I'm paying my tax here and I feel that's justified."
The term tax haven can apply to places with lower tax levels and jurisdictions where taxes can be avoided altogether.
Mr Lansdown said the company he set up pays "an enormous amount" of tax.
"It's not a Starbucks or an Amazon, which have been getting bad publicity [for tax avoidance]. It pays its way.
"We pay an enormous amount of national insurance, corporation tax and PAYE on the staff.
"And those people are living there and working there and are paying their dues and their VAT in the country, so an awful amount of money is still going back into the UK from my hard work over 30 years.
"I think I've made a fair contribution to the country. I don't feel morally obliged to do any more," he added.
One of the world's richest men has said he has "no qualms" about moving from the UK to an offshore tax haven.