Steve Bloom, who runs a second-hand bookshop in Hawes, North Yorkshire, has been the subject of more than 20 complaints to the parish council over the past four years.
Mr Bloom, who has been branded "rude and offensive", charges the fee which is refundable on purchase of a book.
Hawes and High Abbotside Parish Council said he was a discredit to the town.
More on this and other North Yorkshire stories
Mr Bloom, who described himself as "not really a people person", said the council had given the issue more importance than it deserved.
The 63-year-old admitted he should not have called a customer, who went on to complain to the council, "a pain in the arse".
"I regretted it as soon as I said it," he said.
"He arrived just as I was closing, but I allowed him to go in and browse around. But he ignored me completely when I asked for my 50p, which didn't help things at all."
He explained his policy of asking customers for 50p was a way of finding out whether they were serious or not, and that he did not actually take the money.
"There are times when I've wondered whether it's the right thing to do, but I stand by it," he said.
John Blackie, chairman of Hawes and High Abbotside Parish Council, suggested the "dreadfully rude and offensive" bookseller should change his ways.
"He can see the great difficulties, the upset that he causes in the local community," Mr Blackie said.
"He might be better off trying a charm offensive. If you charm them more and offend them less, you might have a business even better than it is now, and we can all live happily ever after."
A bookshop owner who charges an entry fee of 50p has admitted he was wrong to call a customer "a pain in the arse".