Fareham Conservatives advertised a "drinks and canapÃ©s" evening with the health secretary that was moved after details started circulating online.
They said a smaller, members-only event was held instead because of "the very real threat of disturbance".
But doctors who bought tickets said they felt like they had been lied to.
Mr Hunt is engaged in a long-running industrial dispute with junior doctors who are fighting the imposition of new contracts.
The original event was advertised as being at the Lysses House Hotel, in High Street, Fareham, between 18:30 and 20:30 GMT on Friday, with tickets costing Â£15.
Conservative Association chairman Thomas Fyfe said: "The advertised event was cancelled on security grounds.
"A smaller private event for party members to meet Mr Hunt was held instead at another location.
"We regret the inconvenience caused to those who had bought tickets and were unable to attend, all of whom will be offered a full refund."
But Dr Kathryn Carey-Jones said the failure to tell non-Conservative ticket holders about the move was "dishonesty at its best".
She said: "I was angry, I was angry at the dishonesty in the way they let me know about this."
GP Emma Nash, who also bought a ticket, said on Twitter she also felt lied to by the "selective communication" used by the organisers.
Doctors have reacted with anger after being told an event with Jeremy Hunt was cancelled when it actually went ahead at a different venue.