I have never noticed rifles to be that loud. Shotguns maybe.
In any case I would think it advisable for ear protection to be worn when hunting anyway, since as you mention, the noise is still loud.
They should always be worn at a target range.
I hope people don't go hunting with a handgun!