The event at Birmingham Botanical Gardens on Thursday was advertised as having "no bangers or loud noises".
A spokesman said he had been told by the company staging the event that illuminations would give off "a minimal amount of sound, with many being deemed as silent".
He said it would "not be staging this event with this provider again".
Updates on this story and others from Birmingham and Black Country
James Wheeler, botanical gardens chief executive, offered "a sincere apology for any distress caused" to younger visitors.
He said: "This is the first time that the gardens has staged this event...  We are sorry they were not as advertised.
"We have refunded event admission to those who have been in contact and provided proof of their attendance."
Parents sent messages to the attraction's Twitter account claiming the noise left children distressed.
Joanne Evenson, 36, from Bromsgrove, Worcestershire, who has a three-year-old son and an 18-month-old daughter, said: "The only silent [things] about that display were our mouths. We were speechless.
"We weren't going to go to one.... because of loud bangs. About 10 past 5 all of a sudden......the loudest fireworks you could ever hear in the sky.
"All the children were sort of fleeing with their parents and my friend took her little ones inside."
A Birmingham Botanical Gardens spokesman said: "From what I know silent fireworks are nowhere near as spectacular as the big ones.
"They are smaller in size in terms of visually compared to larger displays and they should be totally silent, including setting them off."
Refunds are being given after parents said loud bangs at a "silent fireworks display" frightened their children.