The dummies were metres from where Harlow Edwards was knocked down and killed in Coupar Angus last October.
Residents said the mannequins caused motorists to slow down, but drivers were now breaking the 30mph speed limit following their removal.
Perth and Kinross Council said any "unauthorised signage" was illegal.
Two-year-old Harlow was killed and her six-year-old brother and 17-year-old sister were seriously injured after being struck on the A94 Forfar Road at Larghan Park.
Jimmy Harris installed the mannequins with another local resident last week.
He said: "How can they say a safety measure like that is distracting when it was actually having an impact on reducing the speed of the cars?
"There are signs up and down the street, business signs.
"They've put a big metal horse up in Coupar Angus and that's distracting."
Mr Harris said vehicles regularly drove at speeds of up to 60mph on the road.
He said: "After the accident they slowed down a wee bit but now they're picking up speed again."
Local resident Blair MacDonald said: "They were so effective, it was unbelievable.
"I can't get my head round why they would want to take down something that was actually having an impact.
"It is just crazy, the speeds you see people going up and down that road."
Perth & Kinross Council said it was consulting local elected members and community councils in relation to traffic calming measures.
Councillor John Kellas, enterprise & infrastructure convener, said the first part of the changes, involving an extension of the school-time 20mph limit on Forfar Road and an extension of the 30mph on Coupar Angus Road, would go before councillors on 8 March.
He said: "The council is aware that individuals have placed additional material at the side of the road to discourage speeding.
"Although the council appreciates that this has been done with the best of intentions, it is illegal for any unauthorised signage to be erected at the side of the road as they may distract drivers and could jeopardise any enforcement action taken by Police Scotland."
Two unofficial traffic calming mannequins placed close to where a toddler died after being hit by a car have been removed by council officials.