Extremely dry conditions mean the weeds pile up each day outside a row of homes at Wangaratta, in Victoria's northeast.
Frustrated residents are forced to clear out the weeds for several hours every day, with piles of hairy panic at times reaching roof height.
A nearby farmer is being blamed for failing to tend to his paddock.
"It's physically draining and mentally more draining," resident Pam Twitchett told Prime7 News Albury.
Wangaratta veterinary surgeon Richard Evans told the BBC the weed would lose its toxicity once it dried up.
"The important thing is it's not going to kill people's dogs and cats, it just makes a hell of a mess," he said.
Authorities are unable to help with the clean-up because the tumbleweeds do not pose a fire threat, reports say.
A fast-growing tumbleweed called "hairy panic" is clogging up homes in a small Australian town.