Bristol City Council declared a major incident after building surveys showed Barton House would be unsafe in the event of a fire or explosion.
People were told to pack enough clothes for "a day or two" and stay with friends and family, or go to rest centres, the council said.
About 400 people live in the building.
Council employees knocked on doors, sent texts and made contact "with everyone in the block" in a bid to empty the 15-storey building "as soon as possible", the authority added.
Nuh Sharif, who has lived in Barton House since 2012, said he was "panicking" about what he and his family would do.
A tearful Mr Sharif, who has two children, said his partner had been told the building "might collapse".
"We need to go somewhere quiet because they [his children] panic and can't sleep.
"I am worried where they are going to stay. How am I going to get them to school tomorrow?"
Yousif Ahmed, who lives in Barton House with his wife and three children, aged two, four and six, said the council "should have warned people earlier".
"They have just suddenly come knocking on the door saying you have to leave."
Mr Ahmed was packing some belongings in his car when he spoke to the BBC, and said he did not know where he would go.
The council said it had been carrying out building surveys on tower block in Redfield, which dates from 1958 and is the oldest in its estate, as it decided on the structure's long-term future.
During the surveys, experts discovered the structure could be "compromised" in the event of a fire, explosion or large impact.
Further surveys are planned but in the meantime, the authority said it was going door-to-door telling residents to leave.
Declaring a major incident allows the council, which has confirmed the issue is not related to RAAC concrete, to seek help from outside organisations.
Avon Fire and Rescue Service said it has been liaising with the council since it undertook the survey on the tower block.
"As a precautionary measure and to allow for further, more in-depth surveys, residents in the block are being asked to leave immediately," a spokesperson said.
"The approach the council are taking as responsible owners of the building is appropriate and proportional, we are in support of this to ensure that residents are kept safe."
Local GP surgery Wellspring Settlement said on X (formerly Twitter) it waslikely to be an information point.
The emergency services have been put on standby as a precaution, Bristol City Council said.
Another resident of Barton House said he found out about the evacuation from Facebook and a friend.
He said the tower block "should have been condemned years ago".
"My flat is terrible. It's full of damp and mould.
"Half my ceiling has collapsed literally into my bathroom."
Sheila Barrett said she was moving out and would and stay with her grandson.
"I am really, really shocked," she said.
"I have lived here for 50 years. I thought it was something to do with cladding, but have been told it is structural.
"I have been told it was something to do with a fire-brigade survey and we have to move out. It's a shock."
Hundreds of people living in a tower block were told to leave their homes immediately after "major structural faults" were discovered.