During an interview with Sirius XM radio on Tuesday, Mr Carson suggested people are poor because they learned the "wrong mindset" from their parents.
The retired neurosurgeon oversees a department that manages housing for the country's low-income population.
His comments quickly drew sharp criticism on social media.
"I think poverty to a large extent is also a state of mind," he said in an interview that aired on Wednesday.
"You take somebody that has the right mindset, you can take everything from them and put them on the street, and I guarantee in a little while they'll be right back up there.
"And you take somebody with the wrong mindset, you can give them everything in the world, they'll work their way right back down to the bottom."
The interviewer was Mr Carson's own top adviser during his failed 2016 Republican presidential campaign.
History according to Ben Carson
Mr Carson also suggested there was a "poverty of spirit" and bad parenting could result in "the wrong mindset" and "defeatist attitude".
Mr Carson has previously spoken about his experience of climbing out of poverty in Detroit, Michigan, to become a famous neurosurgeon.
During the interview, he attributed his success to his mother.
"If everybody had a mother like mine, nobody would be in poverty," he said.
"She was a person who absolutely would not accept the status of victim."
Shortly after, Mr Carson faced backlash on Twitter from users including Star Trek actor George Takei.
The interview was recorded as the White House touted its 2018 budget, which proposes cutting more than $6bn (Â£4.6bn) from the Housing and Urban Development department.
But it was not the first time Mr Carson has stirred controversy.
In March, he called African-American slaves "immigrants" and in 2013 he said Obamacare was the "worst thing that has happened in this nation since slavery".
US Secretary of Housing and Urban Development Ben Carson has sparked outrage after describing poverty as a "state of mind".