A Beijing court on Friday cancelled the lawyer's probation saying that he had seriously violated probation rules,  the Xinhua news agency said.
Mr Gao went missing at the beginning of 2009 but resurfaced briefly in 2010 saying he had been tortured.
An outspoken critic of the government, he worked for the rights of some of China's most vulnerable people.
The BBC's Michael Bristow in Beijing says this is the first news about Gao Zhisheng for some time but it leaves questions about what happened to him still unanswered.
Mr Gao's brother told AP that he had not heard from the authorities on Gao's case, but said that he believed the imminent end of the probation period may have been behind the move.
"Are they sending him to a proper prison? Which prison was he at before? Where were they hiding him?" he asked.
Gao Zhiyi said he did not know what the latest court notice signified.
Analysts say the court's decision showed Beijing was determined to prevent Mr Gao speaking publicly about his time in detention.
"There's nothing there [in the Xinhua report] that tells you he is alive and well," Nicolas Bequelin of Human Rights Watch told AFP.
"At the very last minute... they decided they couldn't let the world see him or hear him and decided to take him away for another three years."
Gao's wife, Geng He, left for the United States in early 2009 with their two children. But his whereabouts have remained a mystery.
Human rights advocates often cite Mr Gao's case along with that of Liu Xiaobo, the jailed academic awarded the 2010 Nobel Peace Prize, as examples of what they say is the Chinese Communist Party's increasing persecution of human rights defenders in China.
Chinese human rights lawyer Gao Zhisheng has been sent back to jail for three years, state media reports say.