Boko Haram kidnapped 276 girls in April 2014, but freed 21 of them in October after negotiations with the Red Cross.
The freed girls have been in government custody since their release but were brought home to Chibok for Christmas.
But family members told the BBC that the girls were kept in a politician's house and barred from going home.
They were also prevented from attending church services with their families.
The girls were take to the house of an assembly member in Chibok to be reunited with their parents but weren't allowed to go to their own homes.
"I can't believe my daughter has come this close to home but can't come home," said one father.
"There's no point bringing them to Chibok only to be locked in another prison. They couldn't even go to church on Christmas Day."
Another said a soldier had confiscated his phone when he tried to take a picture of his daughter.
He said: "I snapped picture of myself and my daughter but the security guys came and grabbed me by shoulder and snatched the phone from my hands and told me to delete all the picture I took.
"I told him I'm taking a picture with my daughter who was away for more than two years. He said to me that's not his business, he deleted all the pictures including other pictures that were not taken there."
One mother said: "I can't believe my eyes that now my daughter cannot come home. How can I be happy when they don't have freedom?"
A statement from the office of state governor Kashim Shettima acknowledged that "armed soldiers... escorted the 21 girls to Chibok and remain their strict guards throughout the Christmas".
Speaking earlier this week, one of the girls told Reuters news agency it was a "miracle" that she was home and she was looking forward to church on Christmas Day.
"I never knew that I would return (home)," she said simply. "I had given up hope of ever going home."
Of the 276 students kidnapped, 197 are still reportedly missing, and negotiations for their release are under way.
Many of the Chibok girls were Christian, but were encouraged to convert to Islam and to marry their kidnappers during their time in captivity.
Ms Goni said some were whipped for refusing to marry, but otherwise they were well treated and fed until food supplies recently ran short.
Families of the Nigerian "Chibok girls" freed by Islamist group Boko Haram say they are being closely guarded and were not allowed home for Christmas.