Adolf Hitler Campbell, 4, and his sisters, JoyceLynn Aryan Nation, 3, and Honszlynn Hinler Jeannie, 2, were taken from their New Jersey home in 2009.
The case first came to public attention in December 2008 after a shop refused to decorate a birthday cake for Adolf.
A family court had earlier found there was insufficient evidence of abuse.
But on Thursday, the appeals panel determined that social workers had proved the need for protective services for the three children.
The panel found that the parents, who both suffer from unspecified physical and psychological disabilities, had "recklessly created a risk of serious injury to their children by failing to protect the children from harm and failing to acknowledge and treat their disabilities".
The judges also noted that Deborah Campbell had recently passed a letter to a neighbour saying her husband had threatened "to have me killed or kill me himself".
The children's father, Heath Campbell, told the Associated Press last year that he believed they were taken into foster care because of their names.
He also alleged that the authorities had relied on unproven accusations made by a neighbour and his ex-wife.
A US appeals court has ruled a couple who gave their children Nazi-inspired names should not regain custody, citing the risk of serious injury to them.