While the judge does not condone racial prejudices and agrees that they should be fought, he also says that he cannot ignore that they exist and, if Ellen remains with Frank and Julie, will negatively impact her when she reaches adulthood.
The judge does not support or dismiss racial biases but recognizes their reality. If Ellen resides with Frank and Julie, the judge foresees these biases impacting her negatively as she grows older.