While the judge does not condone racial prejudices and agrees that they should be fought, he also says that he cannot ignore that they exist and, if Ellen remains with Frank and Julie, will negatively impact her when she reaches adulthood.
The judge neither endorses nor ignores racial prejudices; however, he recognizes their presence. Should Ellen choose to live with Frank and Julie, the judge anticipates that these biases could harm her later on in life.