That is the same argument that the PQ used when it tried to introduce a "Charter of Values", i.e. that a non muslim before a hijab wearing judge might feel that  he/she was not getting a fair hearing, depending on the issue.