The schools watchdog is challenging a High Court ruling last year clearing an Islamic mixed-sex faith school of unlawfully segregating the pupils.
Birmingham's Al-Hijrah school separates boys and girls from the age of 10.
Three appeal judges reserved their judgment after a two-day hearing.
In a report last summer, Ofsted ruled that the mixed-sex school was inadequate, saying that its policy of separating the sexes was discrimination under the 2010 Equality Act.
But in November, High Court judge Mr Justice Jay overruled the inspectors, saying that they had taken an "erroneous" view on an issue "of considerable public importance".
Amanda Spielman, England's chief inspector of schools, who appeared in court in person, is appealing over the segregation ruling.
Ofsted says that Al-Hijrah admits boys and girls aged four to 16 but completely separates them from Year 5 for lessons, breaks, school trips and school clubs.
This amounts to direct sex discrimination which particularly impacts on girls and leaves them "unprepared for life in modern Britain", says the watchdog.
Peter Oldham QC, appearing for the school's interim executive board, asked the judges to dismiss the appeal.
Mr Oldham said boys and girls at Al-Hijrah, which is maintained by Birmingham City Council, were "treated entirely equally while segregated" and that was lawful.
He said Ofsted did not claim that separation was discrimination until 2016 and its actions were "the antithesis of proper public decision making".
Helen Mountfield QC, appearing for Ofsted, offered an apology to the court for previous inspections.
Ms Mountfield said: "Ofsted recognises this was not picked up earlier, or at other schools. This is the first case where it has been picked up."
The QC told the judges that if the appeal was allowed, Ofsted would reinspect all mixed-sex schools with segregation policies which might be breaching equality laws.
She said the total number of mixed-sex faith schools which had been inspected amounted to 677 maintained schools and 55 independent schools.
"Ofsted has identified two which definitely segregate on grounds of gender and 16 which may do," she added.
A judgement is expected at a later date.
Schools in England which segregate boys from girls will face fresh inspections if education watchdog Ofsted wins a test case over sex discrimination, the Court of Appeal heard on Wednesday.