Operators Serco told the Home Affairs Select Committee the dismissals related to eight out of 31 cases investigated.
Chief executive Dr Bob McGuiness said it set high standards and apologised for where it had "fallen short".
Committee chairman Keith Vaz said MPs would go there "very shortly".
"We are not just accepting people who come to the select committee and say that everything's alright in the garden and we say 'thank you very much'," Labour MP Mr Vaz said.
"We are a select committee of parliament and we have to pursue this."
The Bedfordshire centre houses up to 400 women under threat of deportation.
The MPs' investigation follows the centre's internal inquiry into the conduct of a male nurse allegedly involved in the sexual assault of a woman from Pakistan.
Bosses confirmed he was not one of those dismissed.
Senior staff at Serco, which runs the centre on behalf of the Home Office, told parliament on Tuesday that out of 30,000 detainees houses in the last seven years there had been 31 cases of alleged sexual abuse investigated.
Dr Bob McGuiness said: "We set very high standards for ourselves and our staff.
"On those instances where we have fallen short of the standards that we have set for ourselves, I absolutely am sorry and apologise for these cases."
Managing director James Thorburn, said the firm was not aware of any cases of inappropriate relationships currently taking place.
A report by HM Inspector of Prisons had found no evidence of a wider culture of "victimisation or systematic abuse", he added.
"If we knew of anyone who is having any inappropriate relationship with any detainee at the centre we would act immediately," he said.
MPs are to visit Yarl's Wood Immigration Centre after bosses admitted 10 staff were dismissed over allegations of improper sexual contact with female detainees.