The alleged fraud relates to management of the Â£285m prison escorting contract with the Ministry of Justice (MoJ).
Serco found evidence prisoners had been recorded as having been delivered ready for court when they were not, it said.
The firm said it will repay profits on the contract for transfers in London and East Anglia since 2011 - about Â£2m.
It will also give up future profits on the contract, which due to run until 2018.
The contract has been put under administrative supervision with immediate effect.
Recording prisoners as being delivered ready for court is a key performance measure for the contract.
Both the MoJ and Serco's directors have asked City of London police to investigate the actions of staff working on Serco's Prisoner Escorting and Custodial Services (PECS).
Serco said no evidence had been found that misreporting was undertaken with the knowledge or approval of management outside the contract.
Justice Secretary Chris Grayling said: "It's become very clear there has been a culture within parts of Serco that has been totally unacceptable, and actions which need to be investigated by the police.
"We have not seen evidence of systemic malpractice up to board level.
"But we have been clear with the company [that] unless it undertakes a rapid process of major change, and becomes completely open with government about the work it is doing for us, then it will not win public contracts in future.
"The taxpayer must know that their money is being properly used."
Chris Hyman, chief executive of Serco Group, said: "I am deeply saddened and appalled at the misreporting of data by a small number of employees on the contract."
A three-month programme has been agreed with the MoJ to "restore performance" on the PECS contract.
Evidence of potentially fraudulent behaviour emerged as part of the audit work announced by Mr Grayling in July.
The audit was launched in the wake of a scandal in which G4S and Serco were accused of overcharging the government by tens of millions of pounds on electronic tagging contracts - including for monitoring dead offenders.
The government has called in police to investigate claims of fraud involving the prisoner transfers contract held by security firm Serco.