The Royal London Hospital has been told by the Care Quality Commission (CQC) that "lax" checking of name bands left babies and families at risk.
A total of 4,645 babies were born at the Whitechapel hospital last year.
Inspectors, who visited the hospital in July, also found there were not enough midwives on the delivery suite to provide safe cover for all women.
Barts Health NHS Trust, which runs the hospital, has been told to "urgently" improve security in maternity after rating the service "inadequate".
The CQC report said some babies born at the hospital had no name tags - creating a "risk that a baby might receive medication intended for another baby, and mother might leave the unit with the wrong baby".
Professor Sir Mike Richards, chief inspector of hospitals, said a number of issues had been raised with the trust, "as a matter for their urgent attention".
Issues included:
Sir Mike added: "We were most concerned about the standard of care around maternity and gynaecology services.
"Staffing on maternity wards was sometimes inadequately covered - but most worrying of all was the lack of a safe and secure environment for new born babies."
Barts Health NHS Trust said it had already taken steps to address baby safety concerns, including the introduction of new baby ID tags.
A spokeswoman said: "We acted immediately to improve the security of babies at The Royal London Hospital.
"It's important to stress that these reports are based on observations from five months ago - since then we have subjected our processes and procedures to forensic scrutiny.
Other key findings during the inspection were nine "never events" - wholly preventable errors that occur in medical care - between August 2015 and July 2016.
These errors included a surgeon leaving an object inside a patient after finishing an operation, a wrong tooth extraction and incorrect medication being given to a patient.
A maternity unit has been ordered to improve security due to fears parents could "leave with the wrong baby".