That was the begining of the empire, not the end. It ended when Muslims sacked Constantinople after the Western crusades weakened it at the insistence of the Pope. Rome the city became a backwater when the empire left. Its see was venerated but not authoritative.