He was suspended from office in December pending a probe into his alleged ties with the former governor of the rich mining province of Ancash.
Ex-governor Cesar Alvarez is in jail while on trial for alleged corruption.
Mr Ramos is accused of covering up for Mr Alvarez while he was head of an auditing department.
Mr Ramos said the allegations were part of "a plot and conspiracy against me". He has five days to appeal against the decision.
A commission which oversees the judiciary voted five to one to fire Mr Ramos.
It said it had found links between Mr Ramos and Mr Alvarez, who is on trial for allegedly granting public work contracts in exchange for bribes.
Investigators said Mr Ramos had arranged to have prosecutors investigating Mr Alvarez fired.
Pablo Sanchez, who has been acting as interim attorney general since Mr Ramos was suspended, is expected to take over the post.
The sacking of the attorney general comes just six weeks after the prime minister was forced to step down over allegations that the country's intelligence services had spied on politicians, journalists and businesspeople.
Analysts say the scandals have hurt President Ollanta Humala, whose approval rating has plummeted to 24%, according to a poll conducted last month.
Peruvian Attorney General Carlos Ramos Heredia has been dismissed over corruption allegations which predate his time in the post.