Officers were called to Wandsworth Prison in south London at 07:35 BST on Monday to reports a male prisoner was dead in a cell, Scotland Yard said.
The man, who has not been formally identified but is believed to be in his 60s, was pronounced dead at hospital at 08:52 BST.
A man in his 40s, also a prisoner, has been arrested on suspicion of murder.
A post-mortem examination is to take place in due course, a Yard spokesman said.
Wandsworth, a category B prison, is the largest in the UK and one of the biggest in western Europe, and can hold more than 1,800 prisoners.
A man has been arrested on suspicion of murder after a prisoner was found dead in a London jail.