Baker was born in Liverpool. His father was a sailor, and was Jewish. His mother was Roman Catholic. Baker was not very academic and struggled at school. He failed the eleven plus exam. Baker became a monk when he was 15. When he decided to change his life, he first worked in the Royal Army Medical Corps, and then as a construction worker. He took up acting first as a hobby, then professionally.