Richardson was born on 7 April 1934 in Edinburgh, Scotland. He was raised in Balgreen, Edinburgh. Richardson studied at Balgreen Primary School, at Tynecastle High School, at the George Heriot's School, and at the College of Dramatic Arts. He was married to Maroussia Frank from 1961 until his death in 2007. They had a son, Miles. Richardson died on 9 February 2007 in London, England from a heart attack, aged 72.