The letter, signed by hundreds of staff and students, was prompted by previous comments made by the historian.
In response, the university said it had taken down the video earlier than it had planned to.
Starkey has refused to comment on the allegations of racism.
Starkey, who attended Fitzwilliam College, Cambridge, took part in the three-minute video "Dear World", which featured other university alumni such as Sir Ian McKellen, Professor Stephen Hawking, and Lily Cole.
The letter calling for the removal of the video - made in a bid to raise Â£2bn for the university - alleges Starkey has "repeatedly made racist statements".
It cites a 2011 Newsnight interview in which the constitutional historian said: "A substantial amount of the chavs have become black. The whites have become black; a particular sort of violent destructive, nihilistic gangster culture has become the fashion."
Referring to his appearance in the video, the letter says: "In our eyes, Starkey's presence both undermines and taints our daily efforts to function as a united community."
The university said it had always intended that the current video would be replaced with a new one, and that whilst they appreciate Starkey has made "controversial statements in the past", he was "representing his affection for the university and its values."
Dr Malachi MacIntosh, director of English Studies at King's College, who signed the letter said "we're glad to see the video taken down".
"We're trying now to arrange to speak with the University about how best to proceed," he said.
"Our ultimate goal is to ensure these kinds of campaigns truly represent the best of Cambridge."
Starkey has not commented to the BBC, but earlier told The Independent: "I shall reserve the right to comment freely but without recrimination."
An open letter has led to the removal by Cambridge University of an online video featuring David Starkey after a row over racism.