Richard Barklie, from Carrickfergus in Northern Ireland, is one of four Chelsea fans facing a football banning order over a confrontation filmed ahead of a Champions League game in February.
Footage of the alleged incident showed several fans chanting: "We're racist and that's the way we like it."
Mr Barklie, 50, denies any wrongdoing.
He told Stratford Magistrates' Court he pushed Souleymane Sylla because there was no room in the train carriage, adding: "Mr Sylla, and it's my view, was the only one using aggression".
When asked by his defence barrister, Nick Scott, if "there was any issue in relation to the colour of his skin", Mr Barklie replied "none whatsoever".
Mr Barklie served as an officer with the Royal Ulster Constabulary and as a director with the World Human Rights Forum.
In addition to Mr Barklie, of Victoria Street in Carrickfergus, three others are challenging the ban: Jordan Munday, 20, of Ellenborough Road, Sidcup, south-east London; Josh Parsons, 20, of Woodhouse Place, Dorking, Surrey; and William Simpson, 26, of Hengrove Crescent in Ashford, Surrey.
The court heard Mr Munday, who is also accused of being involved in the incident, said there was enough space for him to force himself through the carriage and see the aftermath.
However he claimed he did not see the incident and had not been chanting "John Terry is a racist and that's the way we like it, ooh ooh ooh".
Adam Clemens, representing the Met Police, said: "At the end of the ooh ooh ooh your mouth was moving and it closes at the end."
Mr Munday told the court: "There was movement - I was breathing. I have to breathe."
Both Mr Munday and Mr Barklie were in a group of about 150 Chelsea fans walking through Paris on their way to the match.
Video footage appeared to show Mr Munday "fronting up" to a black man and shoving him twice, but Mr Munday claimed he had been stalked by the man and felt threatened.
A fifth man, Dean Callis, 32, of Liverpool Road in Islington, north London, earlier received a five year banning order for his role in Paris and other incidents involving violence.
The case continues.
A former policeman has admitted pushing a black man off a Paris Metro train but said the man started the altercation, a court has heard.