Lord Ahmed of Rotherham said he opened the envelope in his office and was told to stay where he was as it was treated as a chemical or biological incident.
The independent House of Lords member said the powder was later found to be harmless.
Police said they were called at 12:30 GMT to reports of a suspicious package.
"We are examining the contents," a spokesman said.
Lord Ahmed told the BBC the white powder spilled over him and the envelope had contained a note with the words "paki filth".
He said: "The powder went all over my clothes and I immediately shook it off. I realised this powder could be anything.
"It was frightening, and I think that even more than it being frightening for me, it was a big shock for everyone who was around me because security personnel came and told everyone to be calm and remain seated, and you could see the police officers with guns."
He said every five minutes medics were looking at his face to "see if it had turned blue".
He said he thought of the recent death of MP Jo Cox who was killed in a street of her constituency in Birstall, West Yorkshire.
"I immediately thought that this could be more serious than just an abusive letter," he said.
Lord Ahmed said he was "concerned" by what happened and said he had been targeted by racist messages about a dozen times before.
He believed he received those messages because of his status as the first Muslim life peer in the House of Lords, he said.
There has been a security alert in Parliament after an envelope containing white powder and a racially offensive note was sent to a Muslim peer.