Media playback is not supported on this device
Wenger was sent off for reacting angrily to a 93rd-minute penalty given to Burnley, who trailed 1-0.
Andre Gray converted to equalise but Alexis Sanchez then scored another penalty to give Arsenal a 2-1 win.
"I should have shut up - I apologise for not having done that," said Wenger.
"It was nothing malicious. I should have kept my control, even if it was in a hectic time."
Wenger could face action from the Football Association over the incident.
The Frenchman was sent to the stands by referee Jon Moss but stood at the tunnel entrance as he tried to watch the remaining few minutes.
Taylor appeared to tell Wenger to go down the tunnel but he refused to move.
"Look, it was nothing bad," added Wenger. "I said something that you hear every day in football. Overall, nine times out of 10, you are not sent to the stand for that.
"If I am, I am, and I should have shut up completely. I was quite calm for the whole game, more than usual."
Alan Pardew, then Newcastle's manager, was given a two-game touchline ban and fined £20,000 in 2012 when he pushed assistant referee Peter Kirkup during a game with Tottenham.
Media playback is not supported on this device
Burnley manager Sean Dyche, who was taking charge of his 200th game at the club, was also unhappy with the awarding of a penalty.
The 45-year-old felt that Arsenal defender Laurent Koscielny was offside when he was struck by Ben Mee's high foot in the 98th minute.
"It's a tough day for us in the end," he said. "To lose a game in that fashion, with an offside not given, is tough, particularly when you come to tough places like this.
"We have never taken anything for granted. We know how tough this division is but you need officials to make the right decisions and that is the shame today."
Arsenal had to play for 25 minutes with 10 men after Granit Xhaka was sent off for the second time this season.
The midfielder will be banned for the club's next four matches, including a Premier League match at current leaders Chelsea on 4 February.
"The challenge was on the other side of the pitch," added Wenger.
"If it is a red card, we just have to apologise and make sure it does not happen again because that could damage the team too much."
Arsenal manager Arsene Wenger said he "regrets everything" after he followed his dismissal by pushing fourth official Anthony Taylor during Sunday's dramatic victory over Burnley.