Arguing that he was acting in self-defence, Ian Anderson, 45, said he "was fighting for [his] life" when he kicked and punched Ballabeg man Neil Roberts.
Mr Roberts, 60, died at Mr Anderson's house in Castletown on 1 December 2013.
Mr Anderson denies Mr Roberts' murder after confronting him over the affair with his wife Alison.
"It was like I wasn't there, I stopped being who I am because of the continued assault on me, he said.
"I don't know who I became but I didn't murder Neil Roberts, I would not have done that, the person who I am and the person I was then wouldn't have done that".
Mr Anderson described how Mr Roberts became "agitated and confrontational" after being asked whether he was in love with Mrs Anderson.
"He started hitting me, the first was a heavy blow which shook my head - then the punches came in, I felt at least three or four.
"I felt nauseated and confused - I could taste the blood in my mouth.
"He kept coming at me again and again and I thought this man wants to kill me, he wasn't going to stop hitting me," said Mr Anderson.
"His arms were swinging around and I just remember punching him away to protect myself."
The court was told by the prosecution that during the following hour both men sustained injuries.
Prosecutor Linda Watts told the court in Douglas that during the fight Mr Anderson suffered a black eye, cuts and extensive swelling and bruising.
She added that Mr Roberts suffered 23 fractures to his ribs, a fractured sternum, broken facial bones and extensive internal organ damage.
Mrs Watts said: "You broke just about every bone in his face didn't you?"
"I don't know," replied Mr Anderson. "I was using my hands and feet to keep him away from me."
"You kicked his head like a football, didn't you Mr Anderson?" continued Mrs Watts.
"I don't recall, I don't remember. The person I am and the person I was would not have done that," he said.
Mr Anderson denies murder, and the trial continues.
A man accused of murdering his wife's lover in a fight has described himself in a Manx court as "compassionate" and "incapable of harming anyone".