Carl Wood, 58, allegedly took part on the first night of the raid over Easter this year when the gang failed to break into the vault.
The prosecution said he left before they got in on the second night.
After the crime, gang members said "Carl" would be "cursing himself".
Mr Wood is charged with conspiracy to commit burglary and conspiracy to conceal, convert or transfer criminal property, and is defending himself at Woolwich Crown Court.
While under police surveillance, gang members Terrence Perkins and Daniel Jones were also recorded saying the 58-year-old would be thinking of "committing suicide".
Mr Woods said they were talking about a different Carl. He admitted to being heavily in debt at the time of the burglary.
Earlier on Monday he said he was having a family barbecue and spent the evening at home at the time of the heist.
He did not originally tell police where he was due to legal advice, he said.
Prosecutor Philip Evans accused Mr Woods of lying to the jury after he told them he was bad with numbers and could not remember his daughters' birthdays or when he got married.
Mr Evans said: "You are making up your evidence as you go along, aren't you? You are lying to this jury aren't you?"
Mr Woods said he was not.
The 58-year-old told the court his friend Danny Jones, 60 - also on trial - had called him to apologise for not coming to his barbeque.
Mr Wood said Jones told him on the phone: "That thing I was talking about - just keep watching the telly."
He said when news broke of the raid he suspected Jones' involvement as Jones had a previous criminal record and so he knew what he was capable of.
Jones said Mr Wood had earlier alluded to being involved in something to do with a "lot of money".
Asked why there was no contact between himself and Jones after the raid, Mr Wood said: "I had semi-convinced myself Danny was involved and I didn't want to have anything to do with him, and I also assumed Danny was busy and didn't want to have any contact with me."
Jones, of Park Avenue, Enfield; John Collins, 75, of Bletsoe Walk, Islington; Terry Perkins, 67, of Heene Road, Enfield and Brian Reader, 76, of Dartford Road, Dartford, have all previously pleaded guilty to conspiracy to commit burglary.
The trial continues.
Defendants and charges
One of the men accused of the Hatton Garden burglary has denied being mocked by other members of the gang after he allegedly lost his nerve and left without any of the £14m haul.