Ian Stewart, 56, denies murdering Helen Bailey and dumping her body in order to inherit her fortune.
He said one of the men attacked him on the day the Electra Brown writer vanished and warned he would not "see Helen again" if he spoke to police.
Mr Stewart claimed the men had hounded the writer over past business dealings.
Ms Bailey was last seen on 11 April and reported missing by Mr Stewart on 15 April 2016.
She was found beneath the garage at her home, together with that of her dachshund Boris, on 15 July.
The 56-year-old is accused of sedating and murdering his partner before dumping her body in a foul cesspit beneath their garage.
He told his trial at St Albans Crown Court that he lied to Ms Bailey's friends, family and the police about her whereabouts out of fear for her life.
Two men, whom he knew only as Nick and Joe, were said to have been old business associates of her first husband, John Sinfield.
Stewart claimed he was attacked by the tattooed thug called Nick on his doorstep last spring.
He told the court: "As I opened the door, he pushed me back into the hall and he must have tripped me at some point.
"He said: 'Helen is with us, she is helping us solve a problem, don't tell anyone'.
"He said: 'Sorry, we have taken Helen and Boris with us', he said 'we will be back we will see you Friday, if anyone asks tell them she's gone Broadstairs (Kent), don't tell anyone in any way'.
"He then said: 'If you tell anyone you won't see Helen again'."
Mr Stewart continued: "I was still on the floor, I was winded and my eyes were watering and he then hit me again and said: 'Do you understand?' and I nodded.
"He then quoted my telephone number at me and asked if that was correct and I just nodded.
"I was shocked and confused, it was so odd," he added.
The defendant claimed Joe had visited the house earlier and demanded if Ms Bailey knew about his old business dealings, before saying, in hushed tones: "Just think about this."
Mr Stewart, of Baldock Road, Royston, Hertfordshire, denies murder, preventing a lawful burial, fraud, and three counts of perverting the course of justice.
The trial continues.
The fiance of a children's author has told a murder trial she was snatched by two men who warned him not to go to the police.