Journalist Niki Savva's book The Road to Ruin scrutinises Mr Abbott's close relationship with his ex-chief of staff Peta Credlin.
The book reveals that Liberal Party senator Connie Fierravanti-Wells told Mr Abbott he needed to sack Ms Credlin.
"Rightly or wrongly, the perception is that you are sleeping with your chief of staff," she told Mr Abbott.
"That's the perception and you need to deal with it."
The senator also reportedly spoke to Ms Credlin, telling her: "One day, Tony will be sitting on a park bench in Manly feeding the pigeons, and he will blame you."
Ms Fierravanti-Wells confirmed to Australian media that the quotes attributed to her in Savva's book were accurate.
She said Mr Abbott responded by denying the affair rumours and refused to distance himself from Ms Credlin.
The book also claims that Mr Abbott was seen slapping Ms Credlin on the buttocks, and that Ms Credlin fed Mr Abbott at a restaurant using her fork.
It details the influence that Ms Credlin wielded in the prime minister's office and quotes on-the-record sources who criticise her working style as overly controlling and confrontational.
When questioned about the claims, Mr Abbott responded: "I do not respond to scurrilous gossip, full stop, end of story."
Ms Credlin said Savva did not contact her about the allegations before publication.
"This book says a lot more about her lack of ethics than it will ever say about me," Ms Credlin told News Corp.
But Savva defended her decision not to check the claims with the former prime minister and Ms Credlin.
"What I decided was that Tony Abbott and Peta Credlin any day, any night, can get out there and give their version of events, and their version of events often differs very widely from everybody else's," Savva told the Australian Broadcasting Corp.
"There are people who had been abused for years during that administration, who suffered in silence, and I thought they should be given the chance to tell their story."
Savva said that while Mr Abbott and Ms Credlin may not have been having an affair, their conduct destabilised the party.
The book also alleges Ms Credlin "sidelined" Mr Abbott's wife Margie on various occasions and saw her as a "distraction" for the prime minister.
Australia's former prime minister Tony Abbott has refused to comment on "scurrilous" claims made in a new book.