Analysts at investment bank Morgan Stanley said lower migration rates, tighter financial lending rules and over-supply would blunt housing growth.
Australia's official cash rate is a record low 2% but regulators recently tightened lending rules for investors.
Morgan Stanley predicts a fall in the number of houses sold at auction, and slowing price growth.
A prolonged period of low interest rates was meant to help the sector but tighter lending rules were squeezing the market, the research said, according to local media.
"Despite common belief that lower-for-longer Reserve Bank of Australia rates will see strong housing conditions persist, we think macro-prudential [lending rules] is effectively tightening policy settings," Morgan Stanley said.
"We are now calling the peak in the housing cycle, and expect further falls in auction clearance rates and house-price momentum, with a negative impact on construction occurring over 2016."
The report comes ahead of the first spring auctions in Sydney, this weekend, which will give an indication of whether this year's strong buying interest is tapering off.
At a median price of A$1m ($710,000, Â£460,000), Sydney houses prices are among the world's most expensive.
Prices in some other capital cities have already cooled.
Australia's bank regulator, the Australian Prudential Regulation Authority (APRA), has imposed conditions on lenders in a bid to keep total investor loan growth at 10%  for 2015.
The number of immigrants entering Australia has also slowed at a time when housing construction, particularly of apartments, has increased.
Morgan Stanley said any slow down in the housing market would coincide with a slow down in Australia's resources sector, outweighing any positive contribution to the economy from a lower Australian dollar.
Australian housing sector has "peaked", increasing the risk of recession, according to new report.