The Clanmil Housing Association had been planning to redevelop the 24-acre site.
In 2012, Clanmil signed an option agreement with the Irish National Assets Management Agency (Nama), which controls the debt related to the site.
In a statement to the BBC, Clanmil said it made an unsuccessful offer for the site before the option deal expired.
It added that the offer was based on "an independently assessed market valuation".
The association added that it was "extremely disappointed that, despite all our efforts, we have as yet been unable to find a viable way forward that would allow us to progress with the development of this site".
"The main barrier is the substantial cost of decontaminating the site in preparation for its regeneration."
Clanmil has not given up on the site, saying it is "keen to explore options with potential partners to take this project forward".
Hilden Mill was formerly home to the Barbour Thread factory which had occupied the site since 1823.
It closed in 2006 and three years later, planning approval was given for a redevelopment scheme that involved up to 600 homes.
The site is owned by Hilden Developments (NI) whose loans of £45m were transferred to Nama.
Those loans were not sold to Cerberus as part of the Project Eagle deal because the company is not Northern Ireland-owned.
It is ultimately controlled by the London-based developer Galliard.
A Nama spokesperson said: "This property is owned and managed by a Nama debtor so any queries relating to the sale of the property should be addressed to its owner."
The Hilden Mill complex near Lisburn, County Antrim, has been put for up for sale.