Newlands Park, off the M1, would see offices, shops and a hotel built to support the development of Luton Town's proposed Power Court stadium.
A council report said it could damage "trade, footfall and investment".
The club said there was "no evidence" the scheme would have a negative impact and concerns had been addressed.
For more on this story and other Bedfordshire news
Gary Sweet, the Hatters' chief executive, said: "We have conducted an in-depth assessment of the impact on the town centre and our applications show there is no net impact.
"Newlands Park will be an iconic gateway development which will genuinely attract new retail into our town.
"We think it will be considered as one of the most unique and the best places to shop in the country - but it will be distinctly different to our town centre, therefore it will not replace it."
Luton Borough Council's strategic planning team report said, from a "strategic planning perspective", there were some "significant concerns" about the "potential adverse town centre impacts".
It said "significant weight" should be given to the impact on the "vitality and viability" of Luton town centre.
The proposals are yet to be considered by the council's development control planning officers who are responsible for assessing the application.
The borough council said the plans would be considered in the new year.
Luton Town identified the Power Court site, near the railway station, as the ideal location for a new stadium in December 2015.
Luton has been at its current stadium, Kenilworth Road, since 1905. It wants to move into the new ground by 2020.
Plans for a retail development to help fund a new football stadium could have an "adverse impact" on a town centre, council planners have warned.