Developer Pye Homes was granted planning permission by West Oxfordshire District Council for land north of Woodstock's Bladon roundabout.
Originally it wanted to build 1,200 homes, but was forced to scale back after criticism.
Woodstock town councillor Sharone Parnes said there was still "a lot of concern" about the development.
Last year 411 of 484 people who voted in a referendum on the plans - a 19% turnout - said they did not want them to go ahead.
Mr Parnes said: "There's a lot of concern about the road infrastructure and the traffic capacity and the amount of traffic that will inevitably result.
"We don't see how the roads are going to be adapted for that."
In a letter to the district council, the town council wrote that the "historic" open setting of Blenheim Palace should be preserved.
It said: "The setting [allows visitors] to gain a better understanding of an aspect of the social and cultural significance of aristocratic parks exemplified by Blenheim."
Daniel Hayman, spokesman for Pye Homes and the Blenheim Estate, said the housing was "much needed" and the scheme also included some business units.
"We strongly believe there is a need to deliver a range of new homes in Woodstock and this proposal will help to provide much needed new affordable homes in the area," he said.
Plans to build 300 homes on fields owned by Blenheim Palace have been approved, despite local opposition.