One of the modernist writer's most famous novels, To The Lighthouse, was inspired by the view from a family holiday home in St Ives, Talland House.
Developers have put in plans to build six apartments opposite the house where Woolf spent her childhood summers.
Opponents say the development would block the view to Godrevy lighthouse.
To The Lighthouse, published in 1927, is set on the Isle of Skye but is widely acknowledged to have been inspired by Woolf's childhood summer holidays.
In an objection to Cornwall Council the writer's great-niece, Virginia Nicholson, said "legions" of admirers made the "pilgrimage" to St Ives to see the view, which "should remain unobscured for generations to come".
She said: "This would be an ill-advised, grasping and philistine act that would have made my famous ancestor turn in her grave."
Cecil Woolf, nephew of Woolf's husband, Leonard, said he was "appalled" by the proposals.
He said: "This is sheer vandalism and should be stopped now."
The Virginia Woolf Society Of Great Britain said granting planning consent would be an act of "irreparable folly".
Porthminster Beach View Ltd has submitted a planning application to Cornwall Council to build six self-contained apartments on Albert Road.
Planning documents submitted by the developer state the project will "ensure that there will be no harm to any protected areas and that the design of the development will respect local character and distinctiveness".
The developer was unavailable for comment.
The decision on whether to approve or reject the plans will be made by Cornwall Council on a date to be decided.
Relatives and fans of the author Virginia Woolf are trying to block development plans that would "destroy" a famous literary view.