As well as appearing on the long-running show, she taught Prince Rainier and Princess Grace to dance, performed for the Queen and choreographed a dance sequence for the Beatles.
Such was her prestige in the world of dance, she was asked to host Strictly Come Dancing but turned it down.
Spencer died peacefully while listening to waltzes, her daughter said.
Live updates: Tributes to Peggy Spencer, the 'doyenne of ballroom dancing'
Spencer, who moved to Norfolk in her later years, started teaching dancing in air raid shelters during World War Two and continued to teach until just two years ago.
In 1953 Come Dancing - which ran from 1949 to 1998 - became a dancing competition in which separate regions of the UK went head to head, battling for the trophy. Presenters included Terry Wogan, Michael Aspel, and Angela Rippon.
Spencer's role on the show was both as a choreographer and judge.
Daughter Helena Anderson described her mother as having had a "very illustrious career".
Mrs Anderson said: "She actually died in my arms yesterday morning and she was wearing her sparkly shawl and her sparkly gloves and she was wearing her favourite perfume.
"We were listening to Ross Mitchell's waltzes. And she just slipped away very peacefully."
Despite teaching ballet dancer Rudolph Nureyev the tango, Spencer particularly enjoyed instructing novices and encouraging them to take their first steps on the dance floor.
Mrs Anderson said if a beginner had approached her mother saying they had "two left feet", she would not be deterred, saying: "Come on, I can teach anybody to dance."
She added: "I shall miss her. I loved her dearly. I lived with her until four months ago when things got a bit difficult. She had had a long and wonderful life."
Dancing legend Peggy Spencer, known to millions of viewers for her role on BBC TV's Come Dancing, has died aged 95.