The 32-year-old, who also won two World Championships, returned to training in 2015 after a three-year absence, during which she had two children.
But, after recent tests, Watkins felt she was not at the required level.
"This is not a decision that I've taken lightly," she said. "I guess I would have always wondered about what might have been if I hadn't given it a go."
In January 2014, following the birth of her first child, Watkins announced she would not defend her title in Rio.
But last year she told BBC Sport "the lure of the Olympics was just too strong".
Media playback is not supported on this device
Katherine Grainger, who partnered Watkins to double sculls gold at London 2012 and was unbeaten in 23 races with her between 2010 and 2012, came out of retirement in 2014 but formed a new partnership with Vicky Thornley.
They won bronze in the double sculls at last year's European Championships.
Watkins said: "Rowing has been such a big part of my life and I am sure that I will remain involved in some way.
"I am just taking some time to think this over, but I'll continue to support my team-mates in whatever way I can."
GB coach Paul Thompson said: "Anna is an outstanding athlete and an extremely ambitious and talented individual.
"Anna gave this her best shot and it unfortunately hasn't worked out. She is a fabulous ambassador for our sport and a great champion."
Olympic gold medallist Anna Watkins has withdrawn from the British programme for this year's Games in Rio.