Less than 1% of 87,230 sex assaults carried out in the UK targeted victims over 60, Durham University researchers found.
But the team believe this is the tip of the iceberg and many older victims are being put off reporting the crime.
They also found that more than 65% of attackers who targeted the elderly, were under 60 years old.
Researchers believe the study, published in the British Journal of Criminology, challenges the stereotype that sexual violence only affects the young.
They are calling for police forces to be more aware of older victims when carrying out rape and sexual assault awareness campaigns.
Figures, obtained via Freedom of Information requests to 46 UK police forces, revealed that from 2009 until 2013, 655 people over 60 reported being the victim of a sex attack.
Researchers also interviewed victims and many said reported feeling "invisible" and unable to report the crime.
Co-author of the study Hannah Bows, a PhD student in the School of Applied Social Sciences at Durham University, said the findings challenged rape victim stereotypes.
She said: "Our study does show that rape happens to people in later life - rape stereotype tends to centre on a young attractive female attacked by a male stranger late at night.
"If you do a search for rape campaigns, the common images are of young attractive female victims in 'going out' clothing drawing links between age, gender and sexual desirability being the core risk factors for rape as well as alcohol."
They also found that one in five of attacks on older victims took place in care homes and attackers were usually known to their assailants.
Sex attack victims over 60 are failing to report assaults and are being made to feel "invisible", a study has found.