Sarah Rochira said it should be easier to prosecute people such as health and social care providers.
Only 1% of crimes against older people led to convictions compared to 19% for all crime, the commissioner said.
The Ministry of Justice said those who abuse vulnerable elderly people "deserve tough punishments".
Ms Rochira said some "truly horrifying cases" had not resulted in criminal charges despite extensive investigation.
"Just how bad does care, abuse or neglect have to be before it is considered criminal?" she asked.
Ms Rochira made her comments to mark Elder Abuse Awareness Day on Monday.
"As it stands, the law simply does not offer sufficient protection for older people who are the victims of substandard care, abuse or neglect, nor does it provide a suitable deterrent to those who deliver substandard care or to those who abuse or neglect older people," she said.
"This is something that needs to change urgently."
She added: "The shockingly low prosecution and conviction rates for crimes against older people tell us that either the evidential test currently used by the CPS is not fit for purpose, that there is a lack of willingness to prosecute for crimes against older people, or that the law itself is not sufficient.
"A person's age cannot be allowed to be a defining factor in whether or not they have the support and protection of the justice system and the clear inequality that currently exists within this system must be addressed as a matter of priority."
The Ministry of Justice said victims deserved to be treated with "dignity and respect" by all agencies within the criminal justice system.
"Those who abuse vulnerable or elderly people deserve tough punishments. Sentencing guidelines used by the courts say that the vulnerability of the victim should be taken into account," a spokesman added.
"We await the letter from the Older People's Commissioner for Wales and her findings from the exploratory work in the summer."
Older people are not being protected by the legal system against abuse, neglect and other crimes, their commissioner for Wales has said.