Two Canadian firms - Charney Lawyers and Sutts, Strosberg LLP - are bringing the action against Avid Dating Life and Avid Media, which run the site.
They said they were acting on behalf on "all Canadians" affected by the breach.
Hackers, known as the Impact Team, stole the data last month.
It included users' names, email and home addresses, and message history. The hackers threatened at the time to reveal the information unless the site was taken down.
"Numerous former users of AshleyMadison.com have approached the law firms to inquire about their privacy rights under Canadian law," the two firms said in a statement.
"They are outraged that AshleyMadison.com failed to protect its users' information. In many cases, the users paid an additional fee for the website to remove all of their user data, only to discover that the information was left intact and exposed."
One plaintiff is named as Eliot Shore, a disabled widower from Ottawa. According to the statement, he joined the website for a short time after losing his wife, but never met up with anyone in person.
Ashley Madison says it operates in more than 50 countries and has 37 million users, more than a million of whom live in the UK.
It promotes its service with the tagline, "Life is short, have an affair".
Ashley Madison is facing a C$760m ($576m; Â£367m) class-action lawsuit after customer data from the infidelity dating website was leaked onto the so-called dark web last week.