The claims, brought by the broadband industry trade group USTelecom and Texas-based provider Alamo Broadband Inc, are the first in what is expected to be a series of legal challenges.
USTelecom said the planned rules were misguided, but it did not object to "open internet" laws per se.
The US Federal Communications Commission (FCC) called the legal challenges "premature".
The FCC recently agreed a set of rules that would enshrine net neutrality - the principle that internet access cannot be blocked or slowed in favour of services that pay for so-called fast lanes - in US law.
But USTelecom said the rules, under which internet service providers would be defined as public utilities and more heavily regulated, were "arbitrary, capricious, and an abuse of discretion" and violated various US laws, regulations and rulemaking procedures.
The industry body's president, Walter McCormick, said its members supported the enactment of "open internet" principles into law, but not using the new regulatory regime chosen by the FCC.
"We do not believe the Federal Communications Commission's move to utility-style regulation... is legally sustainable," he said.
USTelecom's case was brought in the US Court of Appeals for the District of Columbia, which has rejected net neutrality regulations proposed by the FCC twice already.
Alamo challenged the new rules in the US Court of Appeals for the Fifth Circuit in New Orleans, making a similar argument to that made by USTelecom.
According to the Reuters news agency, industry sources have previously said USTelecom and two other trade groups - CTIA - The Wireless Association, which represents the wireless communications industry, and the National Cable and Telecommunications Association, which acts for the US cable television industry - were expected to lead legal challenges.
Verizon Communications Inc, which won a 2010 case against the FCC, was unlikely to mount its own legal challenge this time around, an industry source familiar with Verizon's plan told the agency.
The FCC said the legal challenges were "premature and subject to dismissal."
Officials added they had been prepared for legal action and the new rules were on much firmer legal ground than previous iterations.
The FCC's rules have yet to be published in the Federal Register and formally go into effect.
USTelecom, in its legal action, said it had filed the challenge on Monday in case the rules were construed to be final on the date of issue.
In a blogpost, the Center for Boundless Innovation in Technology - a free-market group - said the FCC's net neutrality rules were ideological and removed from the "real internet... [where] congestion is commonplace and the interests of content owners are divergent".
It referred to a Wall Street Journal report that major streaming networks were already seeking to separate their services from the public internet in search of higher speeds.
In Europe, regulators have also proposed allowing services other than those used to access the public internet a fast lane if they require a higher quality connection to function.
However, they sought to avoid a detrimental effect on the quality of service to internet users.
What was that article about?
US broadband providers have filed legal challenges to new net neutrality rules.