Judge Shira Scheindlin said police chiefs "turned a blind eye" to evidence that officers had discriminated by race when making the stops.
She did not order an end to the policy, but appointed an independent monitor to reform the practice.
Mayor Michael Bloomberg said he would appeal the ruling.
He warned it could end New York's tenure as "America's safest big city".
About five million stops have been made by city police over the past decade.
City officials have argued the policy has been key in reducing the city's crime rate.
In 2012, police stopped and frisked 533,042 people on the streets of New York, up from about 115,000 in 2002, according to the Center for Constitutional Rights.
That organisation brought the class-action lawsuit behalf of several New Yorkers who said they had been stopped without cause.
About 87% of those were black or Hispanic.
In about half of all cases, police merely asked questions. But in other cases authorities searched a bag or performed a full pat-down search.
About 10% of searches reportedly result in arrests, and sometimes weapons were seized.
During a 10-week trial, Judge Scheindlin heard from 12 people who had been stopped by police, as well as top officials with the NYPD.
Lawyers had argued the NYPD policed itself through several internal processes.
The judge rejected their arguments, writing in her opinion that paperwork by police alone showed at least 200,000 of 4.4 million stops between 2004-12 were made without reasonable suspicion.
"The city and its highest officials believe that blacks and Hispanics should be stopped at the same rate as their proportion of the local criminal suspect population," she wrote in her opinion.
"But this reasoning is flawed because the stopped population is overwhelmingly innocent, not criminal."
New York City's police department violated the civil rights of thousands through its stop and frisk policy, a US judge has ruled.