"I did not resign. Moments ago I was fired," said Preet Bharara on Twitter.
Mr Bharara was first asked to remain in his post by President Donald Trump, when they met after the election.
But on Friday he was included in a list of prosecutors appointed by former President Barack Obama who were asked by the Justice Department to step down.
Mr Bharara's tweet continued: "Being the US Attorney in SDNY (South District New York) will forever be the greatest honour of my professional life."
Presidents often order appointees of the previous administration to resign but the decision to replace so many in one swoop raised eyebrows.
Mr Bharara's inclusion in the list came as a particular surprise. He told reporters in November that he had been asked by Mr Trump to stay on, and had agreed.
The prosecutor rose to prominence after pursuing high-profile corruption cases and cases against Wall Street bankers.
He won a $1.8 billion ($1.5 billion) insider-trading case against hedge fund SAC Capital Advisors, the largest in history. SAC was forced to close down.
He has also prosecuted both Democratic and Republican lawmakers.
Among his current cases was an investigation into fundraising by New York mayor Bill de Blasio, as well as a sexual harassment case against Fox News by its employees.
After it was reported on Saturday that he had refused to resign, top New York Republican Brian Kolb tweeted: "Good for Preet, he is doing the job he was appointed to do!"
On Saturday, New York Attorney General Eric Schneiderman said in a statement: "President Trump's abrupt and unexplained decision to summarily remove over 40 US attorneys has once again caused chaos in the federal government."
"Preet Bharara, like many of the US attorneys dismissed this week, served with honour and distinction."
However, ex-Republican strategist Brian Walsh argued that the firing was "nothing new" as "US attorneys serve at the pleasure of the [president] - even Preet".
Mr Bharara's office and the Justice Department refused to comment further, according to the Associated Press.
A New York federal prosecutor who refused to resign when he and 45 other prosecutors were asked to by the Trump administration says he has been sacked.