A senior police officer told the BBC that the man had been holed up in a house in Lucknow, the capital of northern Uttar Pradesh state.
He added that "the militant was part of an IS-linked group" which planned a train blast in central Madhya Pradesh state that injured nine on Tuesday.
Six more members of the group have been arrested from different locations.
The police operation happened on the eve of the last day of polling for critical assembly elections in Uttar Pradesh, India's most populous state.
Islamic State group: The full story
Battle for Mosul: The story so far
Police spokesperson Rahul Srivastava told the BBC that the operation took 12 hours because they "wanted to catch the gunman alive".
"We had information about two men hiding in the house from our colleagues in Madhya Pradesh. But we found one heavily-armed person hiding there," he said.
Mr Srivastava added that the police had "recovered more than 600 bullets, pistols and an IS flag from the house".
"We recovered a laptop and have also questioned the arrested men. These people were radicalised through social media, and were in contact with other IS members," he said.
Police in India say they have killed a militant allied with so-called Islamic State (IS).