The poem, which appears to ask soldiers to shoot protesters in the Indian-ruled territory, went viral after being tweeted by the Digital India account.
It was later deleted but not before being retweeted by several people, including an opposition politician.
A Twitter user, Sarthak Garg, owned up to the "mistake" and apologised.
His message was retweeted by Digital India itself.
The agency has been warned it may lose its contract, a government official told the BBC.
He added that they had apologised for the incident.
The Hindi-language poem told the army to "lock the doors so no one could escape" and said "those who do not like it here can go graze buffaloes in Pakistan".
It made headlines in India, with many expressing shock.
Kashmir is claimed in its entirety by both India and Pakistan and has been a flashpoint for more than 60 years, sparking two wars between the countries.
The agency managing an Indian government Twitter account has been asked to explain after it tweeted an inflammatory poem about Kashmir.