They had said it could increase hatred and violence against LGBT people.
The health ministry has replaced the "gender identity disorder" category with one on gender and sexuality.
The contest for 13- to 24-year-olds is offering prizes of up to $1,000 (Â£780) for the best videos.
The other categories are sexual health and sex and the internet.
Homosexual activity is illegal in Malaysia under both secular and religious laws and is punishable by a prison sentence or corporal punishment.
Deputy Director-General of Health Lokman Hakim Sulaiman said the decision had been made after a meeting between ministry officials, the Malaysian Aids Council, experts and representatives from "key population groups".
He had previously defended the contest categories, saying their aim was to promote creativity, not to discriminate.
The guidelines had cited LGBT people as suffering from "gender confusion".
They said videos had to show the "consequences" of being LGBT as well as how to "prevent, control and [offer] ways to seek help".
Transgender activist Nisha Ayub said she had welcomed the ministry's change of course.
"We have to create a safe space for discussions and raise awareness. I hope this kind of engagement with officials will continue at other government agencies," she told Reuters news agency.
In March, the release of Disney's Beauty and the Beast was postponed in the country, because it contained a "gay moment".
Despite demands from the Malaysian censorship board, Disney refused to remove the scene, featuring LeFou, its first openly gay character.
The Malaysian board eventually conceded and it was shown uncut.
The Malaysian government has removed a category on "preventing" homosexuality from a sex education video competition for young people, following an outcry among activists.