The remotely piloted Reaper plane was on an armed reconnaissance mission, seeking out IS fighters who had fired on advancing Iraqi government troops.
It spotted them planting improvised explosive devices and fired a Hellfire missile at them.
The RAF is providing air support for the Iraqi army and Kurdish Peshmerga forces in the fight against IS.
The attack follows recent successes on the ground in northern Iraq by the Kurdish Peshmerga, which is being trained and equipped by the British army as part of the coalition trying to reverse IS's advances in Iraq and Syria.
The Islamist militant group controls a large swathe of territory in both countries, where it has declared a caliphate.
On Thursday, Peshmerga forces were reported to have broken the long-running IS siege of Mount Sinjar, where thousands of Yazidis and other displaced Iraqis have been trapped since August.
The Kurdish offensive began early on Wednesday with the most intensive round of air strikes yet by US and coalition forces - 45 in all.
On the ground, about 8,000 Peshmerga fighters launched a two-pronged attack which they said had succeeded in opening a wide corridor to allow the besieged refugees to leave.
Meanwhile, the Pentagon's top officer, Gen Martin Dempsey, chairman of the US Joint Chiefs of Staff, said US air strikes had killed several high-ranking IS military leaders in Iraq.
Britain carried out its first drone attack on Islamic State militants in Iraq last month.
An RAF drone has launched a successful missile attack on fighters from the militant group Islamic State (IS).