The marathon was scheduled for Sunday and would have been Gaza's third.
Hamas said the marathon could go ahead if "local traditions" were respected.
Conservative elements in Gaza have sometimes complained about mixing between the sexes, especially in schools and at sporting events.
The UN Relief and Works Agency (Unrwa) said in a statement that it had taken "the disappointing decision" after "discussions with the authorities in Gaza who have insisted that no women should participate".
Unrwa "is working on a programme of other events, which will be forwarded to those interested as soon as possible," the statement adds.
"We regret this decision to cancel the marathon but we don't want men and women running together," Abdessalam Siyyam, cabinet secretary of the Hamas government, told AFP news agency.
"We did not tell Unrwa to cancel the marathon and we haven't prevented it, but we laid down some conditions: We don't want women and men mixing in the same place," he added.
The Palestinian territory is almost exactly marathon length from top to bottom.
Last year, thousands of runners braved freezing conditions to take part, including some women. Palestinian runner Nader al-Masri won the event on its first two occasions.
In previous years, Hamas has supported the race and provided security.
In the past there have been attacks on the UN's summer camps for children in Gaza after complaints that boys were allowed to mix with girls, the BBC's Jon Donnison reports.
The marathon was due to be part of the UN's fundraising efforts in order to run those camps, our correspondent adds.
The UN agency which organises Gaza's marathon has cancelled the event, blaming the refusal of the territory's governing Islamist Hamas movement to allow women to run.