Omar al-Bashir was allowed to attend an African Union summit in South Africa despite being wanted by the ICC on charges of genocide and war crimes.
After a cabinet meeting, the government said it would review membership "for a number of reasons".
But it stressed it took its international obligations "seriously".
A South African court had ordered Mr Bashir to stay in the country while it ruled whether he could be arrested. The government said he enjoyed diplomatic immunity.
Mr Bashir denies allegations committed atrocities in Sudan's troubled western Darfur region.
A senior South African minister, Jeff Radebe, said his government had done nothing wrong, and always obeyed the law.
He was speaking a day after a judge had angrily accused the government of ignoring the constitution and pushing South African democracy towards collapse.
At issue: a government decision to allow Sudan's President, Omar al-Bashir, to leave South Africa last week, in direct defiance of a court order.
Mr Radebe said the International Criminal Court's rules about arresting a head of state were contradictory, and he accused ICC officials of failing to consult, and of not acting in good faith.
This issue is unlikely to fade away, with a number of senior South African officials facing possible criminal charges, for their role in allowing Sudan's president to fly home.
What is the International Criminal Court?
South Africa had "to balance its obligations to the ICC with its obligations to the AU and individual states", the South African government tweeted.
The country may consider withdrawing from the ICC as a "last resort", it added.
Previously, the AU has urged member states not to cooperate with the ICC, accusing it of bias against Africa.
The South African government is due to explain its decision later on Thursday, although its statement may not be made public.
South Africa has said it might leave the International Criminal Court (ICC) after a row over the court's attempt to have Sudan's president arrested there.