The players were asked to remove the Islamic head scarf before taking on Mongolia but refused and forfeited.
World basketball regulations list headgear and hair accessories among the items that are prohibited on court.
With no sign that the rule would be relaxed before their next match against Nepal, the team decided to pull out.
In a statement, basketball's world governing body Fiba said its regulations applied "on a global scale and without any religious connotation".
The statement added: "While certain groups have interpreted the provisions of the official basketball rules as a ban against the participation of players of certain faiths in basketball competitions, the uniform regulations are of a purely sporting nature."
Other sports at the Asian Games allow athletes to wear the hijab; all four members of the Iranian lightweight women's quadruple sculls team wore it as they rowed to a bronze medal on Wednesday.
Earlier this year, football's governing body Fifa officially authorised the wearing of head covers for religious reasons during football matches.
Basketball remains one of the exceptions, although Fiba said earlier this month that it had held discussions on the issue and was introducing a two-year testing phase on what players can wear.
The Olympic Council of Asia (OCA) issued a statement on Wednesday, saying: "The right of the athletes must be the highest priority."
After forfeiting the Mongolian match, Qatari player Amal Mohamed A Mohamed said they had been assured before they travelled to the Games in Incheon that they would be able to wear the hijab.
"We were told that we would be able to participate in matches by wearing a hijab," she said.
"We will not attend any games in this Asian Games unless the officials change their decision."
The Asian Games, held every four years, are recognised by the International Olympic Committee and feature more than 9,000 athletes from 45 countries competing in 36 sports.
The 2014 edition concludes on 4 October.
The Qatar women's basketball team have withdrawn from the Asian Games in South Korea after being denied permission to wear the hijab during matches.