Public broadcaster NHK and other media forecast Ms Koike as the winner after polls closed at 20:00 (11:00 GMT).
If confirmed, one of her key challenges will be curbing the financial problems plaguing Tokyo's preparations to host the 2020 Olympic Games.
Scandals linked to the Games forced the last two governors to resign.
"I will lead Tokyo politics in an unprecedented manner, a Tokyo you have never seen," Ms Koike, 64, told cheering supporters,
"While feeling the weight of this result, I want to thoroughly push the metropolitan government forward as the new governor."
Tokyo's ill-tempered election
In all, 21 contenders were vying to lead the sprawling capital and a number of other cities in the prefecture.
Ms Koike, politician Hiroya Masuda and journalist Shuntaro Torigoe were the front-runners.
Sunday's election was called after previous governor Yoichi Masuzoe resigned last month following fierce criticism over allegations that he used official funds to pay for holidays, art and comic books for his children.
Mr Masuzoe, who won election promising a scandal-free administration, denied breaking the law, but admitted to ethical lapses around his spending.
His predecessor, Naoki Inose, also quit over a funding scandal in 2013 soon after Tokyo won the right to host the Olympics.
Since then Tokyo's preparations for the 2020 Summer Olympics have been hit by scandals, overspending, administrative fumbles and construction delays.
One of the new governor's first duties will be to travel to Rio at the end of the 2016 Olympics in August to accept the Olympic flag as the next host.
Former Japanese defence minister Yuriko Koike looks set to be elected as the Japanese capital's first woman governor, exit polls project.