At least 170 passengers are reported to have been on board, but so far the coastguard has only rescued 17 people.
Separately, unidentified planes are reported to have bombed militant positions near Tripoli airport, leaving at least 10 people dead.
Libya has been in turmoil with fighting between rival militias who led the 2011 uprising against Col Muammar Gaddafi.
The violence has been centred around the airport and in the eastern city of Benghazi.
Local fishermen raised the alarm at dawn on Saturday, saying that a boat had capsized in the sea east of Tripoli.
It had disappeared underwater by the time rescuers arrived at the scene, and the few people who were still alive were surrounded by the bodies of those who were not, coast guard official Mohammad Abdellatif told Reuters.
Coastguard officials said they had no boats of their own, so had to commandeer fishing vessels.
The vessel was one of many that has set out from North Africa in recent months, heading for Europe.
These boats often come from Libya, where human traffickers are thought to be exploiting the current political turmoil.
Many boats are badly over-loaded and un-seaworthy, correspondents say.
Meanwhile, people living near Tripoli airport heard loud explosions early on Saturday morning.
According to the local TV channel al-Nabaa, war planes attacked fighters from a coalition of Islamist factions, most of whom are from the eastern city of Misrata.
The Misrata group has taken the army base near the airport from a rival militia, who come from Zintan, west of the capital.
Forces of renegade General Khalifa Haftar have claimed previous air strikes, but analysts doubt whether his forces have such planes at their disposal.
Libyan military planes are not equipped to make night flights, and cannot be refuelled mid-flight, says the BBC's Rana Jawad, in Tripoli.
More than three years after Gaddafi was overthrown and killed, Libya's police and army remain weak in comparison with the militias who control large parts of the country.
Dozens of people are thought to have drowned after a boat loaded with migrants sank off the coast of Libya.