An eyewitness at Al-Shifa hospital told the BBC that troops moved in overnight and were interrogating people.
On Wednesday, Israel said it found Hamas' "operational centre" at the hospital, sharing images of what it said were Hamas weapons and equipment.
Hamas denies operating there and the BBC cannot independently verify claims by either side.
UN humanitarian chief Martin Griffiths said he was "appalled" by the Israeli raid, while the World Health Organization (WHO) said it was "extremely worried" for patients and staff, with whom it had lost contact.
The BBC has been speaking to a journalist and a doctor inside the hospital to find out what is happening there, while the Israel Defense Forces (IDF) has also been providing updates.
Khader, a journalist inside Al-Shifa, told the BBC's Rushdi Abualouf that Israeli troops were in "complete control" of the hospital and no shooting was taking place.
He said six tanks and about 100 commandos entered the complex during the night. The Israeli forces then went room to room, questioning staff and patients.
The IDF reportedly asked all men aged 16-40 to leave the hospital buildings, except the surgical and emergency departments, and go to the hospital courtyard.
Soldiers fired into the air to force those remaining inside to come out, Khader said.
However Muhammad Zaqout, the Hamas-run Gaza health ministry's director of hospitals, told Al Jazeera that "not a single bullet" had been fired - because "there are no resistors or detainees" inside.
Early on Wednesday, the IDF said its forces were in the midst of a "precise and targeted operation against Hamas in a specified area" in the hospital.
It said the raid was "based on intelligence information and an operational necessity", calling for the surrender of "all Hamas terrorists" present there.
As the soldiers entered the hospital complex, they engaged with a number Hamas members and killed them, the IDF said.
On Wednesday evening, the IDF said troops found "an operational command centre, weapons, and technological assets" belonging to Hamas inside the MRI building.
It said it was "continuing to operate in the hospital complex", sharing images and videos showing what it said were Hamas weapons.
In a seven-minute video, IDF spokesperson Jonathan Conricus pointed to security cameras that he said had been covered over, and weapons that he said were AK47 rifles hidden behind MRI scanners.
The BBC has not yet been able to verify the video or its location.
But unless Israel has more to reveal, the military's controversial operation inside the hospital did not net a major arsenal of weapons, reports the BBC's Orla Guerin in Jerusalem.
Hamas knew Israel was coming, and therefore, if they were operating beneath the hospital, they would have had weeks to clear out through Gaza's extensive tunnel network, our correspondent adds.
Israel's Army Radio reported that troops had not yet found any sign of any of the 240 hostages taken by Hamas during the 7 October attack on Israel, when 1,200 other people were killed.
The raid on Al-Shifa came shortly after the US publicly backed Israeli claims that Hamas had infrastructure underneath the hospital.
However Dr Ahmed Mokhallalati, a plastic surgeon at Al-Shifa, told the BBC there were only civilians in the hospital.
He said there were tunnels under every building in Gaza, including Al-Shifa hospital.
According to international humanitarian law, hospitals are specially protected facilities.
This means that parties to conflicts cannot attack hospitals, or prevent them performing their medical functions. They can lose their protection if they are used by a party to the conflict to commit an "act harmful to the enemy".
The International Committee of the Red Cross (ICRC) says this could be something like ahospital being used as a base from which to launch an attack, as a shelter for able-bodied fighters, or to shield a military objective from attack.
The IDF said it had repeatedly warned Hamas its "continued military use of Al-Shifa jeopardises its protected status", calling for the evacuation of the hospital before the raid.
The WHO had warned that evacuating patients would be a "death sentence", given that the medical system was collapsing.
Dr Mokhallalati told the BBC on Wednesday the hospital was without power, oxygen and water. On Tuesday, surgeries had been carried out without proper anaesthesia, with patients "screaming in pain". Doctors were unable to help one patient with burns, and had to just "let him die".
Dr Mokhallalati said  six premature babies had died in recent days.
"Why can't they be evacuated," he said. "In Afghanistan, they evacuated the cats and dogs."
The IDF said it was providing incubators, baby food and medical supplies.
The Israeli military has raided Gaza's main hospital in what it describes as a "targeted operation against Hamas".