The Counter-Terrorism Service said it had cleared the south-western area of Tamim on Tuesday after a fierce battle.
Soldiers and militiamen have been preparing for months for a final assault on Ramadi, and recently told residents to leave the city centre.
Ramadi was captured by IS in May in an embarrassing defeat for the Iraqi army.
Last month, government forces completed their encirclement of the city when they retook the Palestine Bridge, which straddles the River Euphrates in the north-west.
The move cut off IS militants inside Ramadi, about 90km (55 miles) west of Baghdad, from their strongholds elsewhere in Anbar province and in Syria.
Iraqi Counter-Terrorism Service spokesman Sabah al-Numani told the AFP news agency that after troops launched their assault on Tamim, IS militants "had no choice except to surrender or fight and they were completely destroyed".
Anbar's police chief, Maj Gen Hadi Irzayij, also confirmed that Tamim, which is separated from the centre of Ramadi by a tributary of the Euphrates, had been retaken.
It is the first significant incursion into Ramadi since the army dropped leaflets over the centre last week warning residents to leave.
"It's a matter of days to control other districts in the heart of the city," an Iraqi army source told the BBC.
"We have retaken key districts like Tamim in the west, Albu Farraj and Albu Ziyab in the north, Tash and Anbar University in the south."
The source said the key challenge now was how to approach the city centre, where roads and buildings were likely to have been rigged with bombs.
"We are using mobile military bridges to cross the Euphrates to the city centre from the north and the west as the main bridges have been destroyed in the intense battles with the terrorists," the source added.
The US military last month estimated there were between 600 and 1,000 IS militants in Ramadi.
On Monday, the US-led coalition against IS said it had carried out six air strikes targeting the group's units, weaponry and positions in the Ramadi area.
Iraqi government forces have recaptured a key district in the city of Ramadi, which is controlled by the jihadist group Islamic State, officials say.