Ryan Wallace struck for the hosts on 55 minutes, capitalising on a mistake by Jason Kerr.
The Fifers won a late penalty when Jamie Insall was impeded by Kyle Turnbull.
But Scott Robinson's effort was saved by Ross Stewart, leaving East Fife fourth in the table.
They remain level on points with Airdrieonians, but with a superior goal difference.
The hosts move to within a point of Queen's Park, who are four points off the top four, with second, third and fourth the promotion play-off spots.
Albion Rovers and Queen's Park are scheduled to play their game in hand on 4 April.
Albion Rovers dented opponents East Fife's promotion hopes in Scottish League One by winning their rearranged match at Cliftonhill.