Media playback is not supported on this device
Trailing 2-1 from the first leg in Iceland, the Irish Cup holders suffered a setback when Kennie Chopart went clear to shoot a fifth-minute opener.
The impressive Chopart then fired in from the edge of the box for 2-0.
A Holmbert Fridjonsson penalty made it 3-0 and further goals by Morten Beck Andersen, Oskar Hauksson and Denis Fazlagic made it 8-1 on aggregate.
It was a desperately disappointing evening for Gary Hamilton's men as they felt the first qualifying round game was very much alive after their narrow defeat in the away leg.
Media playback is not supported on this device
Glenavon could have taken an early lead at Mourneview Park but Eoin Bradley shot side from a great position.
They were punished when Chopart's well-timed run put him clear of a square defence and he slotted in the opener.
Chopart then made it 2-0 on 29 minutes running forward to shoot in from the edge of the penalty area.
Striker Fridjonsson netted a penalty early in the second half after he had been fouled by defender Simon Kelly.
Glenavon visibly wilted and goals by Andersen, Hauksson and Fazlagic piled on the agony for the Irish League side.
Glenavon's hopes of progress in the Europa League ended with a heavy 6-0 home defeat by KR Reykjavik.