Operation Sophia, which the UK supports, appears to have done little to deter migration and its mandate should not be renewed, peers said.
However, they said search and rescue work in the Mediterranean had saved many lives and should continue.
The government said UK ships had led to "fewer children drowning".
Operation Sophia, an initiative undertaken by 25 EU member states, including the UK, was launched in 2015 in the wake of disasters in which hundreds of migrants drowned attempting to reach Europe.
Its aim was to help disrupt organised criminals involved in human smuggling and trafficking networks in the southern central Mediterranean.
But in 2016, detections of migrants in the central Mediterranean were at their highest point yet, the EU External Affairs sub-committee found.
Some 181,436 people arrived in Europe by this route - an increase of 18% on 2015 when the figure was 153,842.
An unintended consequence of the destruction of smugglers' boats has been that they are now sending migrants to sea in unseaworthy vessels, resulting in more deaths at sea, the committee said.
Baroness Verma, chairman of the committee, said a naval mission was the wrong tool for tackling people smuggling because it begins onshore.
And she said that while Operation Sophia had failed in its main objective, it had been a "humanitarian success".
She added: "Future UK and EU action should focus on tackling people smuggling in source and transit countries, and supporting sustainable economic development and good governance in these countries.
"Italy has found itself on the front line of a mass movement of people into Europe, and deserves credit for its efforts to respond."
Other findings from the report included:
The operation's figures show 110 suspected smugglers and traffickers have been apprehended and 463 boats prevented from being re-used since it started.
A UK government spokesman said: "Operation Sophia, and the UK's contribution to it, is saving lives and helping to disrupt the activities of smugglers who continue to exploit migrants trying to reach Europe.
"UK ships mean fewer children drowning and dangerous smuggling boats destroyed before they can be reused.
"The operation is part of the UK government's wider approach to tackling irregular migration at source."
An EU naval mission has failed to achieve its main objective of disrupting people smuggling, a Lords committee report has concluded.