The world champion was 0.331 seconds quicker than Rosberg, despite using a slower set of tyres than the German.
Rosberg was on the ultra-soft tyre with Hamilton and Vettel on the super-soft.
Red Bull's Max Verstappen was an impressive fourth on the soft tyre - the slowest available - as Felipe Massa crashed his Williams heavily.
The Brazilian lost control on the entry to Turn One, spinning on to the grass and smashing into the barriers, emblazoned with the name of F1's newest corporate sponsor, a popular beer brand.
Massa said he had lost the rear as soon as he closed the DRS overtaking aid, which reduces rear downforce to increase straight-line speed.
He was not the only driver to experience problems on the demanding Circuit Gilles Villeneuve, but he was the only one to crash.
Vettel brushed the wall exiting one of the chicanes, while Hamilton, Rosberg, Verstappen and Alonso all cut the final chicane when they misjudged their braking points.
There were also a couple of misunderstandings between drivers out on track, with Verstappen and McLaren's Fernando Alonso respectively incurring the wrath of Williams' Valtteri Bottas and Hamilton as they slowed for the final chicane while their rivals approached on a flying lap.
Both Bottas and Hamilton complained to their teams over the radio about the potential risk of the incidents.
Verstappen received a reprimand by the stewards for impeding the Williams after the session finished.
The only surprise in a somewhat low-key session - held in cool but dry conditions with drivers doing relatively few flying laps - was that Red Bull's Daniel Ricciardo did not set a representative quick time.
The Australian, who missed out on certain victories in the last two races in Spain and Monaco after team strategy and pit errors, was quick early on but did not improve his time as his team-mate did and ended up only 11th fastest.
At the front, Vettel was only 0.157secs slower than Rosberg and only 0.31secs quicker than Verstappen.
Ferrari's Kimi Raikkonen was fifth, ahead of Bottas, Force India's Nico Hulkenberg, Toro Rosso's Carlos Sainz, Force India's Sergio Perez and Alonso.
Canada Grand Prix first practice results
Canada Grand Prix coverage details
Lewis Hamilton headed Mercedes team-mate Nico Rosberg and Ferrari's Sebastian Vettel in first practice at the Canadian Grand Prix.