The Swiss, 34, who pulled out of last week's Madrid Open with a back injury, is seeking a first Rome title, having lost last year's final.
Defending champion Djokovic, 28, also reached the third round with a 7-5 7-5 win over France's Stephane Robert.
"I was expecting to lose in straight sets," said Federer, who decided to play only after completing the warm-up.
World number two Federer, who received a bye in the first round, said he has no chance of winning the tournament - "I'm so far off that it won't happen" - and is not sure he will be fit to face Dominic Thiem in the next round.
"I don't know how I'm going to feel tomorrow," he said. "I hope so, but I'm literally going practice after practice."
Serb Djokovic, the world number one and top seed, was given a stiff test by qualifier Robert, coming back from a break down in the second set.
Spanish fifth seed Rafael Nadal, 29, beat Germany's Philipp Kohlschreiber to set up a third-round meeting with Australian Nick Kyrgios.
Angelique Kerber, Victoria Azarenka and Simona Halep were knocked out in the second round at the Foro Italico complex.
German second seed Kerber, the Australian Open champion, was beaten 6-1 5-7 7-5 by Canada's Eugenie Bouchard, while Belarusian fourth seed Azarenka lost 6-3 6-2 to Romania's Irina-Camelia Begu.
Azarenka, who also pulled out of the Madrid Open with a back injury, said: "It's not good right now. We'll see what happens. I don't know what to tell you.
"I thought I was OK and was ready to play and came back, so I'm just disappointed."
Romanian sixth seed Halep, the Madrid Open champion, was beaten 6-3 4-6 6-3 by Australia's Daria Gavrilova.
Coming soon: We're launching a new BBC Sport newsletter ahead of the Euros and Olympics, bringing all the best stories, features and video right to your inbox. You can sign up here.
Roger Federer returned to action with a 6-3 7-5 win over promising German Alexander Zverev in the Italian Open.