Rosberg ended the session 0.357 seconds quicker than Hamilton, as both explored the limits with off-track moments.
Hamilton in particular had a scrappy session, including a spin, but the virtual safety car hampered his best lap.
Ferrari drivers Sebastian Vettel and Kimi Raikkonen were next, followed by Red Bull's Daniel Ricciardo.
The Australian's team-mate Max Verstappen ended his session in the gravel trap after running wide at Turn Four and breaking his front suspension on the controversial new yellow kerbs introduced to prevent drivers abusing track limits.
Verstappen told his team over the radio that he believed the yellow kerbs were "really dangerous".
The incident brought out the virtual safety car, just as Hamilton was on what looked like being his fastest lap - and possibly the best of all.
Several other drivers hit trouble on the short and tricky low-grip track, including Haas' Romain Grosjean, who lost control avoiding Hamilton's spin.
The session was run in bright sunshine on the picturesque circuit nestled in the Styrian mountains, but rain is predicted for later in the day and may affect the second session, due to start at 13:00 BST.
Following resurfacing since last year's race, the cars are already going faster than in 2015 - Rosberg stopping the clock at one minute 7.373 seconds, 1.1secs quicker than Hamilton's pole position time last year.
Ferrari had been due to run the latest prototype of the 'halo' head protection system at the start of the session but abandoned the plan when Vettel's car hit problems.
The German has had to have a new gearbox fitted after debris was found inside his original one and will receive a five-place grid penalty as a result.
Toro Rosso's Carlos Sainz was sixth fastest, from Williams' Felipe Massa and Verstappen.
McLaren's Jenson Button was 12th, two places ahead of team-mate Fernando Alonso, who was doing comparison tests on a new rear wing and floor.
Austrian Grand Prix practice results
Austrian Grand Prix coverage details
Nico Rosberg headed team-mate Lewis Hamilton as Mercedes dominated first practice at the Austrian Grand Prix.