E.ON had been working with Pelamis Wave Energy, based in Edinburgh, to test one of the company's wave energy converters at the European Marine Energy Centre.
E.ON has blamed the decision on delays in the development of wave energy technology.
It said it will now focus its efforts on wind, biomass and solar energy instead.
A spokeswoman told BBC Scotland the company still believed marine energy could provide commercial opportunities in the future.
She said: "Delays in wave technology progress and a focus in E.ON on other more mature renewable technologies have been part of the decision to reduce our level of effort in the marine area.
"Having successfully built up a leading position in the marine energy area, we will continue to monitor technologies and work towards being prepared when marine energy is more mature and we are in the right position to move forward."
Pelamis Wave Power said the decision by E.ON would not affect its plans to begin generating electricity on a commercial scale.
The company will continue to work with Scottish Power Renewables on the development of its P2 wave energy device at the European Marine Energy Centre, known as EMEC, in Orkney.
The company said: "E.ON's decision at this stage does not in any way change our development trajectory.
"We will be continuing operation of the first P2 machine at EMEC over the coming months, alongside the Scottish Power Renewables-owned P2 unit.
"Together these have now delivered over 8,000 hours of grid connected operation in a broad range of conditions, including some 160 MWh of electricity generation.
"This experience in conjunction with the major development programme we are undertaking with the Energy Technologies Institute gives us a unique and solid platform from which to deliver the first commercial Pelamis machines over the next couple of years."
The German power giant E.ON is pulling out of a marine energy research project in Orkney.