Media playback is not supported on this device
France will not be competing in the Four Nations in October, having lost to Scotland at the 2014 European Cup.
A weakened French side were also beaten 84-4 by England in October 2015, ahead of next year's World Cup in Papua New Guinea, New Zealand and Australia .
"We've got loads of young players but they don't really know where to go," Larroyer told BBC Radio Humberside.
"We are all divided. I think we need to work with each other to be strong, to work together and try to improve rugby league in France because I really think we've got potential."
"We need a strong French championship as well to try to give some new players to Catalans Dragons or give a chance to go to [League One side] Toulouse as well or to go and have a crack in Super League."
Hull KR forward Kevin Larroyer says French rugby league needs greater unity if it is ever to reach its potential.