The line links Inverness to stations in Sutherland and Caithness.
David Spaven, who writes books on the histories of railway networks, said there had been a recent "downgrading" in the quality of services.
ScotRail Alliance, which operates Far North Line services, said it had no plans to close the line and had recently invested in the route.
Mr Spaven has raised his concerns in a new book, Highland Survivor: The story of the Far North Line.
He told BBC Radio Scotland: "On the Far North Line there was the best part of 10 years of rejuvenation with new stations opening, new train services in and out of Inverness and passenger numbers rising.
"But in the last couple of years we have seen a downturn in the quality of services, cancelled trains, late running trains and that degree of unreliability inevitably starts putting passengers off.
"I think, not in the short to medium term but in the long term, the worry is that if we have got a railway that is serving a declining number of people and costing the taxpayer a lot of money every year then questions may be asked."
Dr Paul Monaghan, SNP MP for Caithness, Sutherland and Ross and an honorary president of the Friends of the Far North Line, said the line had recently been downgraded to a tourist line.
He said: "I am not sure what impact that will have, but certainly gives the impression more focus will be on leisure than business and freight, which are vitally important as a connection to the towns and villages of my constituency."
A ScotRail Alliance spokesperson said: "We have no plans to close the Far North Line and have recently invested Â£6m in improving infrastructure along the route.
"We are working hard to deliver improved performance on the line for our customers and are committed to providing as punctual and reliable a service as we can."
A railways expert has raised concerns about the long-term future of Scotland's Far North Line.