A Wales Audit Office report said a lack of transparency had eroded trust between staff and management.
But it also noted it had overcame a number of crises, including a fire and a high-profile employment tribunal.
President Rhodri Glyn Thomas said it was addressing the concerns raised.
Auditors found management at the library in Aberystwyth was improving, but a lack of transparency around board meeting discussions "reflects and has contributed to an erosion of trust between staff and the library's leadership".
The report said the library's pension fund was generous compared to other public sector organisations, but said a long pay freeze until December 2015 meant any changes to pensions benefits "would have further undermined the already fragile morale among staff".
Report author Huw Lloyd Jones also noted the library's income from the Welsh Government had been cut by 17% in real terms in the past few years at a time when it was also under pressure to increase its income.
He said it was using its reserves to balance the books, but this was "unsustainable".
Mr Jones said the library was maintaining the same levels of service, despite cutting 20% of its staff and while he warned "that can't go on forever", he said the library was aware it might have to change the way it operates.
He concluded: "We think the library is broadly on the right path.
"The challenge for them is to integrate the planning of staffing levels, and of the fantastic building they have in Aberystwyth, with the financial picture so everything is pulling in the same direction."
Library president Rhodri Glyn Thomas said measures had been put in place to address some of the concerns raised "so we are working exactly in the way the report is asking us".
He said its financial challenges were down to the way it received funding from the Welsh Government annually and it would be easier to plan if it instead got funding over three years.
"But we are addressing that, we are in the same position as any other institution in the public sector, we are accepting that challenge and we are facing it positively for the future," he added.
The Welsh Government, which provides the majority of the library's funding, has been asked to comment.
The National Library of Wales must overcome declining income and poor staff morale if it is to have a sustainable future, auditors have found.