The facility at Riverside, Haverfordwest, will also house a coffee shop, tourist information office and a gallery.
It is part of a Â£3.4m scheme to regenerate the Pembrokeshire town.
It has been funded by the Welsh Government, the county council and the Wolfson Foundation charity and is due to open in spring next year.
The gallery will exhibit works from the National Library of Wales in Aberystwyth.
Its president Rhodri Glyn-Thomas said: "This excellent project will provide the National Library with a state of the art gallery in Pembrokeshire where it can exhibit its unique works of arts and extend a part of its services for the benefit of the people of south west Wales."
Work has started to transform a former Pembrokeshire indoor market into a library.