The awards are currently run by Literature Wales, on behalf of the Welsh Arts Council.
Literature Wales and the Arts Council will retain some functions such as running the Ty Newydd writing centre.
Mr Skates said the changes were not a reflection on this wider work.
"I realise some people may be surprised at the extent of these changes," he said.
"I should stress these are in response to specific needs in a specific area of activity.
"They are not a reflection on the wider work of the Arts Council, and my appreciation of the great work that Literature Wales is delivering in some areas."
The Book of the Year award is given annually to works of poetry, fiction, and creative non-fiction in Welsh and English.
The change is part of a Welsh Government-commissioned publishing and literature review, chaired by University of Wales, Trinity Saint David Vice-Chancellor Prof Medwin Hughes.
Responsibility for the Book of the Year awards will be transferred to the Welsh Books Council as part of a shake-up of culture funding, Economy Minister Ken Skates has announced.