Alan Kurdi's father Abdullah said that shortly after their boat left Turkey for the Greek island of Kos it was hit by waves and the captain swam off.
"I tried to catch my children and wife but there was no hope. One by one they died," Mr Kurdi said.
The family were among thousands fleeing Syria and seeking a new life abroad.
Images of Alan's body being recovered from a beach near Bodrum caused an outpouring of sympathy for the plight of those fleeing Syria's civil war and criticism of European governments for not doing enough to answer the crisis.
Thousands of migrants have died this year trying to reach Europe by sea.
"I tried to steer the boat but another high wave pushed the boat over. That is when it happened," Mr Kurdi said.
"My children were the most beautiful children in the world. Is there anybody in the world for whom their child is not the most precious thing?" Mr Kurdi said.
"My kids were amazing. They woke me every day to play with me. What is more beautiful than this? Everything is gone.
"I would love to sit next to the grave of my family now and relieve the pain I feel," he said.
The story of Alan Kurdi's family
Alan, his five-year-old brother Ghalib and mother Rehanna were among 12 Syrians who died after two boats capsized soon after setting off for Kos.
Turkish police have arrested four suspected traffickers over the deaths. The four men are all Syrian nationals aged between 30 and 41, according to the Turkish Dogan news agency.
Alan and the rest of the group his family was travelling with are believed to have been Syrians from the besieged town of Kobane who had fled to Turkey last year to escape advancing militants from the Islamic State (IS) group.
A Turkish hospital official told AFP news agency that the bodies of the family would be flown to Istanbul and then to Suruc on the Turkish border before reaching Kobane.
Abdullah's sister in Vancouver, Teema Kurdi, said she had wanted to bring him and his family to Canada but had first tried to sponsor another brother - an application that had failed.
Canada's department of citizenship and immigration confirmed that there was no record of an application for Abdullah Kurdi and his family.
The father of a three-year-old Syrian boy found drowned on a beach in Turkey has told the BBC of the harrowing circumstances of his death.