he agreed to their invitation , but when he tried to tell his grandfather where he would be , the sailors took him prisoner and kept him in the ships hold until they were out to sea headed for the caribbean .