wandering up from the coast with only their rifles and the clothes on their backs , he assumed they were pirates or spies , although both claimed to have been captured by lafitte while traveling on spanish ships .