you could swim across to santa catalina , but then , that was even smaller , so mainly we spent the days fishing and finding frond oysters in small pools , and occasionally having a kind of snarling confrontation with groups of passing natives , ragged , wandering colonists or turtle fishermen .