occasionally there would be one or two other vessels in port whose business tended to be shrouded in mystery , although the skull and cross-bone flags flying atop their masts tended to give the game away .