the longer he remained unattached , the more of a challenge he had become , to married and unmarried women alike , until it reached the point that he could not walk into a ballroom without creating a veritable frenzy amongst the female population .