Van Buren's rather dandyish style and refined tastes made him an easy target for Whig attacks.
The passage suggests that Van Buren's outward appearance and preferences were perceived as a weakness by his political opponents, the Whigs.