according to family legend my grandfather , alton , had gotten into a fight with a man named charlie ferguson , his foreman , a fellow nearly twice granddaddys size and rumored to be the most hated man in waller county .