although allie had n't become fully aware of his existence until the scandal broke , there were plenty of folks who remembered him from when he 'd first come to town , right after the widowed reverend married irene and moved her little family from neighboring booneville to the farm .