Bannister was born on November 2, 1828, in St. Andrews, New Brunswick, near the St. Croix River, to possibly Barbadian parents. His father Edward Bannister died in 1832, so Edward and his younger brother William were raised by their mother, Hannah Alexander Bannister. Early on, Bannister was apprenticed to a cobbler, but his drawing skill was already noted among his friends and family. Bannister credited his mother with igniting his early interest in art. She died in 1844, after which Bannister and his brother lived on the farm of the wealthy lawyer and merchant Harris Hatch. There, he practiced drawing by studying and reproducing two Hatch family portraits, as well as studying the family library and Bible.