when sue ellen brannigan had arrived there five years earlier , she had been nervous about living on the grounds of the penitentiary , and even more apprehensive about having a house full of servants , who were all convicted criminals .