the morton senior residence was more of a mansion than a house and was set high up on the downs above the avon gorge , the buildings dating back to the period when the slave trade had brought great wealth to bristol 's merchants .