this was not the ominous activity that it might appear , since the keys had been lost long ago and latecomers usually threw gravel at the windows of the houses built on top of the wall until they found a friend to lift the bar .