i rarely left her side before she was allowed to leave and join me in the beautiful home we had been given near the palatine , interestingly on the spot where the colosseum should be standing in about forty years or so .