those who criticised him at all , usually did so through jealousy , but pointed to his sheltered life and lack of experience in the real world outside the confines of whitehall , either in industry , commerce or the military .