doing only that you know all the time he was growing up then it's hard to think of it
It is hard to think about him when he was growing up.
Let us dwell about his upbringing, since that is all you know.