no matter how busy their lives became , no matter how many people vied for their attention , hart would make certain that the duke and duchess often retired from the public eye to lie together in joyful silence .