It is not a silly thing at all to enjoy a stone in a tray.
The statement suggests that appreciating the beauty or simplicity of a stone within a tray is not something to be ridiculed, but rather something to be embraced.