Theologian Dale Allison views the concept of "words falling short" in these lyrics as central to "That Is All" and a recurring theme in Harrison's songwriting. Allison also notes the importance of silence in this and other Harrison song lyrics, where it becomes "not an enemy to be shunned but ... a friend to be embraced, for silence can help conduct us to the Divine". On release, part of the middle eight's lyrics, together with those for the final verse, were identified by music critic Stephen Holden as "a sort of Hindu "In Paradisium"". While he considers "That Is All" to be a secular, "adult-oriented" love song, Inglis writes of Harrison's message in the middle eight: "language is insufficient to express the depth of his emotion; 'silence' can be more effective, more intimate, more "loving"."