In other words, the group of all invertible functions on the real line acts on the subset consisting of solutions to Babbage's functional equation by conjugation.
In this context, "invertible functions" and "solutions to Babbage's functional equation" are being used in a mathematical sense. The phrase "by conjugation" refers to a specific operation within the set of invertible functions.