One way to justify this convention of retaining only the linear map aspect of the derivative is to appeal to the (very simple) Lie group structure of  under addition. The tangent bundle of any Lie group can be trivialized via left (or right) multiplication.
One explanation for maintaining only the linear map part of the derivative is to use the simple Lie group characteristics of  under addition. The tangent bundle of any Lie group can be made trivial by applying left (or right) multiplication.