One way to justify this convention of retaining only the linear map aspect of the derivative is to appeal to the (very simple) Lie group structure of  under addition. The tangent bundle of any Lie group can be trivialized via left (or right) multiplication.
One rationale for preserving only the linear map component of the derivative is to leverage (extremely basic) Lie group properties of  under addition. The tangent bundle of any Lie group can be made trivial through left (or right) multiplication.