Transformer as Linear Expansion of Learngene
We propose expanding the shared Transformer module to produce and initialize Transformers with diverse depths, enabling adaptation to dynamic resource constraints. Drawing an analogy to genetic expansibility, we term such module as learngene. To identify the expansion mechanism, we delve into the relationship between the layer position and its corresponding weight value, and find that linear function appropriately approximates this relationship. Building on this insight, we present Transformer as Linear Expansion of learnGene (TLEG), a novel approach for flexibly producing and initializing Transformers of diverse depths. Specifically, to learn learngene, we firstly construct an auxiliary Transformer linearly expanded from learngene, after which we train it through employing soft distillation. Subsequently, we can produce and initialize Transformers of varying depths via linearly expanding the well-trained learngene, thereby supporting diverse downstream scenarios. Extensive experiments on ImageNet-1K classification demonstrate that TLEG achieves comparable or better performance compared to many individual models trained from scratch, while reducing around 2$\times$ training cost. When transferring one model to several downstream classification datasets, TLEG surpasses existing initialization methods by a large margin (e.g., +6.87% on iNat 2019 and +7.66% on CIFAR-100). Under the situation where we need to produce models of different scales adapting for different resource constraints, TLEG achieves comparable results while reducing around 19$\times$ parameters stored to initialize these models and around 5$\times$ training costs, in contrast to the pre-training and fine-tuning approach.