# On Giant’s Shoulders:
Effortless Weak to Strong by Dynamic Logits Fusion

Chenghao Fan1,2, Zhenyi Lu1,211footnotemark: 1, Wei Wei1,2, Jie Tian1,2,
Xiaoye Qu1,
Dangyang Chen3, Yu Cheng4
1School of Computer Science & Technology, Huazhong University of Science and
Technology
2Joint Laboratory of HUST and Pingan Property & Casualty Research (HPL)
3Ping An Property & Casualty Insurance Company of China, Ltd.
4The Chinese University of Hong Kong
{facicofan,luzhenyi529}<EMAIL_ADDRESS>
chendangyang273<EMAIL_ADDRESS>
Equal contribution. Corresponding authors.

###### Abstract

Efficient fine-tuning of large language models for task-specific applications
is imperative, yet the vast number of parameters in these models makes their
training increasingly challenging. Despite numerous proposals for effective
methods, a substantial memory overhead remains for gradient computations
during updates. Can we fine-tune a series of task-specific small models and
transfer their knowledge directly to a much larger model without additional
training? In this paper, we explore weak-to-strong specialization using logit
arithmetic, facilitating a direct answer to this question. Existing weak-to-
strong methods often employ a static knowledge transfer ratio and a single
small model for transferring complex knowledge, which leads to suboptimal
performance. To surmount these limitations, we propose a dynamic logit fusion
approach that works with a series of task-specific small models, each
specialized in a different task. This method adaptively allocates weights
among these models at each decoding step, learning the weights through
Kullback-Leibler divergence constrained optimization problems. We conduct
extensive experiments across various benchmarks in both single-task and multi-
task settings, achieving leading results. By transferring expertise from the
7B model to the 13B model, our method closes the performance gap by 96.4% in
single-task scenarios and by 86.3% in multi-task scenarios compared to full
fine-tuning of the 13B model. Notably, we achieve surpassing performance on
unseen tasks. Moreover, we further demonstrate that our method can
effortlessly integrate in-context learning for single tasks and task
arithmetic for multi-task scenarios. 111Our implementation is available in
https://github.com/Facico/Dynamic-Logit-Fusion

> “If I have seen further than others, it is by standing on the shoulders of
> giants.”  — Isaac Newton in 1675

##  1 Introduction

In recent years, Large Language Models (LLMs) have shown impressive
performance in a wide range of tasks [5, 47] including code generation [10,
41], mathematical reasoning [34, 2], tool-use abilities [46, 42], etc.
However, training such LLMs requires substantial computational resources,
often involving thousands of GPUs and processing trillions of tokens [29, 39],
making the adaptation of the base model for new knowledge inefficient. To
address these challenges, parameter-efficient tuning methods [15, 22, 11] have
emerged, aiming to achieve comparable performance to full fine-tuning while
reducing GPU memory requirements. However, challenges persist in tuning and
deploying large-scale models on common hardware, as they still involve
computation-intensive processes like gradient calculation and back-
propagation. Furthermore, these methods may not be feasible when training data
are private.

This inspires us to ask: Can we fine-tune only small models and then transfer
their knowledge to a much larger model without requiring additional gradient
updates? If we could fuse the strong capabilities of a scaled LLM with the
specialized knowledge acquired by a small model during fine-tuning, it would
yield the practical benefit of approximating the results achieved by fine-
tuning a large model, but without the associated computational costs. However,
it is non-trivial due to the differences in representation width and layer
numbers between the small and large models.

Figure 1: Comparison between our work and previous work. Previous methods only
use pre-tuned parameters $\alpha$ to transfer knowledge from a single expert.
In contrast, our method dynamically adjusts the proportion of knowledge
transferred from multiple experts at each decoding step during inference.

Recently, Mitchell et al. [38] and Liu et al. [32] attempt to address this
challenge by transferring knowledge from a Small Language Model (SLM) to its
larger counterpart through simple logit arithmetic operations during decoding.
For instance, using models from the Llama-2 family, they can transfer the
fine-tuned knowledge from a 7B-scale model to a 13B-scale model by performing
log probability algebra: Llama-2-base 13B + (Llama-2-chat 7B - Llama-2-base
7B), where the first term represents the base log probabilities and the term
in parentheses denotes the behavioral delta [38]. This behavioral delta can be
weighted to adjust the balance between the pretrained knowledge and the
transferred fine-tuned knowledge.

Despite showing promise, logit arithmetic still exhibits a noticeable
performance gap compared to directly fine-tuning large models, primarily due
to two reasons: Firstly, they statically prespecifies the weight of behavioral
delta at each decoding step identically. However, the importance of fine-tuned
knowledge varies significantly across different tasks, inputs, and even
different decoding steps. For instance, in a domain-specific question-
answering process, we need more knowledge from fine-tuned small language
models. Conversely, when decoding factual topics, we may need more knowledge
from pretrained general models. Secondly, for unseen tasks, the lack of pre-
adjusted weights for the behavioral delta prevents logit arithmetic from
executing effectively, making it challenging to transfer tuned knowledge,
especially for complex tasks. As shown in Figure 1, when answering “Peter is
76 years old this year. How old will he be in 500 years?”, using only the math
expert does not ensure the result aligns with factual accuracy. Additionally,
these techniques often assume that experts are trained and tested on the same
data distribution, ignoring the heterogeneity of data that may be encountered
at test time, rendering any single expert insufficient.

In addressing these challenges, we reconsider the practice of logit arithmetic
within a new framework. Specifically, we work with a set of finely tuned SLM
experts, each specializing in different tasks. At each decoding step, we
dynamically allocate weights among these task-specific SLMs. However, it is
non-trivial to effectively determine suitable weights for diverse tasks and
SLM experts, as tranditional approaches such as grid search [28] or
combinatorial search [33] suffer from costly search processes. To practically
automate weight allocation, we reframe the problem of weight search as a
constrained distribution optimization problem. We tackle the fusion of multi-
task knowledge by treating it as a centroid problem in Euclidean space using
Kullback-Leibler divergence [23], which offers interpretability and explicit
guidance.

We conduct thorough experiments on the LLaMA series to evaluate the
effectiveness of our approach, applying our adaptive logit arithmetic method
to task-specific fine-tuning in math, question-answering, summarization, and
multi-domain tasks. We also analyze the performance of multi-model fusion
across seen and unseen tasks. Additionally, we discussed the feasibility of
combining our method with in-context learning for single tasks and with task
arithmetic for multi-task scenarios.

In our work, we make several key contributions. Firstly, we reassess existing
logit arithmetic methods, highlighting the significant impact of fusion
weights and the limitations imposed by a single small model on test
performance. Secondly, we introduce a novel approach that autonomously learns
fusion weights through constraint optimization, approximating the compute-
intensive results of fine-tuning a large base model. Lastly, we conduct
comprehensive experiments to validate our method, demonstrating substantial
improvements in performance, generalization capabilities, and robustness.

##  2 Related Work

###  2.1 Efficient Specialization

Specializing a pretrained model by fine-tuning for downstream tasks has become
a primary paradigm [9, 40, 7]. However, with the increasing scale of models,
full fine-tuning has become impractical due to the large number of parameters,
requiring costly training resources [5, 47].

To tackle this challenge, researchers have developed parameter-efficient
tuning methods [27]. These methods aim to achieve performance similar to full
fine-tuning while reducing the GPU memory required. They typically involve
freezing the original weights and adding task-specific trainable modules [14,
21, 25], low-rank matrices [15, 22, 11], or bias [3]. Despite their benefits,
these approaches still demand significant memory for updating gradients and
might not be suitable for scenarios where training data are private or
inaccessible. Another direction is model merging [49, 16, 51, 35], which
trains task-specific models on different domains, and then combines them into
a single model at deployment with weight average [50], neuron permutation [1],
interpolation [36, 19] or task vectors [16, 52]. However, they typically
suffer from parameter interference between different tasks and static optimal
solutions may struggle with multiple task domains.

There is a growing research interest in specialized large models by eliciting
existing knowledge, such as utilizing curated prompts via in-context learning
[30] or employing small models to offer weak-to-strong supervision [6, 38,
32], generating weak labels or alignment signals. These methods are highly
practical, requiring only the generated output (or logits). Our approach
shares a similar weak-to-strong intuition and proposes adaptive strategies to
achieve a better balance between leveraging the pretrained general knowledge
and acquiring task-specific knowledge. Furthermore, our method can be extended
to multi-task scenarios.

###  2.2 Weak-to-Strong Generation

Different from knowledge distillation [13, 45, 4], where a more capable
teacher guides a student model, we explore using weaker models to teach
stronger ones. Burns et al. [6] empirically demonstrate that weak supervisors
can reliably elicit knowledge from much stronger models (e.g., supervising
GPT-4 with a GPT-2). Such approaches represent a promising direction, as it is
more practical and affordable to specialize in a much smaller model.
Contrastive Decoding [26] enhances outputs by leveraging differences between
large and small LMs, as patterns like repetition and incoherence are often
more pronounced in smaller ones. Speculative sampling [24, 44] speeds up
inference using a lower-parameter version of the LLM as a draft model,
exploiting the fact that draft models can approximate easier subtasks well.
SpecInfer [37] goes further by employing a set of small models to generate
drafts in parallel. Jin et al. [18] train a projector to map the parameter of
the weak expert to a larger version, but the projector needs to be trained and
suffers from poor generation. Ji et al. [17] focuses on utilizing small models
to rephrase unaligned answers from LLMs into more aligned ones. Mitchell et
al. [38] and Liu et al. [32] leverage logits from small, fine-tuned models to
inject specific knowledge into the pretrained LLM with the same vocabulary.
Our approach differs from methods that require pretrained small model adapters
[44] or pre-adjusted parameters [38, 32]. Instead, we dynamically transfer the
capabilities of small models to the large model at each decoding step, without
needing access to the large model’s parameters or training data.

##  3 Methodology

###  3.1 Problem Background

#### Autoregressive Language Models

Modern autoregressive transformers generate continuations for input prompts
token by token. Given a prompt $x_{1:k-1}$ (denoted as $x_{<k}$), the model
computes the logits for the $k$-th token, represented as $M(x_{k}\mid
x_{<k})\in\mathbb{R}^{|V|}$, where V denotes the size of the vocabulary. A
probability distribution $P(x_{k}\mid x_{<k})$ is then obtained through
softmax normalization: $P(x_{k}\mid x_{<k})=\text{softmax}(M(x_{k}\mid
x_{<k}))$. The next token $x_{k}$ is subsequently sampled from this
distribution, i.e., $x_{k}\sim P(x_{k}\mid x_{<k})$.

#### Distance Between Language Model Outputs Distribution

We can utilize the Kullback-Leibler(KL) divergence to measure the similarity
of two distributions $P$ and $Q$ generated from two language models (with the
same vocabulary), which can be viewed as the distance of the two language
models:

$\displaystyle D_{\mathrm{KL}}(P||Q\mid x_{<k})$ $\displaystyle=\sum_{x\in
V}P(x\mid x_{<k})\log\frac{P(x\mid x_{<k})}{Q(x\mid x_{<k})}$ (1)

If this is implied by the context, we will omit the conditioning on $|x_{<k}$
and simply use $D_{\mathrm{KL}}(P||Q)$.

#### Logit Arithmetic

Suppose we have two pretrained auto-regressive models with homogeneous
architecture and the same vocabulary: a small model with parameter set
$\theta^{S}$ and a large model with parameter set $\theta^{L}$. We aim to
fine-tune the small model to obtain $\theta^{S}_{ft}$ and transfer this fine-
tuning knowledge to the large models. Previous work [38, 32] transferred fine-
tuned knowledge to a large model by designing arithmetic between logits,
resulting in the output distribution $\tilde{P}$ for the large model as
follows:

$\displaystyle\tilde{P}(x_{k}|x_{<k})=\mathrm{softmax}(M^{L}(x_{k}|x_{<k})+\alpha\cdot(M^{S}_{ft}(x_{k}|x_{<k})-M^{S}(x_{k}|x_{<k})))$
(2)

where $M^{L}$, $M^{S}$, and $M^{S}_{ft}$ represent the logits of the large
model, small model, and fine-tuned small model, respectively. Their normalized
distributions are denoted by $P$, $Q$, and $Q_{ft}$, respectively. Here,
$\alpha$ is a pre-adjusted hyperparameter that controls the extent of
knowledge transfer from the small model. Our analysis from Appendix B
demonstrates that logit arithmetic attempts to approximate the shift
$(\frac{Q_{ft}(x_{k}\mid x_{<k})}{Q(x_{k}\mid x_{<k})})^{\alpha}$ between the
fine-tuned distribution and the pretrained distribution by controlling the
parameter $\alpha$ before inference.

###  3.2 Adaptive Knowledge Transfer Optimization

However, using a pre-defined $\alpha$ presents the issue that the resulting
LLM’s distribution $\tilde{P}$ has a fixed trajectory, which may not be
suitable for every step in decoding. So we need a dynamic $\alpha$ in the
decoding steps. Nonetheless, solving this problem is non-trivial, so we
attempt to transform it into an equivalent, optimizable objective. Here, we
use the distance between language model outputs as defined in Equation (1) to
represent the shift between distributions. We assume that, for different model
sizes, at each decoding step, the distance between the fine-tuned model
outputs and pretrained model output is the same:

$\displaystyle D_{\mathrm{KL}}(\tilde{P}||P\mid x_{<k})\approx
D_{\mathrm{KL}}(Q_{ft}||Q\mid x_{<k}),\quad D_{\mathrm{KL}}(P||\tilde{P}\mid
x_{<k})\approx D_{\mathrm{KL}}(Q||Q_{ft}\mid x_{<k})$ (3)

Following the assumption in Equation (3), we optimize the following expression
at each decoding step:

$\operatorname*{arg\,min}_{\tilde{P}}\Biggl{[}\Bigl{(}D_{\mathrm{KL}}(\tilde{P}||P)-D_{\mathrm{KL}}(Q_{ft}||Q)\Bigr{)}^{2}+\Bigl{(}D_{\mathrm{KL}}(P||\tilde{P})-D_{\mathrm{KL}}(Q||Q_{ft})\Bigr{)}^{2}\Biggr{]}$
(4)

Based on this optimization problem, we can adaptive transfer the knowledge of
the corresponding SLM expert to the large model at each decoding step in a
single-task setting. Our method has been proven to be effective in
experiments. To guarantee symmetry, a symmetrical term is incorporated:

###  3.3 Extending to the Fusion of Multiple SLMs

When dealing with complex tasks or new domains, a general LLM may lack the
necessary expertise, and a single SLM might not provide sufficient specialized
knowledge due to the capacity gap. To migrate this challenge, our method can
be extended to fuse multiple SLMs and leverage their knowledge to compensate
for the shortcomings in individual domain-specific knowledge. We fine-tune the
SLM $\bm{\theta^{S}}$ on each domain to obtain multiple task-specific experts
$\\{\bm{\theta^{S}_{t}}\\}_{t=1}^{T}$, making it easier to acquire knowledge
and dynamic fused to the LLM. During decoding, knowledge from these domain-
specific SLMs is transferred simultaneously to the LLM. We modify the Equation
(3) as follows:

$\displaystyle D_{\mathrm{KL}}(\tilde{P}\parallel P)\approx
D_{\mathrm{KL}}\left(\text{Joint}(\\{Q_{1...T}\\})\parallel Q\right),\quad
D_{\mathrm{KL}}(P\parallel\tilde{P})\approx
D_{\mathrm{KL}}\left(Q\parallel\text{Joint}(\\{Q_{1...T}\\})\right)$ (5)

where $Q_{t}$ represents the distribution of $\theta_{t}^{S}$ from the $t$-th
domain, $\text{Joint}(\\{Q_{1...T}\\})$ represents joint distribution of
$Q_{1},Q_{2},...,Q_{T}$. When we impose constraints like Equation (5), it
attempts to align the joint distributions of the logits from the domain-
specific small models. However, the distributions of the logits from the
domain-specific small models are usually not independent of each other, so
their joint distribution cannot be estimated through simple multiplication.

Due to the difficulty in obtaining a joint distribution for multiple expert
models, we decompose the joint distribution constraint problem into a multi-
object marginal distribution optimization. By aligning the distributions of
each domain-specific small model, we can infer an approximate optimal solution
within the extension of Equation (2), as shown by:

$\displaystyle\operatorname*{arg\,min}_{\tilde{P}}\sum_{t=1}^{T}\left[KL(\tilde{P}||P)-KL(Q_{t}||Q)\right)^{2}+\left(KL(P||\tilde{P})-KL(Q||Q_{t})\right]^{2}$
(6)
$\displaystyle\text{where}\quad\tilde{P}=\mathrm{softmax}\left[M^{L}(x_{k}|x_{<k})+\sum_{t=1}^{T}\alpha_{t}\left(M^{S}_{t}(x_{k}|x_{<k})-M^{S}(x_{k}|x_{<k})\right)\right]$

Here we use $M^{S}_{t}$ to represent the logit of $t$-th expert. Our algorithm
is outlined in pseudo-code in Algorithm 1 in Appendix C.

Intuitively, this projects the KL divergences between the logits into
Euclidean space and finds a central point with the minimum distance sum as the
best KL value. This optimal KL value corresponds to the output distribution of
the large model with multi-domain knowledge. In our experiments, we optimize
$\alpha\in\mathbb{R}^{T}$ to obtain the optimal KL value. Generally, $\alpha$
is between 0 and 2. In a multi-task setting, we can accelerate the
optimization process by optimizing the boundaries for only one expert,
restricting only one SLM expert to be applied at the current decoding step. We
will provide a more detailed explanation of this in our experiments.

##  4 Experiments

In this paper, we use the LLaMA2 [48] family of models to test our method,
which contains the same vocabulary, enabling us to employ logits arithmetic
easily. Here, we use TinyLLaMA-1.1B [53] and LLaMA2-7B as our pretrained small
models, and LLaMA2-13B as the large model for transfer. We conduct tests in
single-domain and multi-domain settings to demonstrate that our method can
effectively transfer knowledge to the large model in both scenarios.

Table 1: Performance on single-task scenarios. Bold numbers indicate the best-
performing model transferred from the same size. Underlines indicate whether
the method outperforms the expert model being used. Notably, we are unable to
obtain the LoRA adapter for LLAMA2-chat version. Therefore, we set the LoRA
Tuning for the 13B model on TruthfulQA to match the same values as Full Fine-
Tuning, e.g., 61.93.

Model | GSM8K | TruthfulQA | TriviaQA | CNN/DM | MMLU | Avg.
---|---|---|---|---|---|---
(EM.) | (Acc.) | (EM.) | (Rouge 2.) | (Acc.)
13B
Base Model | 6.90 | 46.13 | 36.44 | 8.94 | 51.25 | 29.93
Full Fine-tuning | 47.23 | 61.93 | 56.36 | 15.50 | 57.94 | 47.79
LoRA Tuning | 41.54 | 61.93 | 61.89 | 17.18 | 60.46 | 48.60
Transfer from 1.1B
Full Fine-tuning | 12.51 | 29.01 | 33.66 | 14.22 | 37.26 | 25.33
Proxy Tuning | 16.91 | 31.48 | 48.74 | 13.23 | 39.88 | 31.74
Ours | 18.27 | 37.05 | 53.81 | 14.48 | 48.32 | 34.86
Transfer from 7B
Full Fine-tuning | 37.07 | 60.02 | 52.10 | 15.21 | 56.23 | 44.13
Proxy Tuning | 37.68 | 61.02 | 52.81 | 14.37 | 56.24 | 44.43
Ours | 39.34 | 61.56 | 57.11 | 15.31 | 57.15 | 46.09

Table 2: Performance on multi-task scenarios. “Base” denotes an untrained
model. “Multi-Task Tuning" refers to models trained using data mixing. Bold
numbers represent the best-performing multi-task models among those using
experts of the same size. The leftmost "Avg." represents the average
performance of Seen Tasks and Unseen Tasks (57 tasks in MMLU), calculated by
averaging the mean performance on Seen Tasks and the performance on MMLU.

Model | Avg. | Seen Task | Unseen Task
---|---|---|---
GSM8K | TruthfulQA | TriviaQA | CNN/DM | Avg. | MMLU
Pre-trained model |  |  |  |  |  |  |
1.1B (Base) | 18.29 | 2.04 | 27.41 | 9.6 | 7.21 | 11.56 | 25.02
7B (Base) | 29.16 | 3.80 | 30.96 | 36.7 | 8.81 | 20.06 | 38.26
13B (Base) | 37.93 | 6.89 | 46.13 | 36.5 | 8.94 | 24.61 | 51.25
Multi task with tuning |  |  |  |  |  |  |
13B (Multi-Task Tuning) | 45.68 | 39.03 | 44.39 | 62.79 | 16.95 | 40.78 | 50.58
Single task with tuning |  |  |  |  |  |  |
1.1B-Expert (GSM8K) | 18.50 | 12.51 | 25.38 | 6.12 | 7.75 | 12.69 | 24.30
1.1B-Expert (TruthfulQA) | 16.15 | 2.81 | 29.01 | 2.83 | 7.23 | 10.47 | 21.82
1.1B-Expert (TriviaQA) | 21.72 | 3.26 | 26.25 | 33.66 | 8.03 | 17.80 | 25.63
1.1B-Expert (CNN/DM) | 12.98 | 2.73 | 26.39 | 1.65 | 14.22 | 11.24 | 14.73
Multi-task transfer from 1.1B |  |  |  |  |  |  |
1.1B (Multi-Task Tuning) | 23.90 | 14.40 | 25.76 | 35.05 | 14.26 | 22.37 | 25.42
Ours | 32.59 | 18.65 | 36.33 | 18.84 | 9.38 | 20.80 | 44.38
Single task with tuning |  |  |  |  |  |  |
7B-Expert (GSM8K) | 34.14 | 37.07 | 36.04 | 20.41 | 11.19 | 26.18 | 42.10
7B-Expert (TruthfulQA) | 32.79 | 8.26 | 56.42 | 0.17 | 11.02 | 18.96 | 46.61
7B-Expert (TriviaQA) | 29.35 | 4.62 | 33.66 | 52.10 | 10.30 | 25.17 | 33.52
7B-Expert (CNN/DM) | 22.29 | 4.39 | 34.57 | 0.19 | 15.21 | 13.59 | 30.98
Multi-task transfer from 7B |  |  |  |  |  |  |
7B (Multi-Task Tuning) | 34.89 | 34.72 | 33.28 | 51.54 | 16.30 | 33.96 | 35.82
Ours | 39.42 | 34.87 | 42.25 | 22.48 | 10.52 | 27.53 | 51.31

###  4.1 Datasets

Following the setting in Liu et al. [32], we evaluate on the following
datasets: mathmetical reasoning (GSM8K [8]); factual accuracy (TruthfulQA
[31]); realistic knowledge (TriviaQA [20]); multi-domain general knowledge
(MMLU benchmark [12]); summarization (CNN-DailyMail (CNN/DM) [43]). All
datasets are tested using a 0-shot setting. Detailed information is provided
in Appendix D.

###  4.2 Implementation Details

For all tasks except TruthfulQA, we construct prompts based on the task type
and employ supervised instruction tuning to train each task expert. Following
the previous work [32], we use “Llama-2-7b-chat-hf” as the 7B expert for
TurhfulQA, and TinyLLaMA-chat-version as the 1.1B expert for TurhfulQA. For
full fine-tuning, we set the batch size to 128, learning rate to 2e-5,
optimizer to Adam. For LoRA tuning, we set the rank to 64, learning rate to
1e-4, optimizer to Adam. We train for 3 epochs. For multi-task tuning, we
perform full fine-tuning using a mixed-seed training dataset.

During inference, we use greed decoding and set batch size to 256, top_p to
1.0 and temperature to 0.05. To accelerate inference, we use
VLLM222https://github.com/vllm-project/vllm , synchronizing the signals of
logits during logit arithmetic to achieve efficient inference. All experiments
are performed on H100 GPUs.

###  4.3 Baselines

To demonstrate the effectiveness of our method in efficiently transferring
knowledge from a small model to a large model, we compare it against a small
model fine-tuned in a single domain and a large model without fine-tuning. We
also report the performance of a large model fine-tuned on a single domain or
fine-tuned using LoRA to demonstrate the feasibility of our method as a
replacement for fine-tuning approaches. Additionally, we compare our method
with Proxy Tuning (setting $\alpha=1.0$ as in the original work) to highlight
the superiority of our method in the transfer process. We use the LLaMA2-chat
model as the expert for TruthfulQA (therefore, there is no corresponding LoRA
model). For other experts, we use models fine-tuned on the respective training
sets. In multi-task scenarios, we follow the common training practice of
mixing the training data, with the results serving as our multi-task baseline.
More details can be found in the Appendix D.

Figure 2: Compare the pre-defined $\alpha$ with the dynamic $\alpha$ for
different tasks.

Figure 3: The variation of $\alpha$ in knowledge transfer for the GSM8K
expert.

###  4.4 Performance on Single-Task Scenarios

As shown in Table 1, our method improves upon the original experts when
transferring knowledge from the 1.1B and 7B models to the 13B model. Across
all tasks, our method transfers knowledge from 1.1B and 7B experts to the 13B
model, achieving performance improvements of 37.6% and 4.4% over the original
experts, respectively. This demonstrates that our method can leverage the
existing knowledge in larger models to enhance the knowledge learned by the
expert small models. Notably, we observe significant improvements on GSM8K and
TriviaQA compared to the experts alone, indicating that our method effectively
transfers expert knowledge while leveraging the inherent capabilities of the
large model, such as stronger reasoning abilities and a richer knowledge base.

Our method, which transfers knowledge from 1.1B and 7B experts, can close the
performance gap by 72.9% and 96.4%, respectively, compared to 13B Full Fine-
Tuning, achieving improvements of 6.5% and 3.8% over Proxy Tuning. Compared to
13B Full LoRA Tuning, our method can close the gap by 71.7% and 90.7%,
respectively, when transferring knowledge from 1.1B and 7B experts.
Additionally, it is worth noting that our method outperforms the 13B Full
Fine-Tuning results on TriviaQA. Since our approach only requires fine-tuning
a smaller model, it demands less computational and memory resources compared
to fine-tuning a large model, making our method highly promising.

###  4.5 Performance on Multi-Task Scenarios

In the multi-task scenario, we categorize the multi-domain MMLU task as an
unseen task and the other four tasks as seen tasks. We then calculate the
average performance on seen and unseen tasks to evaluate the overall
generalization capability of our model. As shown in Table 2, our method
achieves a 36.35% and 13.37% improvement over directly fine-tuning on 1.1B and
7B, respectively, using multi-domain fine-tuning. Furthermore, our results
show that transferring knowledge from 7B outperforms 13B overall by 3.93%.
Specifically, the performance improvement is 11.9% for seen tasks and 0.1% for
unseen tasks. This indicates that our approach can alleviate conflicts between
multiple tasks to a certain extent and improve out-of-domain generalization by
leveraging the capabilities of the large model itself.

In multi-task settings, our method achieves a performance improvement of 71.3%
and 86.3% over 13B multi-task tuning when transferring knowledge from 1.1B and
7B, respectively. It can be observed that our method performs worse in domain-
specific tasks compared to multi-task tuning, e.g., TriviaQA. This is
primarily because we cannot access the data needed by the experts, resulting
in a bottleneck in handling task conflicts. In the future, we will explore
more effective ways to effortlessly transfer multiple small expert models to a
large model. Notably, Proxy Tuning is designed for single-domain scenarios
with a known test distribution. It struggles at test time with inputs from
unknown distributions, making it difficult to use in multi-task scenarios.
However, our method achieves the best results on unseen tasks, demonstrating
its effectiveness in enhancing the generalization ability of large models on
new tasks. Moreover, our method can effortlessly transfer knowledge from
pretrained experts across multiple domains to the large model without
requiring access to their training data.

##  5 Analysis

###  5.1 How $\alpha$ control the knowledge transfer from the small models?

Figure 4: The variation of $\alpha$ for the four experts during knowledge
transfer on an unseen task (MMLU: abstract algebra).

Table 3: The time required to train or inference 1,000 data points on a single
GPU. In the inference section, values in parentheses show the factor by which
the inference speed is slower compared to the 13B FFT model. "FFT" denotes
Full Fine-Tuning, and "LT" denotes LoRA Tuning. Our 1.1B/7B expert model use
full fine-tuning.

Model | Training | Inference
---|---|---
13B FFT | 1176s | 60s
13B LT | 836s | 60s
Proxy Tuning (from 1.1B) | 128s | 142s ($\times$2.36)
Ours (from 1.1B) | 128s | 150s ($\times$2.5)
Proxy Tuning (from 7B) | 588s | 158s ($\times$2.63)
Ours (from 7B) | 588s | 166s ($\times$2.76)

We aim to understand how $\alpha$ dynamically changes in single-task and
multi-task scenarios, and what patterns it exhibits across different tasks. As
shown in Figure 3, we compare our method with predefined $\alpha$ values of
0.5, 1.0, and 1.5. It can be observed that our method consistently outperforms
the predefined $\alpha$ settings, demonstrating the effectiveness of adaptive
adjustment at each decoding step. Additionally, the predefined method exhibits
significant variation across different tasks, whereas our method addresses
this issue and avoids the extensive effort required for parameter tuning.

As shown in Figures 3 and 4, we illustrate the variation of $\alpha$ for a
single expert and multiple experts, respectively. In Figure 3, it can be
observed that $\alpha$ fluctuates between the lower bound and upper bound. At
the lower bound, less expert knowledge is transferred, and it is more
important to retain the large model’s inherent capabilities. Conversely, at
the upper bound, more expert knowledge is transferred, requiring more
GSM8K-related capabilities for those steps. In Figure 4, when solving the
“abstract algebra” problem in MMLU using four experts, it is evident that
GSM8K transfers more knowledge overall, especially in the initial steps. Since
“abstract algebra” is a mathematically oriented problem, these results
demonstrate that our method can effectively select and transfer expert
knowledge during the decoding steps.

###  5.2 Efficiency Analysis

In a batch inference with a size of $B$, Proxy Tuning requires a complexity of
approximately $O(BV)$ for each decoding step. In contrast, our method requires
$O(nBV)$ complexity, where $n$ ($\leq 20$) is the number of parameter
searches. Since $n\ll V$, our complexity is comparable to Proxy Tuning.

In multi-task settings, performing parameter searches for each expert and then
merging them would result in exponential growth, with a complexity of
$O(n^{T}BV)$. To optimize this process, we avoid merging parameters between
experts at each step and instead select a single expert for knowledge
transfer. This reduces the algorithm complexity to $O(nTBV)$. Additionally, we
can constrain the parameter search range for each expert to achieve better
efficiency and performance.

Here, we complete the training and inference on an H100 GPU for 1000 data,
while recording the time taken. As shown in Table 4, our method achieves
similar efficiency to Proxy Tuning during inference. Due to the need for
inference and communication between multiple models, our approach is
approximately 2.5 times slower than direct inference with a 13B model.
However, our method only requires training a small model, which provides a
significant advantage in training efficiency.

###  5.3 Comparison the Weak-to-strong Approach with Other Methods

#### Comparison against In-Context Learning

(a) Enhance In-Context Learning (ICL) with our method from transferring 7B
expert knowledge.

(b) Enhance Task Arithmetic (TA) with our method from transferring 7B expert
knowledge.

Figure 5: Enhance in-context learning and task arithmetic using our method.

Enhancing in-context learning with logits arithmetic can improve large
language models. Since both methods work at the token level and need only
black-box access, they can be combined to boost model performance. We
categorize tasks as either Single-Domain (excluding MMLU) or Multi-Domain
(MMLU). The "All" category averages the results of both. Notably, we use
5-shot information as in-context examples, and we apply different few-shot
samples and prompts for various MMLU tasks. In our experiments, we enhance a
13B model’s performance using a 7B expert and compare it to the 13B model
using in-context learning. We present the results in Figure 5(a). We observed
that our method outperforms 5-shot in-context learning on both Single-Domain
and Multi-Domain tasks. Furthermore, when combined with in-context learning,
our method shows a significant improvement on Multi-Domain tasks, resulting in
an overall (All) increase of 18.3%. Specifically, combining our method with
in-context learning does not significantly impact performance on Single-Domain
tasks, indicating that our method alone has already achieved capabilities
similar to in-context learning for individual tasks.

#### Comparison against Task Arithmetic

Task Arithmetic and logits arithmetic are similar in principle, both adjusting
the shift between the expert and base model to control knowledge transfer.
However, logits arithmetic isn’t constrained by parameter size and can merge
across model levels without needing access to specific parameters.
Specifically, our method can be applied to combine experts from task
arithmetic, integrating both approaches for multi-task learning. We use the
same setup as our multi-task scenarios, treating "All" as the average of Seen
and Unseen Tasks. In figure 5(b), our method performs well on Unseen Tasks,
resulting in an overall improvement of 5.7% compared to task arithmetic. When
we combine our method with task arithmetic, we see improvements over task
arithmetic alone, achieving the highest overall performance, with increases of
7.2% and 1.5% over task arithmetic and our method alone, respectively.

##  6 Conclusions

In this paper, we introduce a dynamic logit fusion approach in weak-to-strong
specialization, which utilizes a series of task-specific small models and
allows for adaptive weight allocation among them. Through extensive
experiments, we have demonstrated the effectiveness of our approach in both
single-task and multi-task settings across various benchmarks. By transferring
expertise from the 7B model to the 13B model, we have achieved significant
performance improvements, closing the performance gap by 96.4% in single-task
scenarios and by 86.3% in multi-task scenarios compared to full fine-tuning of
the larger model. Our method also shows promising results on unseen tasks and
can integrate in-context learning for single tasks and task arithmetic for
multi-task scenarios.

## References

  * Ainsworth et al. [2023] Samuel Ainsworth, Jonathan Hayase, and Siddhartha Srinivasa.  Git re-basin: Merging models modulo permutation symmetries.  In _The Eleventh International Conference on Learning Representations_ , 2023.
  * Azerbayev et al. [2024] Zhangir Azerbayev, Hailey Schoelkopf, Keiran Paster, Marco Dos Santos, Stephen McAleer, Albert Q. Jiang, Jia Deng, Stella Biderman, and Sean Welleck.  Llemma: An open language model for mathematics, 2024.
  * Ben Zaken et al. [2022] Elad Ben Zaken, Yoav Goldberg, and Shauli Ravfogel.  BitFit: Simple parameter-efficient fine-tuning for transformer-based masked language-models.  In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio, editors, _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_ , pages 1–9, Dublin, Ireland, May 2022. Association for Computational Linguistics.  doi: 10.18653/v1/2022.acl-short.1.  URL https://aclanthology.org/2022.acl-short.1.
  * Beyer et al. [2022] Lucas Beyer, Xiaohua Zhai, Amélie Royer, Larisa Markeeva, Rohan Anil, and Alexander Kolesnikov.  Knowledge distillation: A good teacher is patient and consistent.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 10925–10934, 2022.
  * Brown et al. [2020] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei.  Language models are few-shot learners.  In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, _Advances in Neural Information Processing Systems_ , volume 33, pages 1877–1901. Curran Associates, Inc., 2020.  URL https://proceedings.neurips.cc/paper_files/paper/2020/file/1457c0d6bfcb4967418bfb8ac142f64a-Paper.pdf.
  * Burns et al. [2023] Collin Burns, Pavel Izmailov, Jan Hendrik Kirchner, Bowen Baker, Leo Gao, Leopold Aschenbrenner, Yining Chen, Adrien Ecoffet, Manas Joglekar, Jan Leike, Ilya Sutskever, and Jeff Wu.  Weak-to-strong generalization: Eliciting strong capabilities with weak supervision, 2023.
  * Chung et al. [2024] Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al.  Scaling instruction-finetuned language models.  _Journal of Machine Learning Research_ , 25(70):1–53, 2024.
  * Cobbe et al. [2021] Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al.  Training verifiers to solve math word problems.  _arXiv preprint arXiv:2110.14168_ , 2021.
  * Devlin et al. [2019] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In Jill Burstein, Christy Doran, and Thamar Solorio, editors, _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics.  doi: 10.18653/v1/N19-1423.  URL https://aclanthology.org/N19-1423.
  * Guo et al. [2024] Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Y. Wu, Y. K. Li, Fuli Luo, Yingfei Xiong, and Wenfeng Liang.  Deepseek-coder: When the large language model meets programming – the rise of code intelligence, 2024.
  * Hayou et al. [2024] Soufiane Hayou, Nikhil Ghosh, and Bin Yu.  Lora+: Efficient low rank adaptation of large models.  _arXiv preprint arXiv:2402.12354_ , 2024.
  * Hendrycks et al. [2020] Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt.  Measuring massive multitask language understanding.  In _International Conference on Learning Representations_ , 2020.
  * Hinton et al. [2015] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean.  Distilling the knowledge in a neural network.  _arXiv preprint arXiv:1503.02531_ , 2015.
  * Houlsby et al. [2019] Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly.  Parameter-efficient transfer learning for nlp.  In _International conference on machine learning_ , pages 2790–2799. PMLR, 2019.
  * Hu et al. [2021] Edward J Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al.  Lora: Low-rank adaptation of large language models.  In _International Conference on Learning Representations_ , 2021.
  * Ilharco et al. [2023] Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Ludwig Schmidt, Hannaneh Hajishirzi, and Ali Farhadi.  Editing models with task arithmetic.  In _The Eleventh International Conference on Learning Representations_ , 2023.  URL https://openreview.net/forum?id=6t0Kwf8-jrj.
  * Ji et al. [2024] Jiaming Ji, Boyuan Chen, Hantao Lou, Donghai Hong, Borong Zhang, Xuehai Pan, Juntao Dai, and Yaodong Yang.  Aligner: Achieving efficient alignment through weak-to-strong correction, 2024.
  * Jin et al. [2023] Feihu Jin, Jiajun Zhang, and Chengqing Zong.  Parameter-efficient tuning for large language model without calculating its gradients.  In Houda Bouamor, Juan Pino, and Kalika Bali, editors, _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_ , pages 321–330, Singapore, December 2023. Association for Computational Linguistics.  doi: 10.18653/v1/2023.emnlp-main.22.  URL https://aclanthology.org/2023.emnlp-main.22.
  * Jin et al. [2022] Xisen Jin, Xiang Ren, Daniel Preotiuc-Pietro, and Pengxiang Cheng.  Dataless knowledge fusion by merging weights of language models.  In _The Eleventh International Conference on Learning Representations_ , 2022.
  * Joshi et al. [2017] Mandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zettlemoyer.  Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension.  In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1601–1611, 2017.
  * Karimi Mahabadi et al. [2021] Rabeeh Karimi Mahabadi, Sebastian Ruder, Mostafa Dehghani, and James Henderson.  Parameter-efficient multi-task fine-tuning for transformers via shared hypernetworks.  In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli, editors, _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 565–576, Online, August 2021. Association for Computational Linguistics.  doi: 10.18653/v1/2021.acl-long.47.  URL https://aclanthology.org/2021.acl-long.47.
  * Kopiczko et al. [2024] Dawid Jan Kopiczko, Tijmen Blankevoort, and Yuki M Asano.  VeRA: Vector-based random matrix adaptation.  In _The Twelfth International Conference on Learning Representations_ , 2024.  URL https://openreview.net/forum?id=NjNfLdxr3A.
  * Kullback and Leibler [1951] Solomon Kullback and Richard A Leibler.  On information and sufficiency.  _The annals of mathematical statistics_ , 22(1):79–86, 1951.
  * Leviathan et al. [2023] Yaniv Leviathan, Matan Kalman, and Yossi Matias.  Fast inference from transformers via speculative decoding.  In _International Conference on Machine Learning_ , pages 19274–19286. PMLR, 2023.
  * Li and Liang [2021] Xiang Lisa Li and Percy Liang.  Prefix-tuning: Optimizing continuous prompts for generation.  In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli, editors, _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 4582–4597, Online, August 2021. Association for Computational Linguistics.  doi: 10.18653/v1/2021.acl-long.353.  URL https://aclanthology.org/2021.acl-long.353.
  * Li et al. [2023] Xiang Lisa Li, Ari Holtzman, Daniel Fried, Percy Liang, Jason Eisner, Tatsunori Hashimoto, Luke Zettlemoyer, and Mike Lewis.  Contrastive decoding: Open-ended text generation as optimization.  In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki, editors, _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 12286–12312, Toronto, Canada, July 2023. Association for Computational Linguistics.  doi: 10.18653/v1/2023.acl-long.687.  URL https://aclanthology.org/2023.acl-long.687.
  * Lialin et al. [2023] Vladislav Lialin, Vijeta Deshpande, and Anna Rumshisky.  Scaling down to scale up: A guide to parameter-efficient fine-tuning, 2023.
  * Liashchynskyi and Liashchynskyi [2019] Petro Liashchynskyi and Pavlo Liashchynskyi.  Grid search, random search, genetic algorithm: a big comparison for nas.  _arXiv preprint arXiv:1912.06059_ , 2019.
  * Liebenwein et al. [2021] Lucas Liebenwein, Cenk Baykal, Brandon Carter, David Gifford, and Daniela Rus.  Lost in pruning: The effects of pruning neural networks beyond test accuracy.  _Proceedings of Machine Learning and Systems_ , 3:93–138, 2021.
  * Lin et al. [2023] Bill Yuchen Lin, Abhilasha Ravichander, Ximing Lu, Nouha Dziri, Melanie Sclar, Khyathi Chandu, Chandra Bhagavatula, and Yejin Choi.  The unlocking spell on base llms: Rethinking alignment via in-context learning, 2023.
  * Lin et al. [2022] Stephanie Lin, Jacob Hilton, and Owain Evans.  Truthfulqa: Measuring how models mimic human falsehoods.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 3214–3252, 2022.
  * Liu et al. [2024] Alisa Liu, Xiaochuang Han, Yizhong Wang, Yulia Tsvetkov, Yejin Choi, and Noah A Smith.  Tuning language models by proxy.  _arXiv preprint arXiv:2401.08565_ , 2024.
  * Liu et al. [2020] Jialin Liu, Antoine Moreau, Mike Preuss, Jeremy Rapin, Baptiste Roziere, Fabien Teytaud, and Olivier Teytaud.  Versatile black-box optimization.  In _Proceedings of the 2020 Genetic and Evolutionary Computation Conference_ , pages 620–628, 2020.
  * Luo et al. [2023] Haipeng Luo, Qingfeng Sun, Can Xu, Pu Zhao, Jianguang Lou, Chongyang Tao, Xiubo Geng, Qingwei Lin, Shifeng Chen, and Dongmei Zhang.  Wizardmath: Empowering mathematical reasoning for large language models via reinforced evol-instruct, 2023.
  * Matena and Raffel [2022a] Michael S Matena and Colin Raffel.  Merging models with fisher-weighted averaging.  In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, _Advances in Neural Information Processing Systems_ , 2022a.  URL https://openreview.net/forum?id=LSKlp_aceOC.
  * Matena and Raffel [2022b] Michael S Matena and Colin A Raffel.  Merging models with fisher-weighted averaging.  _Advances in Neural Information Processing Systems_ , 2022b.
  * Miao et al. [2024] Xupeng Miao, Gabriele Oliaro, Zhihao Zhang, Xinhao Cheng, Zeyu Wang, Zhengxin Zhang, Rae Ying Yee Wong, Alan Zhu, Lijie Yang, Xiaoxiang Shi, Chunan Shi, Zhuoming Chen, Daiyaan Arfeen, Reyna Abhyankar, and Zhihao Jia.  Specinfer: Accelerating large language model serving with tree-based speculative inference and verification.  In _Proceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 3_ , ASPLOS ’24. ACM, April 2024.  doi: 10.1145/3620666.3651335.  URL http://dx.doi.org/10.1145/3620666.3651335.
  * Mitchell et al. [2024] Eric Mitchell, Rafael Rafailov, Archit Sharma, Chelsea Finn, and Christopher D Manning.  An emulator for fine-tuning large language models using small language models.  In _The Twelfth International Conference on Learning Representations_ , 2024.  URL https://openreview.net/forum?id=Eo7kv0sllr.
  * Patterson et al. [2021] David Patterson, Joseph Gonzalez, Quoc Le, Chen Liang, Lluis-Miquel Munguia, Daniel Rothchild, David So, Maud Texier, and Jeff Dean.  Carbon emissions and large neural network training, 2021.
  * Raffel et al. [2020] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _Journal of Machine Learning Research_ , 21(140):1–67, 2020.  URL http://jmlr.org/papers/v21/20-074.html.
  * Rozière et al. [2024] Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Romain Sauvestre, Tal Remez, Jérémy Rapin, Artyom Kozhevnikov, Ivan Evtimov, Joanna Bitton, Manish Bhatt, Cristian Canton Ferrer, Aaron Grattafiori, Wenhan Xiong, Alexandre Défossez, Jade Copet, Faisal Azhar, Hugo Touvron, Louis Martin, Nicolas Usunier, Thomas Scialom, and Gabriel Synnaeve.  Code llama: Open foundation models for code, 2024.
  * Schick et al. [2024] Timo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom.  Toolformer: Language models can teach themselves to use tools.  _Advances in Neural Information Processing Systems_ , 36, 2024.
  * See et al. [2017] Abigail See, Peter J. Liu, and Christopher D. Manning.  Get to the point: Summarization with pointer-generator networks.  In Regina Barzilay and Min-Yen Kan, editors, _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1073–1083, Vancouver, Canada, July 2017. Association for Computational Linguistics.  doi: 10.18653/v1/P17-1099.  URL https://aclanthology.org/P17-1099.
  * Shen et al. [2024] Shannon Zejiang Shen, Hunter Lang, Bailin Wang, Yoon Kim, and David Sontag.  Learning to decode collaboratively with multiple language models, 2024.
  * Stanton et al. [2021] Samuel Stanton, Pavel Izmailov, Polina Kirichenko, Alexander A Alemi, and Andrew G Wilson.  Does knowledge distillation really work?  _Advances in Neural Information Processing Systems_ , 34:6906–6919, 2021.
  * Tang et al. [2023] Qiaoyu Tang, Ziliang Deng, Hongyu Lin, Xianpei Han, Qiao Liang, Boxi Cao, and Le Sun.  Toolalpaca: Generalized tool learning for language models with 3000 simulated cases, 2023.
  * Touvron et al. [2023a] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom.  Llama 2: Open foundation and fine-tuned chat models, 2023a.
  * Touvron et al. [2023b] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al.  Llama 2: Open foundation and fine-tuned chat models.  _arXiv preprint arXiv:2307.09288_ , 2023b.
  * Wang et al. [2024] Hongyi Wang, Felipe Maia Polo, Yuekai Sun, Souvik Kundu, Eric Xing, and Mikhail Yurochkin.  Fusing models with complementary expertise.  In _The Twelfth International Conference on Learning Representations_ , 2024.  URL https://openreview.net/forum?id=PhMrGCMIRL.
  * Wortsman et al. [2022] Mitchell Wortsman, Gabriel Ilharco, Samir Yitzhak Gadre, Rebecca Roelofs, Raphael Gontijo Lopes, Ari S. Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, and Ludwig Schmidt.  Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time.  In _International Conference on Machine Learning_ , 2022.
  * Yadav et al. [2023] Prateek Yadav, Derek Tam, Leshem Choshen, Colin Raffel, and Mohit Bansal.  TIES-merging: Resolving interference when merging models.  In _Thirty-seventh Conference on Neural Information Processing Systems_ , 2023.  URL https://openreview.net/forum?id=xtaX3WyCj1.
  * Yang et al. [2024] Enneng Yang, Zhenyi Wang, Li Shen, Shiwei Liu, Guibing Guo, Xingwei Wang, and Dacheng Tao.  Adamerging: Adaptive model merging for multi-task learning.  In _The Twelfth International Conference on Learning Representations_ , 2024.  URL https://openreview.net/forum?id=nZP6NgD3QY.
  * Zhang et al. [2024] Peiyuan Zhang, Guangtao Zeng, Tianduo Wang, and Wei Lu.  Tinyllama: An open-source small language model.  _arXiv preprint arXiv:2401.02385_ , 2024.

##  Appendix A Model Architecture Diagram

As illustrated in Figure 6, our approach dynamically calculates the fusion
weights of the logits of SLMs at each decoding step, transferring the
knowledge from multiple experts to LLM.

Figure 6: The architecture of our method. The small llama represents the small
model, while the large llama represents the large model. $M^{S}/M^{L}$ denotes
the logits of the small/large language model. $M^{S}_{t}$ represents the
logits of a small expert language model for task $t$. The lower part of the
figure illustrates our optimization in the decoding process, where each circle
represents a decoding step. The upper part of the figure shows how our method
transfers the knowledge of experts in the $k$th step. At each decoding step,
our method dynamically adjusts the $\\{\alpha_{t}\\}_{t=1}^{T}$ value for each
expert, transferring knowledge from the small models to the larger model.

##  Appendix B Proof of Method

Based on the definitions of the probability distributions $Q$, $Q_{ft}$, $P$,
and $\tilde{P}$, the model logits outputs $M^{S}$, $M^{S}_{ft}$, and $M^{L}$
satisfy the following relationships:

$\displaystyle Q(x_{k}\mid x_{<k})=\mathrm{softmax}(M^{S}(x_{k}|x_{<k}))$ (7)
$\displaystyle Q_{ft}(x_{k}\mid
x_{<k})=\mathrm{softmax}(M^{S}_{ft}(x_{k}|x_{<k}))$ $\displaystyle P(x_{k}\mid
x_{<k})=\mathrm{softmax}(M^{L}(x_{k}|x_{<k}))$
$\displaystyle\tilde{P}(x_{k}|x_{<k})=\mathrm{softmax}(M^{L}(x_{k}|x_{<k})+\alpha\cdot(M^{S}_{ft}(x_{k}|x_{<k})-M^{S}(x_{k}|x_{<k})))$

Suppose there are vectors $x$ and $y$ of dimension $n$. The relationship
between them is as follows:

$\displaystyle y=softmax(x)$
$\displaystyle=[\frac{e^{x_{i}}}{\sum_{j=1}^{n}e^{x_{j}}}]_{i=n}^{n}=[\frac{1}{\sum_{j=1}^{n}e^{x_{j}-x_{i}}}]_{i=n}^{n}$
(8)

For each value $x_{i}$, as $x_{i}$ increases, $x_{j}-x_{i}$ decreases, and
$\frac{1}{e^{x_{j}-x_{i}}}$ increases. Therefore, the corresponding $y_{i}$ is
positively correlated with $x_{i}$.

So we can know that $\ln y\propto y\propto x$. Therefore, based on Equation
(LABEL:eq:10), we can derive the following relationship:

$\displaystyle\ln{Q(x_{k}\mid x_{<k})}\propto M^{S}(x_{k}|x_{<k})$ (9)
$\displaystyle\ln{Q_{ft}(x_{k}\mid x_{<k})}\propto M^{S}_{ft}(x_{k}|x_{<k})$
$\displaystyle\ln{P(x_{k}\mid x_{<k})}\propto M^{L}(x_{k}|x_{<k})$
$\displaystyle\ln{\tilde{P}(x_{k}|x_{<k})}\propto(M^{L}(x_{k}|x_{<k})+\alpha\cdot(M^{S}_{ft}(x_{k}|x_{<k})-M^{S}(x_{k}|x_{<k}))$
$\displaystyle=\ln{P(x_{k}\mid x_{<k})}+\alpha\cdot(\ln{Q_{ft}(x_{k}\mid
x_{<k})}-\ln{Q(x_{k}\mid x_{<k})})$ $\displaystyle=\ln{\Biggl{(}P(x_{k}\mid
x_{<k})\cdot\Biggl{(}\frac{Q_{ft}(x_{k}\mid x_{<k})}{Q(x_{k}\mid
x_{<k})}\Biggr{)}^{\alpha}\Biggr{)}}$

Thus, logit arithmetic can be viewed as a product-of-experts ensemble as
follows:

$\displaystyle\tilde{P}(x_{k}\mid x_{<k})\propto P(x_{k}\mid
x_{<k})\Biggl{(}\frac{Q_{ft}(x_{k}\mid x_{<k})}{Q(x_{k}\mid
x_{<k})}\Biggr{)}^{\alpha}$ (10)

Essentially, it assumes a predefined distribution shift
$(\frac{Q_{ft}(x_{k}\mid x_{<k})}{Q(x_{k}\mid x_{<k})})^{\alpha}$ between the
fine-tuned and pretrained distributions of a small model, which can construct
the trajectory of $\tilde{P}$. It then attempts to approximate the shift for
the fine-tuned and pretrained distributions of the large model
$\frac{\tilde{P}(x_{k}\mid x_{<k})}{P(x_{k}\mid x_{<k})}$,.

##  Appendix C Pseudo Code

Algorithm 1 Adaptive Logits-Arithmetic

1:generation prompt $X$, number of tokens to generate $N$, Large model
$\bm{\theta^{L}}$, domain number $T$, expert small models
$\\{\bm{\theta^{S}_{t}}\\}_{t=1}^{T}$, $M^{L}$, $M^{S}$, and $M^{S}_{t}$
represent the logits outputs of the large model, small model, and $t$-th
domain-specific small models. $P$, $Q$, and $Q_{t}$ represent the outputs
distribution of the large model, small model, and $t$-th domain-specific small
models.

2:

3:$k\leftarrow len(X)-1$

4:$m\leftarrow\infty$

5:$\alpha\leftarrow[]$

6:while $\text{len}(X)<N$ and $x_{k-1}\neq\text{[EOS]}$ do

7:

8: for each domain $t$ in $T$ do

9: Get domain expert logits for $x_{k}$ from $\bm{\theta^{S}_{t}}$ as
$M^{S}_{t}(x_{k}|x_{<k})$

10: end for

11:

12: # For multitask scenario, perform $\alpha$ search only for one task, for a
total of $T$ times.

13: for search $\alpha^{\prime}\in\mathbb{R}^{T}$ do $\triangleright$ such as
[0.0,2.0], step is 0.1

14: $\tilde{P}(x_{k}\mid
x_{<k})\leftarrow\mathrm{softmax}\left[M^{L}(x_{k}|x_{<k})+\sum_{t=1}^{T}\alpha_{t}^{\prime}\cdot(M^{S}_{t}(x_{k}|x_{<k})-M^{S}(x_{k}|x_{<k}))\right]$

15:
$\mathcal{L}\leftarrow\sum_{t=1}^{T}\left(\text{KL}(\tilde{P}||P)-\text{KL}(Q_{t}||Q)\right)^{2}+\left(\text{KL}(P||\tilde{P})-\text{KL}(Q||Q_{t})\right)^{2}$

16: if $\mathcal{L}<m$ then

17: $\alpha\leftarrow\alpha^{\prime}$

18: $m\leftarrow\mathcal{L}$

19: end if

20: end for

21:

22: Calculate next token distribution for the large model as
$\tilde{P}\leftarrow\mathrm{softmax}\left[M^{L}(x_{k}|x_{<k})+\sum_{t=1}^{T}\alpha_{t}\cdot(M^{S}_{t}(x_{k}|x_{<k})-M^{S}(x_{k}|x_{<k}))\right]$

23: Sample the next token $x_{k}\sim\tilde{P}(x_{k}|x_{<k})$

24: $X\leftarrow\\{X;x_{k}\\}$

25: $k\leftarrow k+1$

26:end while

27:return generated text $X$

28:

Our algorithm is outlined in pseudo-code in Algorithm 1.

##  Appendix D Dataset Details

  * •

GSM8K [8], a dataset of 8.5K high-quality linguistically diverse grade school
math word problems created by human problem writers. We evaluate using exact
match (EM.).

  * •

TruthfulQA [31], a benchmark to measure whether a language model is truthful
in generating answers to questions. There is no training set for this dataset.
We evaluate using multiple-choice accuracy (Acc.).

  * •

TriviaQA [20], a reading comprehension dataset containing over 650K question-
answer-evidence triples. We evaluate using exact match (EM.).

  * •

MMLU [12], a benchmark designed to evaluate the capabilities of language
models. It comprises approximately 16,000 multiple-choice questions across 57
tasks. We evaluate using multiple-choice accuracy (Acc.).

  * •

CNN-DailyMail (CNN/DM) [43], a dataset for text summarization. We evaluate
using Rouge2.

The GSM8K, MMLU datasets are licensed under the MIT License. The TruthfulQA,
TriviaQA, CNN-Dailymail datasets are licensed under the Apache-2.0 License.
Thus, these datasets are available for non-commercial research purposes.

##  Appendix E Case study

As shown in Table 4, our method achieves the correct output of 18 when
targeting a single task, such as the GSM8K problem. Additionally, it can be
observed that the model’s output contains formatted information, e.g., “####"
which matches the format learned in in-context learning. This indicates that
our method achieves the ability of in-context learning at the logit level.

For unseen tasks like MMLU’s Abstract Algebra and Clinical Knowledge, our
model generates correct answers by leveraging multi-expert knowledge transfer.

Table 4: Case Study GSM8K
---
Q: Janet2̆019s ducks lay 16 eggs per day. She eats three for breakfast every
morning and bakes muffins for her friends every day with four. She sells the
remainder at the farmers’ market daily for $2 per fresh duck egg. How much in
dollars does she make every day at the farmers’ market?
LLAMA2-13B:
n
nAnswer:
LLAMA2-13B + In-Context Learning (5-shot): Janet eats 3 eggs for breakfast
every morning.
She bakes muffins for her friends every day with 4 eggs.
She sells the remainder at the farmers’ market daily for $2 per fresh duck
egg.
So she makes $2 x 13 = $<<2*13=26>>26 every day at the farmers’ market.
#### 26
Ours (LLAMA2-13B + GSM8K expert): Janet eats 3 eggs for breakfast and bakes 4
eggs into muffins for a total of 3 + 4 = <<3+4=7>>7 eggs per day.
She has 16 - 7 = <<16-7=9>>9 eggs left to sell at the farmers’ market.
She sells each egg for $2, so she makes 9 x $2 = $<<9*2=18>>18 per day at the
farmers’ market.
#### 18
MMLU: Abstract Algebra
Q: Statement 1 | Every group of order 42 has a normal subgroup of order 7. Statement 2 | Every group of order 42 has a normal subgroup of order 8.
A. True, True
B. False, False
C. True, False
D. False, True ?
LLAMA2-13B: D. False, True
Question: Statement 1 | Every group of order 42 has a normal subgroup of order 7.
Statement 2 | Every group of order 4
Ours LLAMA2-13B + GSM8K expert + TruthfulQA expert + TriviaQA expert + CNN DM
expert: C. True, False . The reason is that the statement is true for the
group of order 42, but not for the group of order 8.
MMLU: Clinical Knowledge
Q: Wound drainage is described as serous if:
A. it contains red blood cells.
B. it is thick and yellow or green.
C. it contains microorganisms.
D. it is clear or slightly yellow.
LLAMA2-13B: D
Question: Which of the following is not a characteristic of a normal blood
film?
LLAMA2-13B + In-Context Learning (5-shot): D. it is clear or slightly yellow.
Ours (LLAMA2-13B + GSM8K expert + TruthfulQA expert + TriviaQA expert + CNN DM
expert): D. it is clear or slightly yellow.

##  Appendix F Limitations and Future Works

Our approach shares the general limitations of existing logit-arithmetic-based
methods: (1) Since our method operates at the decoding step, this architecture
mainly tests its ability in generative models, without exploring its
effectiveness in discriminative models; (2) Involving multiple models in the
decoding step can slow down inference compared to inference with models of
similar magnitude.

In the future, we will explore more efficient methods of logit arithmetic and
investigate their potential application in other architectural models.

##  Appendix G Broader Impacts

Our approach enables weak-to-strong knowledge transfer without accessing the
model’s parameters or expert-learning data. This is advantageous for
protecting the privacy of expert data. Additionally, this data-free fine-
tuning paradigm significantly reduces the cost of private deployment while
enhancing data security, thus positively impacting societal data security. Our
method does not exhibit any significant negative societal impacts.