# Inferflow: an Efficient and Highly Configurable Inference Engine for Large
Language Models

Shuming Shi1 Enbo Zhao1 Deng Cai1 Leyang Cui1 Xinting Huang1 Huayang Li1,2

1Tencent AI Lab

2Nara Institute of Science and Technology

https://github.com/inferflow/inferflow
Work was done during the internship at Tencent AI Lab.

###### Abstract

We present Inferflow, an efficient and highly configurable inference engine
for large language models (LLMs). With Inferflow, users can serve most of the
common transformer models by simply modifying some lines in corresponding
configuration files, without writing a single line of source code. Compared
with most existing inference engines, Inferflow has some key features. First,
by implementing a modular framework of atomic build-blocks and technologies,
Inferflow is compositionally generalizable to new models. Second, 3.5-bit
quantization is introduced in Inferflow as a tradeoff between 3-bit and 4-bit
quantization. Third, hybrid model partitioning for multi-GPU inference is
introduced in Inferflow to better balance inference speed and throughput than
the commonly-adopted partition-by-layer and partition-by-tensor strategies.

##  1 Introduction

Large language models (LLMs) (OpenAI, 2023; Touvron et al., 2023) have shown
remarkable capabilities across a wide range of NLP tasks. Despite their
impressive performance, deploying LLMs also poses several challenges,
particularly in terms of computational complexity, resource requirements, and
inference latency. The size of LLMs, often consisting of billions of
parameters, makes them difficult to deploy in real-world applications,
especially those that require low-latency and resource-constrained
environments.

In light of these challenges, we release Inferflow, an efficient and highly
configurable inference engine for LLMs. We optimize the inference process by
targeting on inference speed, throughput, result quality, VRAM consumption,
and extensibility.

Figure 1 lists major requirements for an LLM inference engine and possible
technologies to address them. The implementation status of the technologies in
Inferflow is also depicted in the figure.

Figure 1: Implementation status of key technologies in Inferflow.

Compared with existing popular inference engines (llama.cpp, ; DeepSpeed-MII,
; TensorRT-LLM, ; vLLM, ), Inferflow has the following features:

  1. 1.

Extensible and highly configurable (Section 2): A typical way of using
Inferflow to serve a new model is editing a model specification file, but not
adding/editing source codes. We implement in Inferflow a modular framework of
atomic building-blocks and technologies, making it compositionally
generalizable to new models. A new model can be served by Inferflow if the
atomic building-blocks and technologies in this model have been “known” to
Inferflow.

  2. 2.

3.5-bit quantization (Section 3): Inferflow implements 2-bit, 3-bit, 3.5-bit,
4-bit, 5-bit, 6-bit and 8-bit quantization. Among the quantization schemes,
3.5-bit quantization is a new one introduced by Inferflow.

  3. 3.

Hybrid model partition for multi-GPU inference (Section 4): Inferflow supports
multi-GPU inference with three model partitioning strategies to choose from:
partition-by-layer, partition-by-tensor, and hybrid partitioning. Among them,
hybrid partitioning is a new strategy introduced in this report.

  4. 4.

Wide file format support (and safely loading pickle data): Inferflow supports
loading models of multiple file formats directly, without reliance on an
external converter. Supported formats include pickle, safetensors, llama.cpp
gguf, etc. It is known that there are security issues to read pickle files
using Python codes111https://huggingface.co/docs/hub/security-pickle. By
implementing a simplified pickle parser in C++, Inferflow supports safely
loading models from pickle data.

  5. 5.

Wide network type support: Supporting three types of transformer models:
decoder-only models, encoder-only models, and encoder-decoder models.

  6. 6.

GPU/CPU hybrid inference: Supporting GPU-only, CPU-only, and GPU/CPU hybrid
inference.

In this technical report, we give a brief description about the implementation
of some key technologies in Inferflow for an efficient and extensible
inference engine. The source codes of Inferflow can be found at
https://github.com/inferflow/inferflow.

##  2 Modular Framework of Atomic Build-Blocks

Existing LLM inference engines typically adopt two approaches to support
multiple models. The first way is adding “if-else” clauses in multiple places
of the source codes and implementing model-specific logic in corresponding
code sections. The second way is maintaining a dedicated file for each model.
Taking the Huggingface Transformers (Wolf et al., 2020) as an example, the
structure of Llama is defined in models/llama/modeling_llama.py. With the two
approaches, supporting a new model often lead to the adding of new codes. The
second approach should be beneficial for exploring new training and inference
techniques. However, if a new model is a combination of existing techniques,
probably there are better ways to organize an inference engine to support the
new model with less effort.

The design goal of Inferflow is to support a new model with zero code change,
as long as the model is a combination of known techniques. The basic idea is
to implement a modular framework of atomic building-blocks and technologies.
The “if-else” or “switch” clauses are performed on the level of atomic
building-blocks (or called modules) but not model types. Examples of atomic
modules include normalization functions, activation functions, position
embedding algorithms, the grouped-query attention mechanism, etc. In this way,
a model is defined by determining the value or parameters of each atomic
module. Naturally, this can be done with the aid of a configuration file. By
implementing such a modular framework, a typical way of serving a new model in
Inferflow is editing a model specification file, but not adding/editing source
codes. Table 1 shows the contents of three model specification files
(corresponding to three models) for comparison.

Model | Mistral-7B-Instruct | Facebook-m2m100-418M | BERT-base-multilingual
---|---|---|---
Jiang et al. (2023) | Fan et al. (2020) | Devlin et al. (2018)
model_file_format | pickle | pickle | safetensors
tokenizer_file | tokenizer.json | vocab.json | tokenizer.json
tokenization_algorithm | bpe | bpe | bpe
generation_config | generation_config.json | generation_config.json | generation_config.json
network_structure: |  |  |
type | decoder_only | encoder_decoder | encoder_only
normalization_function | rms | std | std
activation_function | silu | relu | gelu
position_embedding | rope | sinusoidal | empty
qk_column_order | 2 | 2 | 2
qkv_format | - | 1 | -
tensor_name_prefix | model. | model. | bert.
tensor_name_mapping | (1 line) | (16 lines) | (26 lines)

Table 1: The specification of three models.

##  3 Quantization

Quantization is a process that reduces the numerical precision of model
weights and/or activations to lower precision data types, such as 8-bit or
4-bit integers, instead of using floating-point numbers (e.g., float32 or
float16). The main benefits of quantization are:

  * •

Reduced memory usage: Lower precision data types require less memory, enabling
the deployment of large models on memory-constrained devices.

  * •

Faster inference: Quantized operations can be significantly faster than their
floating-point counterparts, leading to reduced inference time.

In Inferflow, we focus on post-training quantization (Dettmers et al., 2022b;
a; Yao et al., 2022), which quantizes the model after training. It involves
converting the model’s weights and activations to lower precision data types
and then fine-tuning the quantization parameters to minimize the loss of
accuracy. Like in llama.cpp, we implement fast block-level linear quantization
algorithms in Inferflow. The process can be formulated as follows:

Let $\bm{w}$ be the weight vector in a tensor block of the original model, and
$\bm{q}$ be the corresponding quantized weight vector. The quantization
process maps $\bm{w}$ to $\bm{q}$, where each element in $\bm{q}$ is a k-bit
integer. The mapping is defined as:

$\bm{q}=\mathrm{Round}(\frac{\bm{w}-\min(\bm{w})}{\max(\bm{w})-\min(\bm{w})}\times(2^{k}-1))$
(1)

where $\min(\bm{w})$ and $\max(\bm{w})$ are the minimum and maximum elements
in the block, respectively, and $\mathrm{Round}(\cdot)$ is the element-wise
rounding function.

After quantization, the weights can be dequantized back to their original
precision for inference:

$\bm{w}^{\prime}=\frac{\bm{q}}{2^{k}-1}\times(\max(\bm{w})-\min(\bm{w}))+\min(\bm{w})$
(2)

where $\bm{w}^{\prime}$ is the dequantized weight.

### 3.5-bit quantization.

In $k$-bit post-training quantization, the model’s weights are quantized to
$k$ bits, which significantly reduces the model’s memory footprint and can
speed up inference, especially on hardware that is optimized for low-precision
computations. The value of $k$ can be any integer, with common practice being
2, 3, 4, 5, 6, and 8. However, we empirically found that LLMs perform
reasonably well with 4-bit quantization, but the performance is relatively
poor with 3-bit quantization. Therefore, we propose 3.5-bit quantization,
where 7 bits are used to represent two adjacent weights.

Equation 1 is modified to

$\bm{q}=\mathrm{Round}(\frac{\bm{w}-\min(\bm{w})}{\max(\bm{w})-\min(\bm{w})}\times
10)$ (3)

Then we use a 7-bit number $q$ to represent two adjacent weights
${\bm{q}_{2i}}$ and ${\bm{q}_{2i+1}}$:

$\bm{q}={\bm{q}_{2i}}\times 11+{\bm{q}_{2i+1}}$ (4)

During dequantization, the weights are first decomposed by:

$\begin{split}\bm{q}_{1}&=\lfloor\bm{q}/11\rfloor\\\ \bm{q}_{2}&=\bm{q}\mod
11\end{split}$ (5)

Then, $\bm{q}_{1}$ and $\bm{q}_{2}$ are fed into Equation 2 for
dequantization.

| 4-bit | 3-bit | 3.5-bit
---|---|---|---
$w$ | $q$ | $w^{\prime}$ | $\Delta$ | $q$ | $w^{\prime}$ | $\Delta$ | $q$ | $w^{\prime}$ | $\Delta$
-1 | 0 | -1.000 | 0.000 | 0 | -1.000 | 0.000 | 0 | -1.000 | 0.000
-0.9 | 1 | -0.833 | 0.067 | 0 | -1.000 | 0.100 | 0 | -1.000 | 0.100
-0.6 | 2 | -0.667 | 0.067 | 1 | -0.643 | 0.043 | 2 | -0.500 | 0.100
-0.4 | 4 | -0.333 | 0.067 | 2 | -0.286 | 0.086 | 2 | -0.500 | 0.100
-0.2 | 5 | -0.167 | 0.033 | 2 | -0.286 | 0.086 | 3 | -0.250 | 0.050
0 | 6 | 0.000 | 0.000 | 3 | 0.071 | 0.071 | 4 | 0.000 | 0.000
0.1 | 7 | 0.167 | 0.067 | 3 | 0.071 | 0.029 | 4 | 0.000 | 0.100
0.5 | 9 | 0.500 | 0.000 | 4 | 0.429 | 0.071 | 6 | 0.500 | 0.000
0.7 | 10 | 0.667 | 0.033 | 5 | 0.786 | 0.086 | 7 | 0.750 | 0.050
1 | 12 | 1.000 | 0.000 | 6 | 1.143 | 0.143 | 8 | 1.000 | 0.000
1.3 | 14 | 1.333 | 0.033 | 6 | 1.143 | 0.143 | 9 | 1.250 | 0.050
1.5 | 15 | 1.500 | 0.000 | 7 | 1.500 | 0.000 | 10 | 1.500 | 0.000
Avg error |  |  | 0.031 |  |  | 0.075 |  |  | 0.046
Table 2: An example of quantization error statistics across various quantization bits. Quantization name | FP16 | Q8 | Q6 | Q5 | Q4_B32 | Q4_B64 | Q3H | Q3_B32
---|---|---|---|---|---|---|---|---
Quantization bits | 16-bit | 8-bit | 6-bit | 5-bit | 4-bit | 4-bit | 3.5-bit | 3-bit
Block size | - | 32/64 | 64 | 64 | 32 | 64 | 64 | 32
Actual bits/weight | 16 | 9/8.5 | 6.5 | 5.5 | 5 | 4.5 | 4 | 4
Bloom-3B | 18.056 | 18.056 | 18.095 | 18.169 | 18.304 | 18.557 | 19.208 | 19.882
LLAMA2-7B | 7.175 | 7.177 | 7.173 | 7.198 | 7.454 | 7.569 | 7.914 | 8.817
LLAMA2-13B | 6.383 | 6.382 | 6.391 | 6.434 | 6.514 | 6.647 | 6.897 | 7.278
Falcon-40B | 6.965 | 6.965 | 6.971 | 6.980 | 7.014 | 7.052 | 7.153 | 7.331
Table 3: Perplexity of some models across various quantization bits on
Wikitext-2.

Table 2 presents the quantization errors across various quantization levels,
using an example of 12 weights. It is evident that the proposed 3.5-bit
quantization significantly reduces quantization errors compared to 3-bit
quantization. We report, in Table 3, the perplexity of several models on
Wikitext-2 for different quantization levels. Several observations can be
made: First, 8-bit quantization almost performs as well as the original FP16
version. Second, 6-bit quantization only leads to a tiny quality loss. Third,
3.5-bit quantization performs significantly better than 3-bit quantization.
This justifies 3.5-bit quantization as an additional choice for users. Please
note that two FP16 numbers are stored for each block as auxiliary data for
dequantization. As a result, the actual average bits per weight in memory is
larger than quantization bits, as listed in the table. It can be seen that
Q3_B32 (3-bit quantization with block size 32) has the same actual bits per
weight as Q3H (3.5-bit quantization with block size 64). This observation
means we should always adopt Q3H over Q3_B32.

##  4 Hybrid Partition

For a large model, we may have to split it to multiple GPU devices because it
exceeds the VRAM of a single GPU. Using multiple GPU devices to serve a model
may also increase the inference speed. There are two common strategies for
model partitioning in inference time: layer-wise partition (pipeline
parallelism) and tensor-wise partition (tensor parallelism). In layer-wise
partition, each GPU device is in charge of a subset of model layers. The GPU
devices process the layers sequentially to complete the inference process. On
the other hand, tensor-wise partition involves dividing most tensors in the
model into smaller segments, and then distributing these segments across
multiple GPUs. Each GPU performs a partial computation, and the results are
merged to obtain the final output. This method is particularly effective for
the models with large weight matrices, as it reduces both the memory
requirements and the computational load on each GPU. However, at each layer,
the computation results on the GPUs have to be merged twice (for the self-
attention sub-layer and the feed-forward sub-layer respectively). For machines
with low inter-device communication bandwidth, the latency of merging the
intermediate results may be high, especially when many GPU cards are involved
in the merging process.

We propose and implement a new partitioning strategy called hybrid
partitioning, with the goal of better balancing inference speed and throughput
than the above two strategies. An example of hybrid partitioning on 4 devices
is shown in Table 4. Table 5 shows a preliminary experiment on 4 NVIDIA Tesla
V100 cards for comparing the throughput and decoding speed of various
partition strategies. The proposed hybrid partition surpasses the layer-wise
partition in terms of decoding speed, and exceeds the tensor-wise partition
regarding throughput.

Device | Assignments
---|---
Device-0 | Layers: [1, 20], Heads: [1, 16]
Device-1 | Layers: [1, 20], Heads: [17, 32]
Device-2 | Layers: [21, 40], Heads: [1, 16]
Device-3 | Layers: [21, 40], Heads: [17, 32]
Table 4: An illustration of hybrid partition. Partition | Throughput (#tokens/s) | Decoding speed (#tokens/s)
---|---|---
Layer-wise | 32 | 8
Tensor-wise | 12 | 12
Hybrid | 24 | 12
Table 5: Comparison of different partition strategies (model:
Falcon-40B-Instruct, data type: FP16, device: Tesla V100, batch-size: 1)

(a) Static batching

(b) Dynamic batching

Figure 2: An illustration of two batching algorithms.

##  5 Dynamic Batching

Batching is a technique that allows the model to process multiple input
sequences simultaneously. In traditional static batching, input sequences of
varying lengths are typically padded to the max length in each batch. This
strategy is suitable for offline training and inference, since all input
sequences can be first sorted based on their lengths. This step ensures that
sequences with similar lengths are processed together, reducing the overall
padding required. While during online inference, it is common for different
input sequences to be received and processed in real time. Inspired by Yu et
al. (2022), we implement a dynamic batching technique that allows for
generating responses without waiting for the decoding process of previous
input sequences to be completed. As illustrated in Figure 2, consider two
sentences
$[[S^{1}_{1},S^{1}_{2},S^{1}_{3},S^{1}_{4}],[S^{2}_{1},S^{2}_{2},S^{2}_{3}]]$
undergoing the decoding process. The system receives an additional request
$S^{3}$ to generate at time step 3 ($T_{3}$). With static batching, the
current batch must be completed before generating $S^{3}$ at $T_{5}$. In
contrast, dynamic batching enables immediate inference of $S^{3}$ at $T_{3}$
by incorporating the sentence into the current inference batch. Inferflow
implements two member functions in the InferenceEngine class as the interface
of dynamic batching:

  * •

AddQuery($S$): Add a new query ($S$) to the query pool

  * •

Infer(): Conduct a single-step inference using the query pool, resulting in a
set of next tokens.

Taking Figure 2(b) as an example, a user call AddQuery($S^{3}$) at $T_{3}$,
then call Infer() to return $S^{1}_{3},S^{2}_{3},S^{3}_{1}$.

##  6 Decoding Strategies

Inferflow supports a diverse range of fast decoding strategies.

### Top-$k$ Sampling

(Fan et al., 2018) is used to ensure that the less probable words, which are
in the unreliable tail of the distribution, should not have any chance to be
selected. Only top-$k$ probable tokens are considered for generation.

### Top-$p$ Sampling

(Holtzman et al., 2020) only considers the minimal set of most probable tokens
$\mathcal{V}^{p}$ that cover a specified percentage $p$ of the distribution.

### Frustratingly Simple Decoding (FSD)

(Yang et al., 2023) exploits the contrasts between the LLM and an auxiliary
anti-LM constructed based on the current prefix. The anti-LM can be
implemented as simple as an $n$-gram model. Specifically, the FSD score is
defined as

$\begin{split}\mathrm{FSD}(y|\mathbf{x},\mathbf{y}_{<t})=(1-\alpha)P_{\theta}(y|\mathbf{x},\mathbf{y}_{<t})-\alpha\times
P_{\omega}(y|\mathbf{x},\mathbf{y}_{<t})\end{split}$

where $P_{\theta}$ and $P_{\omega}$ represent the LM and the anti-LM
respectively. The hyperparameter $\alpha\geq 0$ is used to balance the two
scores. In practice, it first selects the top-$k$ most probable tokens
according to $P_{\theta}(\cdot|\mathbf{x},\mathbf{y}_{<t})$, denoted by
$\mathcal{V}^{k}$. The token in $\mathcal{V}^{k}$ with the largest
$\mathrm{FSD}$ score is chosen as the $t$-th token.

### Randomized FSD

is a randomized variant of FSD. Note that FSD is a deterministic decoding
methods and users may require a diverse set of outputs in many real-world
applications. Randomized FSD select between from FSD and sampling in the first
$T$ steps. In practice, we find that this strategy can produce diverse outputs
without compromising the quality.

### Temperature Sampling

is a decoding strategy to control the randomness in the sampling process.
Instead of directly sampling tokens from the predicted distribution,
temperature sampling introduces a hyperparameter “temperature” $\tau$ that is
used to adjust the skewness of the probability distribution:

$P(y|\mathbf{x},\mathbf{y}_{<t})=\frac{\exp(\mathbf{u}_{y}/\tau)}{\sum_{j}\exp(\mathbf{u}_{j}/\tau)}$

where $\mathbf{u}_{y}$ is the logit of the token $y$ before softmax.

### Typical Sampling

(Meister et al., 2023) sorts the vocabulary according to the differences
between distribution entropy and token probabilities. This process aims to
generate output sequences with information content closely aligned to the
model’s expected information content, i.e., the conditional entropy. The
candidate set $\mathcal{V}^{c}$ is a solution of the following problem:

$\begin{split}\min_{\mathcal{V}^{c}}\sum_{y\in\mathcal{V}^{c}}|H(Y_{t}|&\mathbf{x},\mathbf{y}_{<t})+\log
P(y|\mathbf{x},\mathbf{y}_{<t})|\\\
&\text{s.t.}\sum_{y\in\mathcal{V}^{c}}P(y|\mathbf{x},\mathbf{y}_{<t})\geq
p\end{split}$

### Mirostat Sampling

(Basu et al., 2021) directly control the perplexity rate of the generated
text. It firstly estimates the value of $s$ assuming words follow Zipf’s law
where $s$ is an exponent characterizing the distribution. Then it uses top-$k$
sampling to generate the new token where $k$ is a function of the estimated
$s$ and of the target perplexity $\tau$ of the output text.

### MinP Sampling

establishes a minimum probability threshold that a token must achieve to be
included in the sampling process. This minimum value is calculated as a
product of $\alpha$ and $p_{\text{max}}$, where $p_{\text{max}}$ is the top
token’s probability and $\alpha$ is a hyperparameter employed to manage the
diversity of the sampling.

### Tail Free Sampling (TFS)

(Bricken, 2019) sorts token probabilities in descending order and truncates
the tail of the distribution. Specifically, TFS calculates the first and
second derivatives of the sorted distribution. A threshold $z$ is used to
determine what part of the cumulative distribution of the second derivative
weights to define the “tail” of the distribution to be at. This is a
hyperparameter that can be tuned to reduce the impact of most unlikely tokens.

##  7 Grouped-query attention

Grouped-query attention (Ainslie et al., 2023) is proposed to reduce the
memory consumption of the KV cache and to increase inference speed. This
technique has been used in more and more LLMs, such as Llama2-70B, Mistral-7B
(Jiang et al., 2023), and Falcon-40B (Almazrouei et al., 2023). Inferflow
supports grouped-query attention as an atomic module. Grouped-query attention
is automatically enabled when the value of hyper-parameter decoder_kv_heads is
smaller than another hyper-parameter decoder_heads.

Multi-head attention (Vaswani et al., 2017) splits the input into multiple
heads, each responsible for attending to a different aspect of the input. The
formulation of multi-head attention can be expressed as follows:

$\text{MultiHead}(Q,K,V)=\text{Concat}(\text{head}_{1},\dots,\text{head}_{h})W^{O}$
(6)

where each head is computed as:

$\begin{split}\text{head}_{i}&=\text{Attention}(Q_{i},K_{i},V_{i})\\\
Q_{i}&=QW_{i}^{Q},K_{i}=KW_{i}^{K},V_{i}=VW_{i}^{V}\end{split}$ (7)

In this formulation, $Q_{i}$, $K_{i}$, and $V_{i}$ represent the query, key,
and value matrices for the $i$-th head, and $W_{i}^{Q},W_{i}^{K},W_{i}^{V}$
and $W^{o}$ are the projection matrix. The Attention function can be any
attention mechanism, such as scaled dot-product attention.

Despite their success, multi-head attention suffers from memory bandwidth in
loading keys and values. To reduce KV-cache, grouped-query attention
constructs each group key and value head by mean-pooling all the original
heads within that group. Assumed that we have $H$ attention heads with $G$
groups:

$\begin{split}\text{group}_{i}&=\text{Attention}(Q_{i},K^{j},V^{j})\\\
K^{j}&=\text{mean-pooling}(K^{j}_{1},\dots,K^{j}_{H/G})\\\ V^{j}&=\text{mean-
pooling}(V^{j}_{1},\dots,V^{j}_{H/G})\end{split}$ (8)

If the number of groups is equal to the number of heads($H=G$), the grouped-
query attention is reduced to multi-head attention.

Algorithm 1 Speculative Sampling

Input: number of lookahead step $K$, target sequence length $T$, target model
$p(x)$, draft model $q(x)$, prefix input: $x_{1},\dots,x_{t}$, whether accept
based on sampling is_top_sampling

Initialise $n\leftarrow t$.

while $n<T$ do

for $t=1:K$ do

Sample draft auto-regressively $\tilde{x}_{t}\sim
p(x|,x_{1},\dots,x_{n},\tilde{x}_{1},\dots,\tilde{x}_{t-1})$

end for

In parallel, compute $K+1$ sets of logits from drafts
$\tilde{x}_{1},\dots,\tilde{x}_{K}$ :  $q(x|,x_{1},\dots,x_{n}),\
q(x|,x_{1},\dots,x_{n},\tilde{x}_{1}),\ \dots,\
q(x|,x_{1},\dots,x_{n},\tilde{x}_{1},\dots,\tilde{x}_{K})$

for $t=1:K$ do

Sample $a\sim U[0,1]$ from a uniform distribution.

if $q(x|x_{1},\dots,x_{n+t-1})\ in\
\textsc{top\\_sampling}(p(x|x_{1},\dots,x_{n+t-1}))$ and is_top_sampling, then

Set $x_{n+t}\leftarrow\tilde{x}_{t}$ and $n\leftarrow n+1$.

elif
$a<\min\left(1,\frac{q(x|x_{1},\dots,x_{n+t-1})}{p(x|x_{1},\dots,x_{n+t-1})}\right)$,
then

Set $x_{n+t}\leftarrow\tilde{x}_{t}$ and $n\leftarrow n+1$.

else

sample
$x_{n+t}\sim(q(x|x_{1},\dots,x_{n+t-1})-p(x|x_{1},\dots,x_{n+t-1}))_{+}$ and
exit for loop.

end if

end for

If all tokens $x_{n+1},\dots,x_{n+K}$ are accepted, sample an extra token
$x_{n+K+1}\sim q(x|,x_{1},\dots,x_{n},x_{n+K})$ and set $n\leftarrow n+1$.

end while

##  8 Speculative Decoding

To further enhance inference speed, Inferflow will soon incorporates
speculative decoding (Chen et al., 2023; Leviathan et al., 2022), a technique
that expedites the inference process for a large target LLM
$p(\textperiodcentered|x)$ by utilizing token proposals from a small draft
model $q(\textperiodcentered|x)$. The concept involves the draft model
predicting the output $K$ steps ahead, while the target model dictates the
number of tokens to accept from those speculations.

As shown in Algorithm 19, the draft model decodes $K$ tokens in the regular
autoregressive fashion. We get the probability outputs of the target and draft
model on the new predicted sequence. We compare the target and draft model
probabilities to determine how many of the $K$ tokens we want to keep based on
some rejection criteria. If a token is rejected, we resample it using a
combination of the two distributions and do not accept any more tokens. If all
$K$ tokens are accepted, we can sample an additional final token from the
target model probability output.

The time complexity for the above algorithm is

$O(\frac{N}{r(K+1)}(t_{\text{draft}})K+t_{\text{target}})$ (9)

where $r$ is the acceptance rate indicating the percent of the draft tokens
are accepted, which is the key factor of the speed-up ratio. Chen et al.
(2023) accept the token by comparing a threshold $a\sim U[0,1]$ from a uniform
distribution and the probability
$\frac{q(x|x_{1},\dots,x_{n+t-1})}{p(x|x_{1},\dots,x_{n+t-1})})$. To further
increase the acceptance rate $r$, we implement a flexible accept schema by
consulting top-k / top-p decoding. If the draft token is present in the top-k
/ top-p pools, this token will be accepted by the target model.

##  9 Conclusion

We presented Inferflow, an efficient and highly configurable inference engine
for LLMs. This technical report provided an overview of the key features and
capabilities of Inferflow, briefly described the implementation of its key
modules.

## Acknowledgements

Inferflow is inspired by the awesome projects of
llama.cpp222https://github.com/ggerganov/llama.cpp and
llama2.c333https://github.com/karpathy/llama2.c. The CPU inference part of
Inferflow is based on the ggml library444https://github.com/ggerganov/ggml.
The FP16 data type in the CPU-only version of Inferflow is from the Half-
precision floating-point library555https://half.sourceforge.net/. We express
our sincere gratitude to the maintainers and implementers of these source
codes and tools.

## References

  * Ainslie et al. (2023) Joshua Ainslie, James Lee-Thorp, Michiel de Jong, Yury Zemlyanskiy, Federico Lebron, and Sumit Sanghai.  GQA: Training generalized multi-query transformer models from multi-head checkpoints.  In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_ , 2023.
  * Almazrouei et al. (2023) Ebtesam Almazrouei, Hamza Alobeidli, Abdulaziz Alshamsi, Alessandro Cappelli, Ruxandra Cojocaru, Merouane Debbah, Etienne Goffinet, Daniel Heslow, Julien Launay, Quentin Malartic, Badreddine Noune, Baptiste Pannier, and Guilherme Penedo.  Falcon-40B: an open large language model with state-of-the-art performance.  2023\.
  * Basu et al. (2021) Sourya Basu, Govardana Sachitanandam Ramachandran, Nitish Shirish Keskar, and Lav R. Varshney.  Mirostat: a neural text decoding algorithm that directly controls perplexity.  In _9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021_ , 2021.
  * Bricken (2019) Trenton Bricken.  https://www.trentonbricken.com/tail-free-sampling/, 2019.
  * Chen et al. (2023) Charlie Chen, Sebastian Borgeaud, Geoffrey Irving, Jean-Baptiste Lespiau, Laurent Sifre, and John Jumper.  Accelerating large language model decoding with speculative sampling, 2023.
  * (6) DeepSpeed-MII.  https://github.com/microsoft/deepspeed-mii.
  * Dettmers et al. (2022a) Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer.  GPT3.int8(): 8-bit matrix multiplication for transformers at scale.  In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), _Advances in Neural Information Processing Systems_ , 2022a.
  * Dettmers et al. (2022b) Tim Dettmers, Mike Lewis, Sam Shleifer, and Luke Zettlemoyer.  8-bit optimizers via block-wise quantization.  In _The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022_ , 2022b.
  * Devlin et al. (2018) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  BERT: pre-training of deep bidirectional transformers for language understanding.  _CoRR_ , abs/1810.04805, 2018.  URL http://arxiv.org/abs/1810.04805.
  * Fan et al. (2018) Angela Fan, Mike Lewis, and Yann Dauphin.  Hierarchical neural story generation.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , 2018.
  * Fan et al. (2020) Angela Fan, Shruti Bhosale, Holger Schwenk, Zhiyi Ma, Ahmed El-Kishky, Siddharth Goyal, Mandeep Baines, Onur Celebi, Guillaume Wenzek, Vishrav Chaudhary, Naman Goyal, Tom Birch, Vitaliy Liptchinsky, Sergey Edunov, Edouard Grave, Michael Auli, and Armand Joulin.  Beyond english-centric multilingual machine translation, 2020.
  * Holtzman et al. (2020) Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi.  The curious case of neural text degeneration.  In _8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020_ , 2020.
  * Jiang et al. (2023) Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed.  Mistral 7b, 2023.
  * Leviathan et al. (2022) Yaniv Leviathan, Matan Kalman, and Yossi Matias.  Fast inference from transformers via speculative decoding, 2022.
  * (15) llama.cpp.  https://github.com/ggerganov/llama.cpp.
  * Meister et al. (2023) Clara Meister, Tiago Pimentel, Gian Wiher, and Ryan Cotterell.  Locally typical sampling.  _Transactions of the Association for Computational Linguistics_ , 11, 2023.
  * OpenAI (2023) OpenAI.  Gpt-4 technical report.  _ArXiv preprint_ , abs/2303.08774, 2023.
  * (18) TensorRT-LLM.  https://github.com/nvidia/tensorrt-llm.
  * Touvron et al. (2023) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al.  Llama 2: Open foundation and fine-tuned chat models.  _ArXiv preprint_ , abs/2307.09288, 2023.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett (eds.), _Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA_ , 2017.
  * (21) vLLM.  https://github.com/vllm-project/vllm.
  * Wolf et al. (2020) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush.  Transformers: State-of-the-art natural language processing.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_ , 2020.
  * Yang et al. (2023) Haoran Yang, Deng Cai, Huayang Li, Wei Bi, Wai Lam, and Shuming Shi.  A frustratingly simple decoding method for neural text generation, 2023.
  * Yao et al. (2022) Zhewei Yao, Reza Yazdani Aminabadi, Minjia Zhang, Xiaoxia Wu, Conglong Li, and Yuxiong He.  Zeroquant: Efficient and affordable post-training quantization for large-scale transformers, 2022.
  * Yu et al. (2022) Gyeong-In Yu, Joo Seong Jeong, Geon-Woo Kim, Soojeong Kim, and Byung-Gon Chun.  Orca: A distributed serving system for Transformer-Based generative models.  In _16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22)_ , 2022.  ISBN 978-1-939133-28-1.