# Leaderless Byzantine Fault Tolerant Consensus

Jianyu Niu and Chen Feng  The University of British Columbia (Okanagan Campus)

##  I Introduction

Byzantine fault tolerant (BFT) consensus has recently gained much attention
because of its intriguing connection with blockchains. Several state-of-the-
art BFT consensus protocols have been proposed in the age of blockchains such
as Tendermint [5], Pala [9], Streamlet [8], HotStuff [23] and Fast-HotStuff
[17]. These protocols are all leader-based (i.e., protocols run in a series of
views, and each view has a delegated node called the leader to coordinate all
consensus decisions). To make progress, leader-based BFT protocols usually
rely on view synchronization, which is an ad-hoc way of rotating leader and
synchronizing nodes to the same view with the leader for enough overlap time.
However, many studies and system implementations show that existing methods of
view synchronization are complicated and bug-prone [15, 16, 19, 2]. In this
paper, we aim to design a leaderless Byzantine fault tolerant (LBFT) protocol,
in which nodes simply compete to propose blocks (containing a batch of
clients’ requests) without the need of explicit coordination through view
synchronization. LBFT also enjoys several other desirable features emphasized
recently by the research community, such as the chain structure, pipelining
techniques, and advanced cryptography [5, 6, 9, 23, 17]. With these efforts,
LBFT can achieve both good performance (e.g., $O(n)$ or $O(nlog(n))$ message
complexity) and prominent simplicity.

##  II System Model and Preliminaries

###  II-A System Model

We consider a system with $n$ nodes. We consider two roles of nodes in the
system: proposers, who propose blocks, and voters, who vote for received
blocks.111In a system, a single node may play both roles. The system consists
of $2m+1$ proposers and $3f+1$ voters.222Clearly, we have $n\geq 2m+1$, $n\geq
3f+1$, and $n\leq 2m+1+3f+1$. In particular, if every node plays both roles,
then we have $n=2m+1=3f+1$. At most $m$ (resp. $f$) proposers (resp. voters)
may be _Byzantine_ at any time. We assume a public-key infrastructure (PKI) of
voters; each voter has a pair of keys for signing messages. In addition, we
assume each proposer with identical ability to produce block proofs
(introduced shortly). For simplicity, all the Byzantine nodes (i.e., Byzantine
proposers and voters) are assumed to be controlled by a single adversary. The
adversary is computationally bounded and cannot (except for negligible
probability) forge honest nodes’ messages.

We assume that honest nodes are fully and reliably connected, i.e., every pair
of honest nodes is connected with an authenticated and reliable communication
link. We adopt the partial synchrony model of Dwork et al.[11], in which there
are a known delay bound $\Delta$ and an unknown Global Stabilization Time
(GST) such that all message transmissions between two honest nodes arrive
within the bound $\Delta$ after GST. In other words, the system is running in
_synchronous_ mode after GST and in _asynchronous_ mode if GST never occurs.

###  II-B Preliminaries

Block and Block Format. A block has a three-tuple $\left\langle
parent\\_qc,txs,\sigma\right\rangle$, where $parent\\_qc$ denotes a quorum
certificate of its previous block (also referred to as its parent block),
$txs$ is a collection of application-specific transactions, and $\sigma$ is
the block’s proof. (The quorum certificates and block proofs will be explained
shortly.) There exists a hard-coded genesis block $\mathcal{G}_{0}$. Every
block except the genesis block must specify its parent block by including a
quorum certificate of that block.

Block Tree. Blocks are chained by a sequence of certificate references
$\mathcal{G}_{0}\leftarrow B_{1}\leftarrow B_{2}\leftarrow\cdots$. A block’s
height is its distance from the genesis block. (The height of the genesis
block is $0$.) A chain’s length is defined as the number of blocks in the
chain excluding the genesis block. For example, a chain of length 2 contains
three blocks, namely, the genesis block, a height-1 block, and a height-2
block. A block $B$ is called a descendant of another block $B^{\prime}$ if
there is a chain from $B^{\prime}$ to $B$. Conversely, block $B^{\prime}$ is
an ancestor of block $B$. Two (distinct) blocks $B$ and $B^{\prime}$ conflict
if neither is a descendant of the other. Because of the possibility of
conflicting blocks, each node maintains a block tree (referred to as
$blockTree$) of received blocks.

Quorum Certificate. A block’s quorum certificate (QC) is proof that more than
$2f+1$ voters have signed this block. Here, a QC could be implemented as a
simple set of individual signatures or aggregated signatures [4]. We say a
block is certified when its QC is received, and certified blocks are ranked by
their heights. Each node keeps track of all signatures for all blocks and
keeps updating the certified blocks.333To make the signature collection more
efficient, nodes keep track of some “fresh” uncertified blocks instead of all
blocks. Besides, nodes can send signatures with some signed block information
(e.g., hash values or height) to accelerate other nodes’ verification
processes.

Block Proof. In LBFT, proposers participate in a lottery to win the right of
producing a block, and winning probabilities are proportional to their
abilities. Once winning the lottery, a proposer sends out the block together
with the proof, which is also called block proof. The lottery can be realized
by different cryptographic means to generate the randomness needed. For
example, the solution of a Hash puzzle is used in Proof-of-Work (PoW) lottery
[18], while verifiable delay functions (VDFs) and verifiable random functions
(VRFs) are widely used in Proof-of-Stake (PoS) lottery [13, 3]). Here, we
exclude the cryptographic realizations of the lottery since they have already
been carefully studied and implemented. Instead, we only provide the necessary
abstractions for us to construct LBFT and prove its safety and liveness (See
Sec. IV-B for details).

###  II-C Design Goals

We aim to design a leaderless BFT (LBFT) protocol in the partial synchronous
network model. From a very high level, all proposers in LBFT can compete to
propose blocks once they find some block proofs (at any time), and some of
these blocks will eventually be committed. Therefore, there is no explicit
leader (which is also called primary in PBFT [7]). Instead, multiple nodes can
propose blocks at any given interval (which is known as view [7], and phase
[11]). As a BFT consensus protocol, LBFT satisfies the following safety and
liveness guarantees:

  * •

Safety: No two honest nodes commit to two different blocks at the same height.

  * •

Liveness: If some honest proposer receives a collection of transactions $txs$,
these $txs$ will eventually be included in all honest nodes’ committed blocks.

In particular, according to the FLP theorem [12], LBFT protocol can always
keep safety regardless of the network condition but guarantee liveness only
when the network is synchronous.

##  III Protocol Design

###  III-A Overview

In this section, we will present an overview of the LBFT protocol.
Specifically, it contains three key components: block proposing, voting and
committing rules.444In blockchains, block proposers and voters may be two
different groups of nodes. Proposers compete to generate new blocks to extend
some certified blocks chosen by the proposing rule. Specifically, a proposer
is allowed to generate a new block once it finds an associated block proof.
Once producing a new block, the proposer immediately broadcasts the block to
other nodes.555The immediate publish of proposed blocks without additional
confirmation of leadership makes the system secure against the adaptive
adversary. A detailed discussion will be provided in Sec.V-B. After receiving
the new block, nodes first check whether this new block is valid. If yes,
nodes insert this block into their local $blockTree$s. Meanwhile, voters will
broadcast votes for this valid block if it satisfies the voting rule. When
nodes collect more than $2f+1$ votes for the block, this block will be
certified, and nodes will check whether there exist new committed blocks by
following the committing rule.

Unlike other leader-based BFT protocols, LBFT has no predefined views and
associated delegated leaders. Therefore, there may exist conflicts between
blocks (possibly produced by different proposers). For example, when one block
is being broadcast, some different blocks at the same height may be produced
simultaneously. As a result, nodes may observe multiple blocks (of the same
height), which leads to forks. Hence, nodes have to solve block conflicts and
guarantee that only one block is eventually committed at each height through
careful protocol design, which can be a challenging task. Below we will
explain how the aforementioned three rules address this issue.

###  III-B Protocol Details

In this section, we specify the block proposing, voting, and committing rules
of LBFT.

Proposing Rule. A proposer tries to produce a new block on top of the
_highest_ certified block that it has seen. In addition, if there are two
certified blocks with the same highest height, the proposer randomly chooses
one to extend.666This proposing rule is similar to the longest chain rule,
i.e., proposers extend the longest chain that they have seen. Here, we adopt
this version for simplicity. Note that a valid block should contain
transactions that are consistent with the transactions included in its
ancestor blocks, a valid QC, and block proof.

(a) A voter sends witVote for block $B_{k+1}$.

(b) A voter sends comVote for block $B_{k+1}$.

Figure 1: A voter sends different votes for block $B_{k+1}$ that extends the
highest certified block that it has seen.

Voting Rule. When a node receives a new block, it first does the validity
check for this block (including transactions, QC, and block proof). If this
block does not satisfy the validity, the node will abandon this block.
Otherwise, it will insert this block (not certified yet) into its $blockTree$.
After that, if this node is a voter, it will check whether to vote for this
block. Specifically, the voter votes for this block only if the block extends
the highest certified block that it has seen. Additionally, when voting, a
voter should choose the type of votes, which are mainly determined by the new
block’s parent block (denoted by _parent block_). That is, if there exists one
conflicting block of height no lower than that of the parent block in this
voter’s local $blockTree$s and the voter has voted for this conflicting block
before, the voter sends a _witness vote_ (short for _witVote_); Otherwise, the
voter sends a _commit vote_ (short for _comVote_).777For implementation, a
$1$-bit field can be included to denote the type of votes. In particular, when
a voter broadcasts its witVote, it must include at least one proof (e.g., hash
digest) of the conflicting blocks that have heights no lower than the parent
block. Note that for each new block, a voter only votes once, and a block is
certified if there are at least $2f+1$ votes for this block (regardless of the
type of votes).

To better understand the voting rule, we illustrate some simple cases in
Fig.1. In both cases, the new arriving block $B_{k+1}$ extends the highest
certified block $B_{k}$ that a voter has seen, so the voter will broadcast a
vote for it. But under different conditions, the types of votes are different.
In Fig.1(a), the voter sends a witVote for the block. This is because there
exists a conflicting block $B^{\prime}_{k}$ whose height is the same as block
$B_{k+1}$’s parent block $B_{k}$, and this voter has voted for block
$B^{\prime}_{k}$ before. By contrast, the voter broadcasts a comVote in
Fig.1(b).

Committing Rule. When having a new certified block, a node will check whether
there are new committed blocks. Specifically, if the node observes that the
new certified block’s QC contains no less than $2f+1$ comVotes, it will commit
all uncommitted ancestor blocks of this block, as shown in Fig. 2. Otherwise,
it will commit no new blocks. Note that nodes maintain a chain containing
committed blocks, and this chain is referred to as the _main_ chain.

Here, we provide some insights behind the voting rule design, especially on
role of the vote types. First, limiting voters to _only_ vote for blocks that
extend the longest certified chain can prevent the adversary from creating
blocks with low heights.888The longest certified chain extension requirement
can be relaxed, which will be discussed in our future work. Second, allowing a
voter to vote for multiple blocks at the same height (which all extend the
longest certified chain that it has seen) is for liveness. This is because, if
voters are required to only vote for one block at each height, they may vote
for different blocks with the same height, and then no block is certified with
$2f+1$ votes. As a result, no higher blocks can be produced without
referencing a previous certified block, i.e., no liveness. Third, it is
necessary to have two types of votes for safety. To see this, let us consider
a setting where there is only one type of votes. Then, under our previous
voting rule, nodes cannot commit blocks or guarantee that committed blocks are
safe. For example, when a node recovers from a network partition and receives
a chain from others, it cannot tell whether blocks in this chain are committed
as there possibly exists another forking chain. Hence, another type of vote is
required to carry the information that voters have confidence in some “pre-
committed” blocks (the only block at the height when voting). Furthermore, if
the majority of honest voters send out the promises that a parent block is the
only block at the height, and they refuse to vote for any blocks which do not
extend this block, this parent block will be committed and safe (i.e., no
conflicting committed blocks can be produced at its height).

Figure 2: A simple case of committing block. Block $B_{k}$ and other ancestor
blocks are committed when its child block $B_{k}$ is certified with at least
$2f+1$ comVotes.

###### Remark 1

In LBFT, blocks have three possible states: valid, certified and committed. A
valid block contains valid transactions, block proof and a QC of its parent
block; A certified block has been voted by at least $2f+1$ voters; A committed
block will be eventually committed by every honest node.

##  IV Safety and Liveness Proofs

In this section, we provide safety and liveness proofs for LBFT. We consider
no signature forgery or hash collision events as stated in Sec. II. All the
lemmas and theorems below hold except for a negligible probability (in the
security parameter) due to these events.

###  IV-A Safety Analysis

The safety property guarantees that there are no conflicting committed blocks
at the same height. It holds for the asynchronous network as well. Before our
analysis, we present two useful lemmas here.

###### Lemma 1

If two certified blocks $B$ and $B^{\prime}$ are observed by some honest
nodes, there must exist an honest voter who has voted for both of them.

###### Proof:

Let $\mathcal{S}$ (resp, $\mathcal{S}^{\prime}$) denote the set of honest
voters that have voted for block $B$ (resp. $B^{\prime}$). As there are at
most $f$ corrupted voters (who can voted twice for both block $B$ and
$B^{\prime}$), set $\mathcal{S}$ and $\mathcal{S}^{\prime}$ at least contain
$f+1$ honest voters. The intersection of these two sets (of honest voters) is
$|\mathcal{S}\cap\mathcal{S}^{\prime}|\geq(f+1)+(f+1)-(2f+1)=1>0$. This
implies that at least one honest voter must have voted for both block $B$ and
$B^{\prime}$. ∎

###### Lemma 2

Suppose that a certified chain adopted among honest nodes contains two blocks
$B_{k}$, $B_{k+1}$ at height $k$ and $k+1$, respectively, and block
$B_{k+1}$’s QC contains no less than $2f+1$ comVotes. Then, there cannot exist
a conflicting block $B^{\prime}_{k}\neq B_{k}$ such that $B^{\prime}_{k}$ is
also certified.

###### Proof:

Assume for contradiction that $B^{\prime}_{k}\neq B_{k}$ is certified in at
least one honest node, as shown in Fig. 3. By Lemma 1, there must exist one
honest voter who has voted for these two blocks. In particular, this voter has
sent a comVote for block $B_{k+1}$. Let $t_{0}$ (resp. $t_{1}$) denote the
time when this voter voted for block $B_{k+1}$ (resp, $B^{\prime}_{k}$). No
matter $t_{0}<t_{1}$ or $t_{0}>t_{1}$, we will drive a contradiction. (Here, a
voter is assumed to sequentially vote for blocks, so we do not consider the
case $t_{0}=t_{1}$.)

  * •

$t_{0}<t_{1}$. The honest voter first voted for block $B_{k+1}$. According to
the voting rule, when the voter receives block $B^{\prime}_{k}$, it would not
vote for it. This is because the voter at least has seen a block at height $k$
(i.e., block $B_{k}$), and obviously, $B^{\prime}_{k}$ does not extend the
highest certified block. This leads to a contradiction.

  * •

$t_{0}>t_{1}$. The honest voter first voted for block $B^{\prime}_{k}$. When
the voter later received $B_{k+1}$, it would send a witVote for it according
to the voting rule (because block $B^{\prime}_{k}$ has the same height as
block $B_{k+1}$’s parent block $B_{k}$, and this voter has voted it.). This
leads to a contradiction that this voter has sent a comVote for block
$B_{k+1}$.

∎

Figure 3: The safety violation cases (impossible).

Lemma 2 shows that once a block $B_{k+1}$ is certified with at least $2f+1$
comVotes, its parent block will be the only certified block at height $k$.
Furthermore, the parent block will be the only committed block at height $k$,
which guarantees the safety of committing this block. However, this is not
enough to prove the safety property. Recall that, according to the committing
rule, all uncommitted ancestor blocks of the parent block are also committed.
The following theorem will show that the safety property holds for all
committed blocks.

######  Theorem 1 (Safety)

If blocks $B_{k}$ and $B^{\prime}_{k}$ at height $k$ are committed by some
honest nodes, then $B_{k}=B^{\prime}_{k}$.

###### Proof:

Assume for contradiction that $B^{\prime}_{k}\neq B_{k}$ is committed by some
honest nodes. (Note that the two blocks could also be committed by a single
node.) First, according to the committing rule, there must exist two chain
branches: one chain branch containing block $B_{k}$ ended with two certified
blocks $B_{v}$ and $B_{v+1}$ (with heights $v$ and $v+1$, respectively), and
another one containing block $B^{\prime}_{k}$ ended with two certified blocks
$B^{\prime}_{m}$ and $B^{\prime}_{m+1}$ (with heights $m$ and $m+1$,
respectively) adopted by at least one honest node. Both QCs of blocks
$B_{v+1}$ and $B^{\prime}_{m+1}$ contain no less $2f+1$ comVotes. By the
committing rule, block $B_{v}$ together with its ancestor blocks, and block
$B^{\prime}_{m}$ together with its ancestor blocks are all committed.
Obviously, $v,m\geq k$. With loss of generality, assume that $v\leq m$. By
Lemma 1, there must exist one honest voter who has sent comVotes for both of
blocks $B_{v+1}$ and $B^{\prime}_{m+1}$. Let $t_{0}$ (resp. $t_{1}$) denote
the time when this voter voted for block $B_{v+1}$ (resp, $B^{\prime}_{m+1}$).
No matter $t_{0}<t_{1}$ or $t_{0}>t_{1}$, we will drive a contradiction.

  * •

$t_{0}<t_{1}$. The honest voter first voted for block $B_{v+1}$. By Lemma 2,
there cannot exist a certified block $B^{\prime}_{v}$. As $v\leq m$, there
cannot exist certified blocks $B^{\prime}_{m}$. Furthermore, this voter could
not vote for block $B^{\prime}_{m+1}$ without receiving its parent block. This
leads to a contradiction.

  * •

$t_{0}>t_{1}$. The honest voter first voted for block $B^{\prime}_{m+1}$. By
Lemma 2, there cannot exist a conflicting certified block at $m$. This implies
$k+1<m$. In addition, to vote for $B^{\prime}_{m+1}$, this voter at least has
received the parent block $B^{\prime}_{m}$. Obviously, block $B_{k+1}$ did not
extend the highest certified block (Block $B_{k}$ has a smaller height than
block $B^{\prime}_{m}$.), this voter would not vote for it. This leads to a
contradiction.

∎

###  IV-B Liveness Analysis

In this section, we prove liveness for LBFT. The liveness property guarantees
that honest blocks (including clients’ transactions) will be eventually
committed no matter what the adversary does. Specifically, the liveness
property holds only when the network is synchronous, i.e., after the GST. This
is because when the network is partitioned and delays can be arbitrarily long,
no certified blocks can be produced (without enough votes), and consequently,
no blocks can be committed.

####  IV-B1 Proof Overview

Let us first revisit the block committing case in Fig. 2. In this case, block
$B_{k}$ is the only certified block at height $k$, and is later extended by
some descendant blocks $B_{k+1}$ and $B^{\prime}_{k+1}$. During the voting
process for blocks $B_{k+1}$ and $B^{\prime}_{k+1}$, there are no other blocks
at height $k$ except for block $B_{k}$. Obviously, honest voters will send
comVotes for blocks $B_{k+1}$ and $B^{\prime}_{k+1}$ according to the voting
rule. Eventually,999When the network is synchronous, it takes at most
2$\Delta$ to certify a block by Lemma 4. when nodes receive at least $2f+1$
comVotes for either block $B_{k+1}$ or block $B^{\prime}_{k+1}$, they will
commit block $B_{k}$ together with its ancestor blocks by the committing rule.
Therefore, if there exists such a certified _honest_ block, and the block
remains unique at its height until it is later extended by a certified
descendent block (i.e., with at least $2f+1$ comVotes), the block together
with its ancestor blocks will be committed. The following analysis will show
that, with high probability, there always exist such certified _honest_ and
_unique_ blocks (referred to as certified unique blocks for short), which will
be committed no matter what the adversary does.

To this end, we first prove that the liveness property can hold from any time
$t_{0}$ after the GST under an ideal assumption that _the adversary does not
withhold any block at time $t_{0}$_. Specifically, we prove that there exist
certified unique blocks with high probability no matter what the adversary
does. The existence of certified unique blocks can guarantee that there are
honest committed blocks after $t_{0}$. We then relax this ideal assumption and
show that the adversary can only hide a finite number of blocks at time
$t_{0}$, which does not affect the existence of certified unique blocks.

####  IV-B2 Detailed Analysis

We first introduce a mathematical model of the generation process of block
proofs. For concreteness, we assume that LBFT adopts the well-known PoW
lottery to generate block proofs, which implies that the generation process
can be modeled as a Poison process with rate $\lambda$ [20, 22, 1,
10].101010LBFT can also adopt other cryptographic means to realize the
lottery. In [10], the authors provided a unified framework to model the block
producing process of PoW, Proof-of-Stake, and Chia Proof-of-Space lotteries.
The liveness analysis can be easily extended to different models. In
particular, the generation process at each proposer is a Poison process with
rate $\lambda/(2m+1)$ as proposers have identical capability in our system
model. Let $\beta$ ($\beta\leq m/(2m+1)$) be the fraction of proposers
corrupted by the adversary. Let $\lambda_{h}$ and $\lambda_{a}$ denote the
rate at which the honest proposers and the adversary produce block proofs,
respectively. We have $\lambda_{a}=\beta\lambda$ and
$\lambda_{h}=(1-\beta)\lambda$. As $\beta<1/2$, we have
$\lambda_{a}<\lambda_{h}$. We next present some useful lemmas.

###### Lemma 3

If an honest node has observed a certified block at time $t_{0}$
($t_{0}\geq\textsf{GST}$), then every honest node observes a chain ended with
this certified block by the time $t_{0}+\Delta$.

###### Proof:

First, by the strong $\Delta$-bounded assumption during periods of synchrony,
all honest nodes will have observed this block by $t_{0}+\Delta$. Second, for
a certified block, at least $f+1$ honest voters have signed for this block at
or before time $t_{0}$, and they must have observed a certified blockchain
extended by this block by time $t_{0}$. Hence, by the time $t_{0}+\Delta$,
every honest node will observe a chain ended with this certified block. ∎

###### Lemma 4

If an honest node has observed a block $B_{\ell}$ (not certified yet) that
extends a certified blockchain at time $t_{0}$ ($t_{0}\geq\textsf{GST}$), then
every honest proposer can start to produce blocks of height at least $\ell+1$
by the time $t_{0}+2\Delta$.

###### Proof:

First, this block $B_{\ell}$ will reach all the honest nodes by time
$t_{0}+\Delta$ by the $\Delta$-bounded assumption during periods of synchrony.
Next, by Lemma 3, its certified ancestor blocks will reach all the honest
nodes by time $t_{0}+\Delta$. Then, all honest voters will send votes for this
block (no matter what is the type) unless that some of the honest voters have
observed a certified chain with no less than $\ell$ length at time
$t_{0}+\Delta$. If all honest voters vote for this block at time
$t_{0}+\Delta$, then at least $2f+1$ votes will arrive at all honest nodes by
time $t_{0}+2\Delta$, and then all nodes will observe a certified chain ended
with this certified block. Otherwise, at least one honest voter has seen a
certified chain of no less than $\ell$ length at time $t_{0}+\Delta$, and by
Lemma 3, all nodes will observe this chain by time $t_{0}+2\Delta$. Therefore,
no matter in which cases, honest nodes will observe a certified chain with no
less than $\ell$ length at time $t_{0}+2\Delta$. Every honest proposer can
start to produce blocks with at least a height of $\ell+1$. ∎

Next, we introduce a special kind of honest blocks called _unique_ blocks. An
honest block produced at time $t$ is called a unique block if there is no
honest block produced in the previous and next $2\Delta$ time. We have the
following two lemmas for unique blocks.

###### Lemma 5

Suppose block $B$ is a unique block of height $\ell$, then no other honest
block can be of height $\ell$.

###### Proof:

Suppose for contradiction that two honest blocks $B$ and $B^{\prime}$ of
height $\ell$ are produced at time $t$ and $t^{\prime}$ respectively. Since no
other honest block is produced between time $t-2\Delta$ and $t+2\Delta$, we
have have $t^{\prime}\geq t+2\Delta$ or $t^{\prime}\leq t-2\Delta$. If
$t^{\prime}\geq t+2\Delta$, by Lemma 4, every honest node observes a certified
chain of length at least $\ell$ by time $t^{\prime}$, and meanwhile, proposers
produce blocks on top of it. Therefore, no honest proposer will produce a new
block of height $\ell$ after time $t^{\prime}$, leading to a contradiction.
Similarly, if $t^{\prime}\leq t-2\Delta$, every honest node observes a
certified chain of length at least $\ell$ before the time $t$ (or even
earlier), leading to a contradiction. ∎

The following lemma gives the bounds on the number of unique blocks in a time
interval of $t$.

###### Lemma 6

Let $\eta=e^{-2(1-\beta)\lambda\Delta}$. For any $0<\delta<1$, the number of
unique blocks produced in a time interval $t$ is at least
$(1+\delta)\eta^{2}(1-\beta)\lambda t$, except for probability
$e^{-\Omega\left(t\right)}$.

###### Proof:

Let $N_{H}(t)$ denote the number of honest block produced in a time interval
$t$, and note that
${\mathbb{E}}\left[N_{H}(t)\right]=\lambda_{h}t=(1-\beta)\lambda t$. Then, for
any $\delta_{1}\in(0,1)$, we have
$\Pr[N_{H}(t)\leq(1-\delta_{1})(1-\beta)\lambda t]\leq
e^{-\delta_{1}^{2}(1-\beta)\lambda t/2}=e^{-\Omega\left(t\right)}$ by Lemma
10. In particular, let $k=(1-\delta_{1})(1-\beta)\lambda t$ be an integer by
choosing a suitable $t$. We enumerate the first $k$ honest blocks produced
since the start of the time interval as blocks $1,2,...,k$. Without loss of
generality, we assume there is a block $0$ (resp. block $k+1$) that is the
last honest block produced before (resp. after) the interval.

Let $X_{i}$ denote the block interval between $(i-1)$-th and $i$-th block.
Recall the block production process of honest proposers is the Poisson process
with rate $(1-\beta)\lambda$. Hence, $X_{i}$ follows i.i.d. exponential
distribution with the same rate. Let $Y_{i}$ denote an indicator random
variable which equals one if the $i$-th block is unique block and zero
otherwise. Define $Y=\sum_{i=1}^{n}Y_{i}$. It is easy to see that the $i$-th
block is a unique block if $X_{i}\geq 2\Delta$ and $X_{i+1}\geq 2\Delta$.
Since $X_{i}$ and $X_{i+1}$ are independent, we have
$\Pr[Y_{i}=1]=\Pr[X_{i}\geq 2\Delta]\Pr[X_{i+1}\geq
2\Delta]=e^{-4(1-\beta)\lambda\Delta}=\eta^{2}$. Note that $Y_{i}$ and
$Y_{i+1}$ are not independent since they both depend on the event that
$X_{i+1}\geq 2\Delta$, but $Y_{i}$ and $Y_{i+2}$ are independent. Thus, $Y$
can be broken up into two summations of independent Boolean random variables
$Y=\sum_{odd}{Y_{i}}+\sum_{even}{Y_{i}}$. By Lemma 11, we have
$\Pr[Y\leq(1-\delta)\eta^{2}(1-\beta)\lambda t]\leq
e^{-\delta^{2}\eta^{2}(1-\beta)\lambda t/2}=e^{-\Omega\left(t\right)}$. ∎

The following lemma presents the condition to guarantee that there exist
certified unique blocks in the longest certified chain.

###### Lemma 7

Suppose $\eta^{2}(1-\beta)>(1+\delta)\beta$. In a time interval $t$, there
exist certified unique blocks in the longest chain except for
$e^{-\Omega\left(t\right)}$ probability.

###### Proof:

Let $N_{H}(t)$ (resp. $N_{A}(t)$) denote the unique (resp. adversarial) block
produced in the interval $t$. By Lemma 6, the number of unique blocks is
$N_{H}(t)>(1-\delta_{1})\eta^{2}(1-\beta)ft$ except for
$e^{-\Omega\left(t\right)}$ probability. Similarly, the expected time for the
adversary to produce a block is $\frac{1}{\lambda_{a}}$. In the best case, the
adversary can immediately transmit a block to honest voters, and then obtain
their votes without delay. This means that the adversary can immediately
produce the next block on its new block. Thus, during a time interval $t$, the
adversary can produce blocks at most $(1+\delta_{2})\lambda_{a}t$ except for
probability $e^{-\delta_{2}^{2}\lambda_{a}t/3}=e^{-\Omega(t)}$ by Lemma 10.

By setting $\delta_{1}=\delta_{2}=\delta/4$ and noticing
$\frac{1+\delta/4}{1-\delta/4}<1+\delta$, we have
$(1-\delta_{1})\eta^{2}(1-\beta)>(1+\delta_{2})\beta$. Therefore,
$N_{H}(t)>N_{A}(t)$ except for $e^{-\Omega\left(t\right)}$ probability. This
means that the adversary cannot create conflicting blocks to match each unique
block. In other words, there exist unique blocks that every honest voter will
vote for it (as it is the only block that extends the longest certified
chain), and then this block will be certified. ∎

######  Theorem 2 (Liveness)

Suppose $\eta^{2}(1-\beta)>(1+\delta)\beta$. In a time interval $t$, there
exist committed honest blocks in the main chain except for
$e^{-\Omega\left(t\right)}$ probability.

###### Proof:

By Lemma 7, there exists at least one certified unique block in the longest
chain except for $e^{-\Omega\left(t\right)}$ probability. Without loss of
generality, we assume that such a block $B$ is produced at time $t_{1}$. By
time $t_{1}+2\Delta$, all honest proposers can produce the next block on block
$B$ by Lemma 4. When a new block is proposed, honest voters will send comVotes
for this new block by the voting rule, and this new block will be certified at
each node within $2\Delta$ interval. By that time, the unique block $B_{l}$
together with its ancestor blocks will be committed by each nodes. ∎

Finally, we will relax the ideal assumption and allow the adversary to
withhold some blocks from honest proposers at time $t_{0}$ and then publish
them to match unique blocks produced by honest proposers. The following lemma
shows that the adversary cannot hide more than a finite number of blocks that
are higher than the highest block that any honest nodes know after
$\textsf{GST}+2\Delta$.

######  Lemma 8 (Bounded number of hidden blocks)

At any time $t_{0}\geq\textsf{GST}+2\Delta$, the number of unknown blocks to
any honest nodes is bounded with high probability.

###### Proof:

Without loss of generality, we assume block $B_{\ell}$ is the highest block
published by the adversary before the time $t_{0}-2\Delta$. Specifically, to
produce block $B_{\ell}$, all ancestor blocks of block $B_{\ell}$ have been
certified. This implies, at least $f+1$ honest nodes have seen and voted for
block $B_{\ell}$’s parent block before the time $t_{0}-2\Delta$. (In the ideal
case, we assume the adversary immediately collect all votes for block $B$’s
parent block, and then generate block $B$ without delay.) By Lemma 4, all
honest nodes would observe a certified chain with no less than $\ell$ length
by time $t_{0}$. Let $N_{A}(2\Delta)$ denote the number of produced
adversarial blocks between the time $t_{0}-2\Delta$ and time $t_{0}$. As the
block proof production process of the adversary is a Poisson process with rate
$\lambda_{h}$, the probability of generating $k$ new blocks that extend block
$B_{\ell}$ is $\frac{e^{-2\lambda_{a}\Delta}(2\lambda_{a}\Delta)^{K}}{k!}$,
which drops exponentially with the increase of $k$. ∎

This lemma implies that any honest nodes do not know a finite number of blocks
that are higher than the highest block that they have known. Therefore, by
increasing the interval $t$, LBFT can guarantee these exist certified unique
blocks, which will be committed with high probability. This establishes the
liveness of LBFT.

##  V Discussion

###  V-A Message Complexity

In this section, we briefly discuss how to reduce the message complexity. Let
us first revisit the message complexity of the current version of LBFT. When
receiving a new block (satisfying the voting rule), a voter will broadcast its
vote of this block to all nodes. Hence, the message complexity of broadcasting
all votes equals to the product of the number of voters and the number of
nodes, i.e., $O(n^{2})$ message complexity. When a subsequent proposer
produces a new block on this block, it has to include a QC with at least
$2f+1$ votes and broadcast the new block to all nodes, which also occurs
$O(n^{2})$ message complexity.

We can reduce the message complexity in the following ways. First, we can use
aggregated signatures [4] for the QC to reduce the message complexity of
broadcasting a block to $O(n)$.111111In the worst case, there are two types of
votes for a block. Hence, at most two aggregated signatures of votes are
needed. Second, we can change the message transmitting pattern to reduce the
message complexity. Specifically, we can use a gossip protocol or a leader-
based pattern. With a gossip protocol, the message complexity for proposers
(resp, voters) to transmit blocks (resp, votes) is $O(nlog(n))$. With a
leader-based pattern, the producer of a new block becomes a “leader”, and all
voters only send their votes for this new block to this leader. When
collecting enough votes (forming a certificate), the leader will send the QC
of its block to every nodes. In this way, the message complexity of votes and
blocks become $O(n)$. Note that changing the message transmitting pattern does
not affect the block proposing, voting, and committing rules.

###  V-B Adaptive Adversary

In LBFT, proposers participate in a lottery to win the right to produce a
block. Once winning a lottery, proposers immediately publish their blocks.
There is no further special role or actions after the block publishing is
performed. Therefore, the block proposing is unpredictable, which makes LBFT
secure against a fully adaptive adversary. By contrast, some leader-based BFT
systems require nodes to first participate in a lottery to win the leader
candidacy [14, 21]. After that, one candidate is eventually chosen as the
leader by some reductions or agreements, and the single leader is responsible
for coordinating the consensus decisions. However, the additional reductions
or agreements may expose these candidates’ identity, making it possible for a
fully adaptive adversary to corrupt them. Consequently, a corrupted leader
would hinder the progress.

##  VI Prior work

In this section, we review several state-of-the-art BFT protocols [5, 23, 9]
in the age of blockchains that are mostly related to LBFT. As we will see,
LBFT enjoys many desirable features achieved by these protocols.

Tendermint. Tendermint [5] features a continuous leader rotation (also called
the democracy-favoring leader rotation [9]) based on PBFT protocol.
Specifically, Tendermint embeds a round-change mechanism into the common-case
pattern, and the leader is re-elected from all the nodes by some desired
policy after every block, resulting in better leadership fairness.

Casper FFG. Buterin and Griffith [6] proposed a protocol called Casper FFG,
which works as an overlay atop NC to provide “finality gadget”. Casper FFG
applies an elegant pipelining idea to the classical BFT protocol, i.e., if
each block requires two rounds of voting, one can piggyback the second round
on the next block’s voting. This pipelining idea enables the system to have
one identical round (rather than multiple rounds with different
functionalities and names121212In PBFT, for committing one proposal, there are
two phases: prepare and commit phases, and each phase has different
functionalities.), and so significantly simplifies the protocol design.

Pala and Streamlet. Pala [9] is a simple BFT consensus protocol that adopts
the pipelining idea. However, for high throughput, it uses a stability-
favoring leader rotation policy. Based on this work, Chan et al. [8] proposed
Streamlet, which further simplifies the voting rule. Streamlet aims to provide
a unified, simple protocol for both teaching and implementation.

HotStuff. HotStuff proposed by Yin et al. [23] creatively adopts a three-phase
commit rule (rather than the two-phase commit rule used in Casper FFG, Pala,
and Streamlet) to enable the protocol to reach consensus at the pace of actual
network delay. In addition, HotStuff adopts the threshold signature to realize
linear message complexity, and can also be pipelined into a practical protocol
for building large-scale blockchains.

##  VII Conclusion

In the paper, we propose a leaderless BFT (LBFT) consensus protocol, which
does not require coordinators to make progress. Consequently, LBFT eliminates
the often complicated view synchronization, leading to better security and
simplicity. LBFT also enjoys many desirable features, such as the chain
structure, pipelining techniques, and advanced cryptography. All these efforts
make LBFT both practical (i.e., $O(n)$ or $O(nlog(n))$ message complexity
after optimizations), secure and simple to be implemented for large-scale
applications like blockchains.

## References

  * [1] Vivek Kumar Bagaria, Sreeram Kannan, David Tse, Giulia C. Fanti, and Pramod Viswanath.  Prism: Deconstructing the blockchain to approach physical limits.  In Proceedings of the 2019 ACM SIGSAC Conference on Computer and Communications Security, CCS 2019, London, UK, November 11-15, 2019., pages 585–602, 2019.
  * [2] Shehar Bano, Mathieu Baudet, Avery Ching, Andrey Chursin, George Danezis, François Garillot, Zekun Li, Dahlia Malkhi, Oded Naor, Dmitri Perelman, et al.  State machine replication in the Libra blockchain, May 2020.
  * [3] Iddo Bentov, Rafael Pass, and Elaine Shi.  Snow white: Provably secure proofs of stake.  IACR Cryptology ePrint Archive, 2016(919), 2016.
  * [4] Dan Boneh, Craig Gentry, Ben Lynn, and Hovav Shacham.  Aggregate and verifiably encrypted signatures from bilinear maps.  In Eli Biham, editor, Advances in Cryptology — EUROCRYPT 2003, pages 416–432, Berlin, Heidelberg, 2003. Springer Berlin Heidelberg.
  * [5] Ethan Buchman.  Tendermint: Byzantine fault tolerance in the age of blockchains.  M.Sc. Thesis, University of Guelph, Canada, Jun 2016.
  * [6] Vitalik Buterin and Virgil Griffith.  Casper the friendly finality gadget.  CoRR, abs/1710.09437, 2017.
  * [7] Miguel Castro and Barbara Liskov.  Practical Byzantine fault tolerance.  In Proceedings of the Third Symposium on Operating Systems Design and Implementation, OSDI ’99, pages 173–186, Berkeley, CA, USA, 1999\. USENIX Association.
  * [8] Benjamin Y. Chan and Elaine Shi.  Streamlet: Textbook streamlined blockchains.  IACR Cryptol. ePrint Arch., 2020:88, 2020.
  * [9] T-H Hubert Chan, Rafael Pass, and Elaine Shi.  Pala: A simple partially synchronous blockchain.  IACR Cryptology ePrint Archive, 2018:981, 2018.
  * [10] Amir Dembo, Sreeram Kannan, Ertem Nusret Tas, David Tse, Pramod Viswanath, Xuechao Wang, and Ofer Zeitouni.  Everything is a race and nakamoto always wins.  IACR Cryptol. ePrint Arch., 2020:601, 2020.
  * [11] Cynthia Dwork, Nancy Lynch, and Larry Stockmeyer.  Consensus in the presence of partial synchrony.  Journal of the ACM (JACM), 35(2):288–323, 1988.
  * [12] Michael J. Fischer, Nancy A. Lynch, and Michael S. Paterson.  Impossibility of distributed consensus with one faulty process.  In Proceedings of the 2nd ACM SIGACT-SIGMOD Symposium on Principles of Database Systems, PODS ’83, page 1–7, New York, NY, USA, 1983\.
  * [13] Yossi Gilad, Rotem Hemo, Silvio Micali, Georgios Vlachos, and Nickolai Zeldovich.  Algorand: Scaling Byzantine agreements for cryptocurrencies.  In Proceedings of the 26th Symposium on Operating Systems Principles, SOSP ’17, pages 51–68, New York, NY, USA, 2017. ACM.
  * [14] Eleftherios Kokoris Kogias, Philipp Jovanovic, Nicolas Gailly, Ismail Khoffi, Linus Gasser, and Bryan Ford.  Enhancing Bitcoin security and performance with strong consistency via collective signing.  In 25th USENIX Security Symposium (USENIX Security 16), pages 279–296, Austin, TX, 2016. USENIX Association.
  * [15] Leslie Lamport.  Brief announcement: Leaderless byzantine paxos.  In International Symposium on Distributed Computing, pages 141–142. Springer, 2011.
  * [16] Andrew Miller, Yu Xia, Kyle Croman, Elaine Shi, and Dawn Song.  The honey badger of BFT protocols.  In Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security, CCS ’16, pages 31–42, New York, NY, USA, 2016. ACM.
  * [17] Jianyu Niu Mohammad M. Jalalzai and Chen Feng.  Fast-hotstuff: A fast and resilient hotstuff protocol.  arXiv preprint arXiv:2010.11454, 2020.
  * [18] Satoshi Nakamoto.  Bitcoin: A peer-to-peer electronic cash system.  Working Paper, 2008.
  * [19] Oded Naor, Mathieu Baudet, Dahlia Malkhi, and Alexander Spiegelman.  Cogsworth: Byzantine view synchronization.  arXiv preprint arXiv:1909.05204, 2019.
  * [20] Jianyu Niu, Chen Feng, Hoang Dau, Yu-Chih Huang, and Jingge Zhu.  Analysis of Nakamoto consensus, revisited.  arXiv preprint arXiv:1910.08510, 2019.
  * [21] Rafael Pass and Elaine Shi.  Hybrid consensus: Efficient consensus in the permissionless model.  In 31st International Symposium on Distributed Computing (DISC 2017), volume 91 of Leibniz International Proceedings in Informatics (LIPIcs), pages 39:1–39:16, Dagstuhl, Germany, 2017.
  * [22] Ling Ren.  Analysis of Nakamoto consensus.  Cryptology ePrint Archive, Report 2019/943, 2019.
  * [23] Maofan Yin, Dahlia Malkhi, Michael K. Reiter, Guy Golan Gueta, and Ittai Abraham.  HotStuff: BFT consensus with linearity and responsiveness.  PODC ’19, pages 347–356, 2019.

###  -A Concentration Bounds

In this section, we provide the concentration bounds that we use in the
analysis. We denote the probability of an event $E$ by $\Pr[E]$ and the
expected value of a random variable $X$ by ${\mathbb{E}}\left[X\right]$.

######  Lemma 9 (Chernoff bound)

Let $X_{1}$, . . . ,$X_{n}$ be independent random variables, and let
$\mu:={\mathbb{E}}\left[\sum_{i=1}^{n}X_{i}\right]$. For $0<\delta<1$, we have
$\Pr[\sum_{i=1}^{n}X_{i}>(1+\delta)\mu]<e^{-\delta^{2}\mu/3}$ and
$\Pr[\sum_{i=1}^{n}X_{i}<(1-\delta)\mu]<e^{-\delta^{2}\mu/2}$

######  Lemma 10 (Chernoff bounds for Poisson random variables)

Let $X$ be a Poisson random variable with mean $\mu$. Then, for $0<\delta<1$,
$\Pr\left(X\geq(1+\delta)\mu\right)\leq e^{-\delta^{2}\mu/3}$ and
$\Pr\left(X\geq(1-\delta)\mu\right)\leq e^{-\delta^{2}\mu/2}$.

######  Lemma 11 (Chernoff bound for dependent random variables [20])

Let $T$ be a positive integer. Let $X^{(j)}=\sum_{i=0}^{n-1}X_{j+iT}$ be the
sum of $n$ independent indicator random variables and
$\mu_{j}={\mathbb{E}}\left[X^{(j)}\right]$ for $j\in\\{1,\ldots,T\\}$. Let
$X=X^{(1)}+\cdots+X^{(T)}$. Let $\mu=\min_{j}\\{\mu_{j}\\}$. Then, for
$0<\delta<1$, $\Pr\left[X\leq(1-\delta)\mu T\right]\leq e^{-\delta^{2}\mu/2}$.


# Vosh: Voxel-Mesh Hybrid Representation for Real-Time View Synthesis

Chen-hao Zhang  Beijing Institute of TechnologyBeijingChina
<EMAIL_ADDRESS>,  Yong-yang Zhou  Beijing Institute of
TechnologyBeijingChina<EMAIL_ADDRESS>and  Lei Zhang  Beijing Institute
of TechnologyBeijingChina<EMAIL_ADDRESS>
###### Abstract.

The neural radiance field (NeRF) has emerged as a prominent methodology for
synthesizing realistic images of novel views. While neural radiance
representations based on voxels or mesh individually offer distinct
advantages, excelling in either rendering quality or speed, each has
limitations in the other aspect. In response, we propose a pioneering hybrid
representation named _Vosh_ , seamlessly combining both voxel and mesh
components in hybrid rendering for view synthesis. Vosh is meticulously
crafted by optimizing the voxel grid of NeRF, strategically with selected
voxels replaced by mesh. Therefore, it excels in fast rendering scenes with
simple geometry and textures through its mesh component, while simultaneously
enabling high-quality rendering in intricate regions by leveraging voxel
component. The flexibility of Vosh is showcased through the ability to adjust
hybrid ratios, providing users the ability to control the balance between
rendering quality and speed based on flexible usage. Experimental results
demonstrates that our method achieves commendable trade-off between rendering
quality and speed, and notably has real-time performance on mobile devices.

Neural Radiance Field, View Synthesis, Real-Time Rendering.

††journal: TOG††copyright: none††ccs: Computing methodologies
Reconstruction††ccs: Computing methodologies Neural networks††ccs: Computing
methodologies Volumetric models Figure 1. Left: The hybrid representation
_Vosh_ , combines both voxel component (highlighted with pseudo colors) and
mesh component (indicated by gray triangles) in hybrid rendering. Right: The
proposed method facilitates real-time view synthesis on various mobile
devices, e.g., laptops and mobile phones.

##  1\. Introduction

The novel view synthesis based on the Neural Radiance Field (NeRF) leverages
neural networks to model three-dimensional scenes and generates images from
specified viewpoints (Mildenhall et al., 2021). In general, NeRF-based
approaches demonstrate proficiency in producing high-quality rendering for
novel views with sparse inputs, effectively handling materials with specular
and translucent properties. This versatility positions them for diverse
applications in autonomous navigation (Kwon et al., 2023), virtual
reality/augmented reality (Li et al., 2022a), and beyond (Sun et al., 2022b;
Jiang et al., 2023). Despite these merits, current methods face challenges in
achieving a balance between high-quality rendering and real-time rendering,
particularly when deployed on mobile devices.

The rendering speed of neural radiance fields is significantly influenced by
the chosen representation, with notable implications for sampling requirements
and query costs (Yu et al., 2021). Typically, the original NeRF uses deep
multilayer perceptrons (MLPs) to model neural radiance fields, relying on
dense sampling for querying to approximate integrals in volume rendering.
Unfortunately, this approach typically causes excessively long rendering
times. A substantial amount of effort has explored alternatives such as
implicit voxel representations (Sun et al., 2022a; Hedman et al., 2021; Liu et
al., 2020; Fridovich-Keil et al., 2022; Müller et al., 2022; Reiser et al.,
2021) and explicit mesh representations (Chen et al., 2023; Tang et al., 2023;
Rakotosaona et al., 2023; Yariv et al., 2023), aiming to exclude irrelevant
regions for computation. These methods reduce the number of sampling points
and employ smaller MLPs to expedite rendering. However, even though approaches
based on implicit voxels can achieve high-quality rendering in complex scenes,
the dense sampling in non-empty regions is prohibitively expensive.

Methods based on mesh representation have demonstrated significantly
accelerated rendering speeds, enabling real-time rendering on common mobile
devices. However, the challenge lies in the reconstruction of scene-level mesh
structure, particularly in areas with intricate geometry and textures (see
Figure 1), such as thin structures or undetermined regions like backgrounds
(Reiser et al., 2023). It is observed that meshes can match the rendering
quality akin to voxels in regions with simple geometry and textures, as well
as offering highly efficient rendering. Therefore, a promising solution
involves employing hybrid representations for reconstruction towards both
high-quality rendering and fast speed. Existing works on hybrid
representations (Wang et al., 2023; Guo et al., 2023), focus primarily on
object or proximal region reconstruction, thus limiting flexibility to
represent complex scenes and control over different parts of the hybrid
representation. Alternatively, we leverage the strengths of voxels and mesh,
to design a hybrid representation towards a better balance between quality and
speed, especially enabling high-quality rendering on mobile devices (see
Figure 1).

In summary, the main contribution of this paper is a new hybrid representation
namely _Vosh_ (VOxel-meSH), by combing voxels and mesh for NeRF-based real-
time view synthesis. It also allows for a dynamic control of the balance
between voxel and mesh components by voxel adjustment. Such flexibility
results in a versatile spectrum of NeRF representations, catering to the
varied computational capabilities that facilitate real-time view synthesis
across different devices. Experimental results exhibit the super performance
of our method compared to state-of-the-art methods.

Figure 2. An overview of the proposed methodology. The training phase starts
from grid training for obtaining initial voxels. Then, a portion of voxels
becomes mesh in the voxels to mesh conversion. Subsequently, the combination
of voxels and mesh are optimized through hybrid rendering and voxels
adjustment to obtain the final hybrid representation Vosh. The inference phase
realizes real-time hybrid rendering with Vosh even on mobile phones.

##  2\. Related Work

There are numerous published approaches to view synthesis and NeRF, and we
refer interested readers to (Tewari et al., 2022; Gao et al., 2022) for a
comprehensive overview. In this context, we focus on the real-time view
synthesis and NeRF representations most related to our method.

###  2.1. Real-time view synthesis

NeRF (Mildenhall et al., 2021) as a well-known technology for view synthesis,
has been the focus of numerous endeavors aimed at enhancing its rendering
speed while maintaining high-quality rendering. SNeRG (Hedman et al., 2021)
achieves fast rendering without CUDA, using sparse voxel representations and a
compact neural network for deferred rendering. Plenoxels (Fridovich-Keil et
al., 2022) and PlenOctrees (Yu et al., 2021) expedite ray marching with sparse
voxel representation, refining voxels near object surfaces. MERF (Reiser et
al., 2023) enhances rendering speed and spatial occupancy by replacing pure
voxel grids with a low-resolution voxel grid and a high-resolution triplane
projection representation (Chan et al., 2022). These methods mostly excel on
high-end GPUs, whereas real-time rendering on mobile devices still remains a
challenge.

MobileNeRF (Chen et al., 2023) achieves fast rendering by constraining NeRF to
triangles and using rasterization. NeRF2Mesh (Tang et al., 2023) uses NeRF to
initialize geometric structures, iteratively refining mesh attributes to
improve rendering quality. BakedSDF (Yariv et al., 2023) optimizes a blended
neural signed distance function representation, enabling real-time rendering
of complex scenes on a high-quality mesh. While mesh-based methods offer fast
rendering, they may suffer quality loss compared to voxel-based methods.

3D Gaussian Splatting (Kerbl et al., 2023) creates a 3D Gaussian
representation around each point starting from a point cloud model, utilizing
the Splatting method for real-time rendering of high-resolution scenes. But it
suffers elongated and popping artifacts, and high memory usage for deployment
on mobile phones. In contrast, our hybrid representation combines the
strengths of voxel and mesh representations, allowing high-quality rendering
of complex scenes on mobile devices.

###  2.2. Hybrid Scene Representation

Combining explicit mesh representation with implicit fields introduces
functionalities such as NeRF editing and rendering acceleration. NeuMesh (Yang
et al., 2022) encodes a neural implicit field with disentangled geometry and
texture on mesh vertices, enabling a range of neural rendering-based editing
operations. EyeNeRF (Li et al., 2022b) integrates an explicit surface with an
implicit volume representation to model the human eye. Mesh-Aware-RF (Qiao et
al., 2023) embeds a known polygonal mesh into NeRF, employing ray tracing and
ray marching alternately to update radiance along rays.

The hybrid representation like Adaptive Shells (Wang et al., 2023) and VMesh
(Guo et al., 2023) has been used in view synthesis. Adaptive Shells confines
volume rendering to a narrow band around the object for effective NeRF
rendering. VMesh, closely related to our work, uses a textured mesh and
auxiliary sparse voxels but differs by employing SDF field and volume density
field to simultaneously learn the entire scene. However, VMesh relies on
gradient descent optimization that lacks controllability and flexibility.
Indeed, VMesh mainly focuses on rendering individual objects or bounded
scenes, while the proposed method in this paper can handle more scenes, such
as unbounded outdoor scenes.

##  3\. Method Overview

We start with an overview of our methodology to establish a context for the
core algorithms described in the next section. As depicted in Figure 2, our
method contains two key phases: the training phase for constructing a hybrid
representation and the inference phase for view synthesis rendering. In the
training phase, a set of multi-view images is employed to train a voxel-mesh
hybrid representation, and the associated assets are exported. In the
inference phase, our method achieves real-time rendering based on the baked
hybrid representation assets.

Concretely, the training phase consists of three stages: grid training, voxels
to mesh conversion and optimization. In the grid training stage, we train a
voxel grid model through volume rendering. Then, leveraging the trained voxel
grid, we extract an explicit surface and train the surface using
differentiable surface rendering for both appearance and geometry
optimization. This converts the initial voxel grid into a combination of voxel
and mesh components. Finally, the voxels and mesh are further optimized
through a training process involving hybrid rendering and voxels adjustment to
obtain the desired hybrid representation. Once training is complete, we export
assets containing sparse voxels and textured meshes. Subsequently, real-time
view synthesis on mobile devices is implemented using WebGL. Next, we
elaborate the algorithm details in the training and inference phases.

##  4\. Hybrid Representation construction

The construction of a hybrid representation in the training phase commences
with the initialization of a volume comprising solely of voxels. This volume
is then subject to a conversion process wherein a portion of the voxels is
replaced with an explicit mesh according to rendering quality on par with the
original voxel representation.

###  4.1. Grid training for initial voxels

Initially, the entire scene is reconstructed using a high-resolution voxel
grid. It retains sufficient geometric and appearance information essential for
subsequent mesh extraction and hybrid representation. Here, we employ SNeRG++
(an improved version of SNeRG (Hedman et al., 2021) in MERF (Reiser et al.,
2023)) as the benchmark for our voxel grid, due to its capability to
effectively model unbounded scenes and generate real-time rendering assets.

Concretely, SNeRG++ is built upon the volume rendering principles of vanilla
NeRF, which involves sampling along the direction of a ray originating from
the ray origin $o$ along the direction $d$ to obtain sample points
$\mathrm{\mathbf{x}}_{i}=o+t_{i}\mathrm{\mathbf{d}}$. Based on the spatial
position of these sample points, it uses the neural radiance field to
calculate volumetric density $\sigma_{i}$, resulting in rendering weights
$w_{i}$:

(1)
$w_{i}=\alpha_{i}T_{i},\;\;\;\;\;\;\;\;T_{i}=\prod^{i-1}_{j=1}(1-\alpha_{j}),\;\;\;\;\;\;\;\;\alpha_{i}=1-e^{-\sigma_{i}\delta_{i}},$

where $T_{i}$, $\alpha_{i}$ and $\delta_{i}$ represent the transmittance,
opacity and spacing of sampling point $i$, respectively. SNeRG++ adopts the
concept of deferred rendering, decomposing the appearance information of the
neural radiance field into diffuse color $\mathrm{\mathbf{c}}_{d,i}$ and view-
dependent features $f_{i}$. By cumulatively sampling the appearance
information along the ray, as shown in Figure 4a, we obtain the diffuse color
$C_{d}$ and view-dependent features $F$ of the ray as

(2)
$\mathrm{\mathbf{C}}_{d}=\sum_{i}w_{i}\mathrm{\mathbf{c}}_{d,i},\;\;\;\;\;\;\;\;\mathrm{\mathbf{F}}=\sum_{i}w_{i}\mathrm{\mathbf{f}}_{i}.$

Finally, the view-dependent color is calculated by inputting the diffuse color
$\mathrm{\mathbf{C}}_{d}$, view-dependent features $F$, and the ray direction
$d$ into a miniature MLP. The result is then added to the diffuse color
$\mathrm{\mathbf{C}}_{d}$ to obtain the final color $C$ of the ray.

Besides, in order to enhance the modeling of regions situated far from the
center within unbounded scenes and facilitate efficient ray marching
calculations, SNeRG++ incorporates a contraction function founded on piecewise
projection. This contraction function is defined as follows:

(3) $\text{contract(x)}_{j}=\begin{cases}x_{j},&\text{if }||x||_{\infty}\leq
1\\\ \frac{x_{j}}{||x||_{\infty}},&\text{if }x_{j}\neq||x||_{\infty}>1\\\
(2-\frac{1}{|x_{j}|})\frac{x_{j}}{|x_{j}|},&\text{if
}x_{j}=||x||_{\infty}>1\\\ \end{cases},$

where $||\cdot||_{\infty}$ denotes the infinity norm, defined as
$||x||_{\infty}=\text{max}_{j}|x_{j}|$.

###  4.2. Voxels to mesh conversion

Based on the above high-resolution voxel grid, we employ Marching Cubes
(Lorensen and Cline, 1998) to extract an explicit mesh. However, surfaces
derived solely from density information in volume rendering often exhibit
roughness and are unsuitable for direct use in hybrid representation. Inspired
by NeRF2Mesh (Tang et al., 2023), we adopt a synergistic approach involving
differentiable rendering and surface operations to refine the explicit mesh.

The objective is to convert a subset of voxels that demonstrate similar
rendering quality in both voxel and explicit mesh representations into a
refined mesh. Typically, these converted voxels correspond to regions in the
scene characterized by uncomplicated geometry and texture. The surface
training specifically entails the coarse selection of voxels and subsequent
surface refinement using differentiable rendering techniques.

Coarse selection of voxels. By analyzing the rendering results of NeRF2Mesh
(Tang et al., 2023) (Figure 5) and BakedSDF (Yariv et al., 2023), we found
that using mesh representation to reconstruct the unbounded background region
far from the camera presents a formidable challenge. Achieving rendering
quality comparable to voxels in this remote area is inherently demanding.
Moreover, influenced by the contraction function, triangles extracted from
voxels in the contraction region tend to enlarge and may break apart as they
extend away from the camera. The presence of large triangles can adversely
affect rendering quality, particularly on WebGL platforms, while introducing
subdivision for these large triangles to enhance rendering quality incurs
additional storage overhead.

Thus, we only convert voxels with surfaces located in the region near the
camera into mesh. Criteria such as edge length and the number of triangles in
isolated patches guide the decision on which explicit surfaces to retain. By
default, we preserve the cube area with $90\%$ of the length of the bounding
box in outdoor scenes. This conversion strategy, depicted in Figure 3,
contrasts with indiscriminate conversion of voxels containing implicit
surfaces into triangular meshes, which results in diminished rendering
quality. So these explicit surfaces are further refined in the next
differentiable appearance optimization.

Figure 3. The voxels to mesh conversion without (w/o) coarse selection of
voxels obtains worse rendering results than with (w/) coarse selection, as
shown by the diffuse of Vosh for voxel and mesh components respectively.

Differentiable surface refinement. We perform differentiable rasterization
operations on the mesh to obtain intersections between rays and surfaces.
Appearance information is obtained by querying the voxel representation based
on the corresponding spatial positions. We also employ nvdiffrast (Laine et
al., 2020) for differentiable rendering to optimize the appearance information
of the mesh, while simultaneously tracking the rendering error corresponding
to each face for surface optimization. It is worth noting that we do not
enforce the mesh to represent the entire scene. Therefore, during
differentiable rendering, we only calculate the loss for pixels corresponding
to rays that intersect with the surface. We also perform face merging and mid-
point subdivision operations based on the normal change of the faces and
rendering errors. In contrast to NeRF2Mesh (Tang et al., 2023), which solely
relies on rendering errors for operations, we additionally utilize the local
normal change rate of faces as an optimization criterion. This enhances face
density in regions with complex geometry and reduces storage overhead for
faces in areas with simple geometry.

Following the conversion from voxels to mesh, we have obtained an initial
hybrid representation that includes both voxels and mesh. Nevertheless,
recognizing significant potential for optimization in rendering quality and
volume occupancy for both representations, we are committed to further
refining the hybrid representation through further fine-tuning.

Figure 4. The proposed hybrid rendering (c) integrates volume rendering (a)
and surface rendering (b) for the Vosh.

###  4.3. Optimization

After the mesh conversion and optimization stage, we obtain an explicit
surface suitable for representing regions in the scene with simple geometry
and textures, as well as a optimized high-resolution voxel grid. By combining
these two components, we can obtain a coarse hybrid representation with a
significant amount of redundancy between the components. This may result in
low rendering efficiency and potential rendering conflicts. So we further
perform hybrid rendering optimization and voxel adjustment based on the
components for a more compact and efficient representation.

Compared to volume rendering, surface rendering only considers the
intersection points between rays and surface, significantly reducing the time
cost associated with ray marching. This inspires us to confine the ray
marching in volume rendering to occur only before the rays hit the surface.
Simultaneously, performing volume rendering between the viewpoint and the
surface also provides possibilities for enhancing surface rendering quality.
Therefore, we straightforwardly combine volume rendering and surface
rendering, treating the surface points obtained through rasterization as the
final points reached by ray marching in volume rendering. The hybrid rendering
can be expressed by the following three equations:

(4)
$\mathrm{\mathbf{C}}_{vosh}=\mathrm{\mathbf{C}}_{vosh,d}+\text{MLP}(\mathrm{\mathbf{C}}_{vosh,d},\mathrm{\mathbf{F}_{vosh},\mathrm{\mathbf{d}}}),$

(5)
$\mathrm{\mathbf{F}}_{vosh}=\sum_{i}w_{i}\mathrm{\mathbf{f}}+w_{m}\mathrm{\mathbf{f}}_{m},$

(6)
$\mathrm{\mathbf{C}}_{vosh,d}=\sum_{i}w_{i}\mathrm{\mathbf{c}}_{d,i}+w_{m}\mathrm{\mathbf{c}}_{d,m},$

where $\mathrm{\mathbf{C}}_{vosh}$, $\mathrm{\mathbf{C}}_{vosh,d}$ and
$\mathrm{\mathbf{F}}_{vosh}$ represent the color, diffuse color, and view-
dependent features after hybrid rendering, respectively.
$w_{m}=(1-\sum_{i}w_{i})$ represents the rendering weight for the mesh, which
is the remaining portion of the cumulative voxel weights.
$\mathrm{\mathbf{c}}_{d,m}$ and $\mathrm{\mathbf{f}}_{m}$ represent the
diffuse color and view-dependent features at the intersection point where the
ray hits the surface. Three different rendering methods are shown in Figure 4.

Based on the optimized mesh, we attempt to dynamically adjust the voxel grid,
forming a high-quality hybrid representation. Since the trained mesh already
possesses the capability to represent regions with simple geometry and
textures, we need to consider how to simplify the voxel representation for the
entire scene. We focus on retaining voxels in regions with complex geometry
and textures to better complement the rendering of the hybrid representation
with the mesh. Inspired by PlenOctrees (Yu et al., 2021), we propose a voxel
adjustment loss based on surface mesh weights. Specifically, we calculate the
rendering weight of mesh $w_{m}$ for rays hitting the surface and force it
close to 1. The implementation is as follows:

(7)
$\mathcal{L}_{voxel}=\lambda_{voxel}\frac{1}{K}\sum_{k}(1-\text{exp}(w_{m}-1)),$

where $\lambda_{voxel}$ represents the weight of the voxel adjustment loss,
and $K$ is the number of samples taken in the rays hitting the surface.

The number of retained voxels is an important factor in determining the
balance between rendering quality and speed. Concretely, retaining more voxels
can enhance rendering quality but slow down rendering speed, while retaining
fewer voxels can accelerate rendering speed but compromise rendering quality.
Setting different voxel adjustment weights to limit the volumetric density
during ray marching is one of the methods to achieve different trade-offs
between speed and quality. Additionally, voxels and mesh may overlap at the
same spatial location, potentially affecting rendering quality and the volume
of exported assets. To address this issue, we first calculate the occupancy
grid of the meshes under the training poses. During voxel adjustment, we
ignore the voxels in the mesh-occupied grid. We also adjust the ratio of
sparse voxels in the hybrid representation by setting different resolutions
$r_{mesh}$ for the mesh-occupancy grid.

Table 1. Quantitative evaluation of the proposed method and SOTA methods with 3 outdoor scenes and 4 indoor scenes from Mip-NeRF 360 dataset (Barron et al., 2022). The methods that can render in real-time on mobile phones are highlighted.  |  | Outdoor | Indoor | GTX 1660Ti | iPhone 14 Pro
---|---|---|---|---|---
Methods | Representations | PSNR ↑ | SSIM ↑ | LPIPS ↓ | PSNR ↑ | SSIM ↑ | LPIPS ↓ | FPS @ 1920*1080 | FPS @ 344*707
NeRF | MLP | 21.46 | 0.458 | 0.515 | 26.84 | 0.790 | 0.370 | - | -
Mip-NeRF 360 | MLP | 25.92 | 0.747 | 0.244 | 31.72 | 0.917 | 0.179 | - | -
Instant-NGP | Voxel | 23.90 | 0.648 | 0.369 | 29.47 | 0.877 | 0.273 | - | -
SNeRG++1024 | Voxel | 24.11 | 0.625 | 0.377 | 27.32 | 0.802 | 0.276 | 9.0 | -
MERF | Voxel + Plane | 24.41 | 0.678 | 0.298 | 27.80 | 0.855 | 0.271 | 9.7 | 25.3
BakedSDF | Mesh | 23.40 | 0.577 | 0.351 | 27.20 | 0.845 | 0.300 | 131.0 | -
MobileNeRF | Mesh | 22.90 | 0.524 | 0.463 | 25.75 | 0.757 | 0.453 | 92.7 | 153.3
NeRF2Mesh | Mesh | 22.74 | 0.497 | 0.463 | 25.20 | 0.721 | 0.340 | 231.0 | 210.0
Vosh-Light | Voxel + Mesh | 23.63 | 0.586 | 0.392 | 27.02 | 0.792 | 0.294 | 79.7 | 70.0
Vosh-Base | Voxel + Mesh | 23.74 | 0.592 | 0.389 | 27.07 | 0.791 | 0.291 | 38.0 | 33.7

##  5\. Real-Time Rendering

The real-time rendering based on Vosh is realized as a JavaScript 3D web
application, leveraging the capabilities of the MERF real-time renderer. The
rendering process is also hybrid and structured into two passes: the
rasterization pass and the ray marching pass. Both passes are employed to make
rendering result consistency between training on the GPU and inference on the
web.

In the rasterization pass, meshes are uniformly shaded. The fragment shader
takes charge of sampling appearance textures, incorporating diffuse color and
view-dependent feature. The output of the rasterization pass includes
corresponding diffuse, view-dependent features and depth map. These output is
then used in the subsequent ray marching pass.

In the ray marching pass, we also employ a multi-resolution hierarchical
structure for the occupancy grid introduced in SNeRG (Hedman et al., 2021) to
mark the position of occupied voxels. This structure creates binary mask
images in multiple resolution levels through max-pooling. We use the multi-
resolution occupancy grid to perform spatial skips during ray marching.
Simultaneously, the depth map obtained in the rasterization pass further
reduces the sampling distance.

For each spatial coordinate after ray marching, we compare the transformed
depth with the depth map from the rasterization pass to determine whether the
ray has intersected the surface position. If the ray reaches the surface of
the mesh, the marching stops, and the accumulated result of ray marching is
weightedly combined with the appearance result from the rasterization pass.
Leveraging the surface depth information from the rasterization pass proves
instrumental in significantly diminishing the number of required sampling
points during ray marching, thereby effectively reducing the computational
cost associated with volume rendering.

##  6\. Experiments

We have implemented real-time view synthesis using the proposed hybrid
representation Vosh, and carried out comprehensive evaluations encompassing
both rendering quality and speed. We also compare our method with state-of-
the-art (SOTA) methods, like offline classical methods (NeRF (Mildenhall et
al., 2021), Mip-NeRF 360 (Barron et al., 2022), Instant-NGP (Müller et al.,
2022)) and real-time methods (Mobile-Nerf (Chen et al., 2023), NeRF2Mesh (Tang
et al., 2023), BakedSDF (Yariv et al., 2023), MERF (Reiser et al., 2023),
SNeRG++ (Hedman et al., 2021)). As done by SOTA methods, we used the
challenging unbounded dataset Mip-NeRF 360 (Barron et al., 2022) for
evaluation. Next, we elaborate the details of the experimental results.

Figure 5. The rendering results and zoomed-in images obtained by our method,
as well as some SOTA methods based on voxels or mesh representation.

###  6.1. Rendering quality comparison

We conduct both quantitative and qualitative evaluation of rendering quality
for view synthesis obtained by the proposed method and SOTA methods.
Quantitative evaluation uses the metrics of PSNR, SSIM (Wang et al., 2004) and
LPIPS (Zhang et al., 2018) for the assessment of rendering quality. Table 1
shows the statistics of the quantitative evaluation of different methods.

Here, we classify these methods based on their ability to achieve real-time
rendering on mobile devices like laptops and mobile phones. Methods such as
SNeRG, MERF, and BakedSDF can only attain real-time rendering on select
laptops with robust GPUs. It can be seen there exists an obvious gap between
representations based on voxels and meshes. While BakedSDF narrows the quality
gap with its millions of triangles, its substantial memory consumption makes
it impractical for deployment on mobile phones.

Vosh demonstrates superior rendering quality in real-time rendering on mobile
devices, even surpassing some methods that cannot achieve real-time rendering
on mobile devices. For the outdoor scenes, Vosh enhances rendering quality
over mesh-based methods by leveraging sparse voxels through the use of a
contraction function (see Figure 5, 6 and 7). However, for the indoor scenes,
methods have limitations in the representational capacity of tiny view-
dependent MLPs. While Vosh achieves real-time rendering on mobile devices by
employing a tiny view-dependent MLP, it has slightly lower rendering quality
in indoor scenes compared to MLP-based methods.

We provide a qualitative comparison with SOTA methods based on voxels (Reiser
et al., 2023) and mesh (Tang et al., 2023) on the examples in Figure 5. We
select one distant view and one close-up view for local magnification for
better exhibition. The mesh-based methods often lack the ability to represent
background regions far from the camera location, whereas voxel-based methods
necessitate higher computational and storage costs to achieve enhanced
rendering quality. Contrarily, our method improves rendering speed while
essentially maintaining the rendering quality comparable to that of voxel-
based methods. We refer the reader to the companion video for visual
demonstrations of the results.

Comparison with VMesh. It is worth noting that VMesh (Guo et al., 2023) is
also built on the hybrid representation of volume and mesh, but primarily
designed for rendering individual objects or scenes with boundaries, posing
challenges in handling unbounded scenes like the examples in Figure 5 and 7.
Nonetheless, our method produces better rendering results than VMesh, even for
the synthetic bounded object as shown in Figure 6.

Figure 6. The rendering results obtained by ours and VMesh (Guo et al., 2023).

###  6.2. Rendering speed comparison

We evaluated the performance of the methods applicable for rendering on mobile
devices, including SNeRG++ (Hedman et al., 2021), MERF (Reiser et al., 2023),
BakedSDF (Yariv et al., 2023), MobileNeRF (Chen et al., 2023), and NeRF2Mesh
(Tang et al., 2023). For SNeRG++ and MERF, we follow their configurations with
progressive upsampling turned off. We conduct testing on the iPhone 14 Pro
equipped with the A16 chip with a resolution of 344*707 and on the Legion
Y7000P 2019, which features NVIDIA GeForce GTX 1660Ti (Mobile version, 80W)
and operates with a resolution of 1920*1080.

Rendering speed tests for all comparative methods were conducted on the Mip-
Nerf 360 dataset (Barron et al., 2022) in three outdoor scenes, using
identical device settings, resolutions, and viewpoints. As shown on the right
side of Table 1, our method can run with the real-time frame rate (¿30 FPS) on
both iPhone 14 Pro and GTX 1660Ti. Particularly on the 1660Ti, our method
demonstrates a considerable speed improvement compared to MERF and SNeRG. The
latter two methods experience slow rendering speeds due to the necessity to
sample each alive voxel traversed by every ray. BakedSDF is difficult to
render on mobile phones and laptops with limited VRAM due to its substantial
assets. MobileNeRF and NeRF2Mesh achieve rendering speeds that reach the
maximum refresh rates of their respective devices (60Hz for Safari on iPhone
14 Pro and 144Hz for Legion Y7000P 2019). We make statistics on their
potential rendering speeds using overlapping multiple passes.

###  6.3. Ablation Study

We conduct ablation experiments on the voxels to mesh conversion and voxel
adjustment parts of the hybrid representation in Table 2. The assessment
involves a quantitative comparison, considering both rendering quality and
speed.

The ‘full bound’ row refers to the results of the variant that do not use
coarse selection of voxels in the voxels to mesh conversion. The experimental
results show that converting all voxels containing surfaces to mesh not only
reduces the rendering quality of the final hybrid representation, but also
significantly increases the storage of the assets.

In the voxel adjustment stage, we compare the complete model with variants of
Vosh-Base, incorporating different loss weights or resolutions for mesh
occupancy grids. The rows with $\lambda_{voxel}$ show the results of different
hyper-parameter weights of the loss function for voxel adjustment. The rows
with $r_{mesh}$ indicate the results of using mesh occupancy grids with
different resolutions. For the Vosh-Base model, we use $\lambda=0.001$ and
$r_{mesh}=128$ as hyper-parameters. For the Vosh-Light model, we use
$\lambda=0.1$ and $r_{mesh}=32$ as hyper-parameters. Experimental results
suggest that increasing the weight of voxel adjustment loss or reducing the
resolution of the mesh occupancy grid aids in compressing the number of
voxels, leading to speed benefits at the expense of rendering quality.

Table 2. The ablation study in the construction of hybrid representation. Variants | PSNR ↑ | SSIM ↑ | LPIPS ↓ | FPS ↑
---|---|---|---|---
Vosh-Base | 23.74 | 0.592 | 0.389 | 33.7
full bound | 23.49 | 0.583 | 0.395 | 36.0
$\lambda_{voxel}=0$ | 23.73 | 0.591 | 0.389 | 30.0
$\lambda_{voxel}=0.01$ | 23.72 | 0.590 | 0.389 | 38.7
$\lambda_{voxel}=0.1$ | 23.69 | 0.590 | 0.390 | 46.7
$r_{mesh}=0$ | 23.80 | 0.593 | 0.386 | 32.0
$r_{mesh}=1024$ | 23.79 | 0.593 | 0.387 | 32.0
$r_{mesh}=64$ | 23.70 | 0.590 | 0.389 | 37.7
Vosh-Light | 23.63 | 0.586 | 0.392 | 70.0

###  6.4. Limitations

Although the proposed hybrid representation Vosh brings valuable advantages to
real-time view synthesis, it is not without limitations. We adopt the same
view-dependent MLP as SNeRG and MERF, consequently sharing some similarities
with their drawbacks. Notably, it faces challenges such as the incapability to
model view-dependent colors for translucent objects, and represent extremely
large scenes or objects with complex reflections. These limitations, in turn,
may result in potential degradation of mesh optimization quality, thereby
impacting the rendering quality based on the hybrid representation.

##  7\. Conclusion

We have presented a hybrid representation Vosh, composed of both voxels and
mesh, for NeRF-based real-time view synthesis. By capitalizing on the
strengths of voxel and mesh components in our representation, we attain a
controllable balance between rendering quality and speed. Experimental
evidence demonstrates that our approach outperforms state-of-the-art methods,
showcasing exceptional flexibility for deployment across a range of mobile
devices.

In our future work, we intend to explore more efficient data structures for
organizing and storing the hybrid representation, aiming to optimize memory
consumption and further enhance rendering efficiency. Additionally, we are
considering more compact integration of classic representations to achieve
fast and high-quality novel view synthesis specifically tailored for mobile
devices.

## References

  * (1)
  * Barron et al. (2022) Jonathan T Barron, Ben Mildenhall, Dor Verbin, Pratul P Srinivasan, and Peter Hedman. 2022.  Mip-nerf 360: Unbounded anti-aliased neural radiance fields. In _Proceedings of CVPR_. 5470–5479.
  * Chan et al. (2022) Eric R Chan, Connor Z Lin, Matthew A Chan, Koki Nagano, Boxiao Pan, Shalini De Mello, Orazio Gallo, Leonidas J Guibas, Jonathan Tremblay, Sameh Khamis, et al. 2022\.  Efficient geometry-aware 3D generative adversarial networks. In _Proceedings of CVPR_. 16123–16133.
  * Chen et al. (2023) Zhiqin Chen, Thomas Funkhouser, Peter Hedman, and Andrea Tagliasacchi. 2023.  Mobilenerf: Exploiting the polygon rasterization pipeline for efficient neural field rendering on mobile architectures. In _Proceedings of CVPR_. 16569–16578.
  * Fridovich-Keil et al. (2022) Sara Fridovich-Keil, Alex Yu, Matthew Tancik, Qinhong Chen, Benjamin Recht, and Angjoo Kanazawa. 2022.  Plenoxels: Radiance fields without neural networks. In _Proceedings of CVPR_. 5501–5510.
  * Gao et al. (2022) Kyle Gao, Yina Gao, Hongjie He, Dening Lu, Linlin Xu, and Jonathan Li. 2022.  Nerf: Neural radiance field in 3d vision, a comprehensive review.  _arXiv preprint arXiv:2210.00379_ (2022).
  * Guo et al. (2023) Yuan-Chen Guo, Yan-Pei Cao, Chen Wang, Yu He, Ying Shan, and Song-Hai Zhang. 2023.  VMesh: Hybrid volume-mesh representation for efficient view synthesis. In _SIGGRAPH Asia 2023 Conference Papers_. 1–11.
  * Hedman et al. (2021) Peter Hedman, Pratul P Srinivasan, Ben Mildenhall, Jonathan T Barron, and Paul Debevec. 2021.  Baking neural radiance fields for real-time view synthesis. In _Proceedings of ICCV_. 5875–5884.
  * Jiang et al. (2023) Kaiwen Jiang, Shu-Yu Chen, Hongbo Fu, and Lin Gao. 2023.  NeRFFaceLighting: Implicit and Disentangled Face Lighting Representation Leveraging Generative Prior in Neural Radiance Fields.  _ACM Transactions on Graphics_ 42, 3 (2023), 1–18.
  * Kerbl et al. (2023) Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis. 2023.  3D Gaussian Splatting for Real-Time Radiance Field Rendering.  _ACM Transactions on Graphics_ 42, 4 (2023), 1–14.
  * Kwon et al. (2023) Obin Kwon, Jeongho Park, and Songhwai Oh. 2023.  Renderable Neural Radiance Map for Visual Navigation. In _Proceedings of CVPR_. 9099–9108.
  * Laine et al. (2020) Samuli Laine, Janne Hellsten, Tero Karras, Yeongho Seol, Jaakko Lehtinen, and Timo Aila. 2020.  Modular primitives for high-performance differentiable rendering.  _ACM Transactions on Graphics_ 39, 6 (2020), 1–14.
  * Li et al. (2022a) Chaojian Li, Sixu Li, Yang Zhao, Wenbo Zhu, and Yingyan Lin. 2022a.  RT-NeRF: Real-Time On-Device Neural Radiance Fields Towards Immersive AR/VR Rendering. In _Proceedings of ICCAD_. 1–9.
  * Li et al. (2022b) Gengyan Li, Abhimitra Meka, Franziska Mueller, Marcel C Buehler, Otmar Hilliges, and Thabo Beeler. 2022b.  EyeNeRF: a hybrid representation for photorealistic synthesis, animation and relighting of human eyes.  _ACM Transactions on Graphics_ 41, 4 (2022), 1–16.
  * Liu et al. (2020) Lingjie Liu, Jiatao Gu, Kyaw Zaw Lin, Tat-Seng Chua, and Christian Theobalt. 2020.  Neural sparse voxel fields.  _Proceedings of NeurIPS_ 33 (2020), 15651–15663.
  * Lorensen and Cline (1998) William E Lorensen and Harvey E Cline. 1998.  Marching cubes: A high resolution 3D surface construction algorithm.  In _Seminal graphics: pioneering efforts that shaped the field_. 347–353.
  * Mildenhall et al. (2021) Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. 2021.  Nerf: Representing scenes as neural radiance fields for view synthesis.  _Commun. ACM_ 65, 1 (2021), 99–106.
  * Müller et al. (2022) Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller. 2022.  Instant neural graphics primitives with a multiresolution hash encoding.  _ACM Transactions on Graphics_ 41, 4 (2022), 1–15.
  * Qiao et al. (2023) Yi-Ling Qiao, Alexander Gao, Yiran Xu, Yue Feng, Jia-Bin Huang, and Ming C. Lin. 2023.  Dynamic mesh-aware radiance fields. In _Proceedings of ICCV_. 385–396.
  * Rakotosaona et al. (2023) Marie-Julie Rakotosaona, Fabian Manhardt, Diego Martin Arroyo, Michael Niemeyer, Abhijit Kundu, and Federico Tombari. 2023.  NeRFMeshing: Distilling Neural Radiance Fields into Geometrically-Accurate 3D Meshes.  _arXiv preprint arXiv:2303.09431_ (2023).
  * Reiser et al. (2021) Christian Reiser, Songyou Peng, Yiyi Liao, and Andreas Geiger. 2021.  Kilonerf: Speeding up neural radiance fields with thousands of tiny mlps. In _Proceedings of ICCV_. 14335–14345.
  * Reiser et al. (2023) Christian Reiser, Rick Szeliski, Dor Verbin, Pratul Srinivasan, Ben Mildenhall, Andreas Geiger, Jon Barron, and Peter Hedman. 2023.  Merf: Memory-efficient radiance fields for real-time view synthesis in unbounded scenes.  _ACM Transactions on Graphics_ 42, 4 (2023), 1–12.
  * Sun et al. (2022a) Cheng Sun, Min Sun, and Hwann-Tzong Chen. 2022a.  Direct voxel grid optimization: Super-fast convergence for radiance fields reconstruction. In _Proceedings of CVPR_. 5459–5469.
  * Sun et al. (2022b) Jingxiang Sun, Xuan Wang, Yichun Shi, Lizhen Wang, Jue Wang, and Yebin Liu. 2022b.  Ide-3d: Interactive disentangled editing for high-resolution 3d-aware portrait synthesis.  _ACM Transactions on Graphics (ToG)_ 41, 6 (2022), 1–10.
  * Tang et al. (2023) Jiaxiang Tang, Hang Zhou, Xiaokang Chen, Tianshu Hu, Errui Ding, Jingdong Wang, and Gang Zeng. 2023.  Delicate Textured Mesh Recovery from NeRF via Adaptive Surface Refinement. In _Proceedings of ICCV_. 17739–17749.
  * Tewari et al. (2022) Ayush Tewari, Justus Thies, Ben Mildenhall, Pratul Srinivasan, Edgar Tretschk, Wang Yifan, Christoph Lassner, Vincent Sitzmann, Ricardo Martin-Brualla, Stephen Lombardi, et al. 2022\.  Advances in neural rendering. In _Computer Graphics Forum_ , Vol. 41. 703–735.
  * Wang et al. (2004) Zhou Wang, A.C. Bovik, H.R. Sheikh, and E.P. Simoncelli. 2004.  Image quality assessment: from error visibility to structural similarity.  _IEEE Transactions on Image Processing_ 13, 4 (2004), 600–612.
  * Wang et al. (2023) Zian Wang, Tianchang Shen, Merlin Nimier-David, Nicholas Sharp, Jun Gao, Alexander Keller, Sanja Fidler, Thomas Müller, and Zan Gojcic. 2023.  Adaptive Shells for Efficient Neural Radiance Field Rendering.  _ACM Transactions on Graphics_ 42, 6 (2023), 1–15.
  * Yang et al. (2022) Bangbang Yang, Chong Bao, Junyi Zeng, Hujun Bao, Yinda Zhang, Zhaopeng Cui, and Guofeng Zhang. 2022.  Neumesh: Learning disentangled neural mesh-based implicit field for geometry and texture editing. In _Proceedings of ECCV_. 597–614.
  * Yariv et al. (2023) Lior Yariv, Peter Hedman, Christian Reiser, Dor Verbin, Pratul P. Srinivasan, Richard Szeliski, Jonathan T. Barron, and Ben Mildenhall. 2023.  BakedSDF: Meshing Neural SDFs for Real-Time View Synthesis. In _ACM SIGGRAPH 2023 Conference Proceedings_. 1–9.
  * Yu et al. (2021) Alex Yu, Ruilong Li, Matthew Tancik, Hao Li, Ren Ng, and Angjoo Kanazawa. 2021.  Plenoctrees for real-time rendering of neural radiance fields. In _Proceedings of ICCV_. 5752–5761.
  * Zhang et al. (2018) Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. 2018.  The unreasonable effectiveness of deep features as a perceptual metric. In _Proceedings of CVPR_. 586–595.

Figure 7. More rendering results and zoomed-in images obtained by our method,
as well as some SOTA methods based on voxels or mesh representation.