11institutetext: International Institute of Information Technology, Hyderabad
11email: {kunal.jain<EMAIL_ADDRESS>22institutetext: Indian
Institute of Technology, Dharwad
22email<EMAIL_ADDRESS>
# Bayesian Optimization for Function Compositions with Applications to Dynamic
Pricing

Kunal Jain 11 Prabuchandran K. J 22 Tejas Bodas 11

###### Abstract

Bayesian Optimization (BO) is used to find the global optima of black box
functions. In this work, we propose a practical BO method of function
compositions where the form of the composition is known but the constituent
functions are expensive to evaluate. By assuming an independent Gaussian
process (GP) model for each of the constituent black-box function, we propose
EI and UCB based BO algorithms and demonstrate their ability to outperform
vanilla BO and the current state-of-art algorithms. We demonstrate a novel
application of the proposed methods to dynamic pricing in revenue management
when the underlying demand function is expensive to evaluate.

###### Keywords:

Bayesian optimization revenue maximization function composition dynamic
pricing and learning

##  1 Introduction

Bayesian Optimization (BO) is a popular technique for optimizing expensive-to-
evaluate black-box functions. Such a function might correspond to the case
where evaluating it can take up to hours or days, which for example, is the
case in re-training massive deep learning models with new hyper-parameters
[31]. In some cases, functions can be financially costly to evaluate, such as
drug testing [23] or revenue maximization [22]. In such black-box optimization
problems, one often has a fixed budget on the total number of function
evaluations that can be performed. For example, one typically has a budget on
the computational capacity spent in the hyper-parameter tuning of neural
networks [5, 27]. In such cases, BO proves to be very effective in providing a
resource-conserving iterative procedure to query the objective function and
identify the global optima [25].

The key idea in BO is to use a surrogate Gaussian Process (GP) model [24] for
the black-box function, which is updated as and when multiple function
evaluations are performed. To identify the location (in the domain) of the
following query point, an acquisition function (a function of the surrogate
model) is designed and optimized. The design of the acquisition function
depends not only on the application in mind but also on the trade-off between
exploration and exploitation that comes naturally with such sequential
optimization problems. Some popular acquisition functions used in the
literature are the Expected Improvement(EI), Probability of Improvement(PI),
Knowledge Gradient(KG) and Upper Confidence Bound(UCB) [8, 17, 10].

In this work, we consider Bayesian optimization of composite functions of the
form $g(x)=h(f_{1}(x),f_{2}(x),\ldots f_{M}(x))$ where functions $f_{i}$ are
expensive black-box functions while $h$ is known and easy to compute. More
specifically, we are interested in maximizing $g$ and identifying its optima.
A vanilla BO approach can be applied to this problem, ignoring the composite
structure of the function [10, 17]. In this approach, one would build a GP
posterior model over the function $g$ based on previous evaluations of $g(x)$
and then select the next query point using a suitable acquisition function
computed over the posterior. However, such a vanilla BO approach ignores the
available information about the composite nature of the functions, which we
show can easily be improved upon. In this work, we model each constituent
function $f_{i}$ using an independent GP model and build an acquisition
function that uses the known structure of the composition. Our algorithms
easily outperform the vanilla BO method in all the test cases and practical
applications we have considered. Our algorithms are also more practical and
less computationally intensive than the curent state-of-art BO algorithm for
composite function as proposed in [1].

Note that function compositions arise naturally in the real world. One such
example is the revenue maximization problem based on the composition of price
and demand function. Another example could be the optimization of the F1 score
in classification problems that can be seen as a composition of precision and
recall metrics [19]. A key novelty of our work lies in the application of our
BO algorithms to dynamic pricing problems in revenue management. To the best
of our knowledge, ours is the first work to perform dynamic pricing for
revenue optimization using Bayesian optimization methods.

###  1.1 Related Work

Optimization of function compositions has been studied under various
constraints such as convexity, inexpensive evaluations and derivative
information [4, 30, 15, 26]. The scope of these work are somewhat restrictive
and differ from our key assumption that the constituent functions in the
composition are black-box functions and are relatively expensive to evaluate.
Our work is closely related to Astudillo and Frazier [1] who optimize black-
box function compositions of the form $g(x)=h(f(x))$ using Bayesian
Optimization. In this work, the constituent function $f(x)$ is expensive to
evaluate and is modelled as a Multi-Ouput Gaussian Process (MOGP). This work
was further improved by Maddox et al. [21] using Higher Order Gaussian Process
(HOGP). These work assume that the member functions in the composition are
correlated and dedicates a significant amount of computational power to
capturing these correlations. They propose an EI-based acquisition function
for estimating the value of $g$ using a MOGP over the functions $f$. The
calculation of this acquisition function requires them to compute the inverse
of the lower Cholesky factor of the covariance matrix, which is a
computationally expensive task (runtime increases with order
$\mathcal{O}(N^{3})$ where $N$ is the size of the covariance matrix),
especially when optimizing high dimensional problems.

Our work differs from them in that we consider a composition of multiple
constituent functions with single output and assume an independent GP model
for each such constituent. This results in significantly lesser computational
requirements and faster iterations. Our work focuses on practical deployments
of the technique, as showcased in Section 4. We also propose a UCB-based
algorithm where the problem of matrix inversion does not arise. The UCB-based
algorithm allows the user to trade-off between exploration and exploitation
during iterations, making it more practical for our use cases. Finally, our
key contribution lies in applying the proposed methods to dynamic pricing
problems, a brief background of which is discussed in the next subsection.

###  1.2 Dynamic Pricing and Learning

Dynamic pricing is a phenomenon where the price for any commodity or good is
changed to optimize the revenue collected. Consider the scenario of a retailer
with a finite inventory, finite time horizon and a known probabilistic model
for the demand. On formulating this as a Markov decision problem, it is easy
to see that a revenue optimal pricing policy would be non-stationary,
resulting in different optimal prices for the same inventory level at
different time horizons. In this case, the dynamic nature of pricing is a by-
product of finite inventory and horizon effects. See [22] for more details.

Now consider a second scenario of a retailer with an infinite horizon and
infinite inventory, trying to find the optimal price for his product. Assuming
that the underlying probabilistic demand model is unknown to the retailer,
this becomes a simultaneous demand learning and price optimization problem. To
learn the underlying demand function, the retailer is required to probe or
explore the demand for the product at various prices, and use the information
gathered to converges to an optimal price over time. Clearly, in this setting,
uncertainty in the demand process naturally leads to exploration in the price
space, resulting in the dynamic nature of the pricing policy. See [7] for more
details. This second scenario (black-box demand function) is of particular
interest to us, and we apply our BO algorithms in this setting, something that
has not been done before.

Dynamic pricing with learning is a traditional research topic in Operations
Research with a long history (see [14] for a historical perspective). Lobo and
Boyd [16] introduced an exploration-exploitation framework for the demand
pricing problem, which balances the need for demand learning with revenue
maximization. The problem has been studied under various conditions, such as
limited [2] and unlimited inventory [18], customer negotiations [20], monopoly
[13, 11], limited price queries [12] etc. One recurrent theme in these work is
to assume a parametric form for the demand function in terms of price and
other exogenous variables. Reinforcement learning (RL) methods are then used
to simultaneously learn the unknown parameters of the demand function and set
prices that have low regret. See, for example, Broder and Rusmevichientong [9]
where linear, polynomial and logic demand function models have been assumed.
To the best of our understanding, these assumptions on the demand function are
rather over-simplified and are typically made for technical convenience.
Further, an RL method suited for a particular demand model (say linear demand)
may not work when the ground truth model for the demand is different (say,
logit model). There have been recent models which try to avoid this issue by
modelling the demand as a parameterized random variable (here price is a
parameter) but end up making similar convenient assumptions on the parametric
form of the mean or variance of the demand, see [6] and references therein.

To keep the demand function free from any specific parametric form, in this
work, we assume that it is a black-box expensive to evaluate function and
instead model it using a Gaussian process. The revenue function can be
expressed as a composition of the price and the demand and this allows us to
apply our proposed methods (of Bayesian optimization for function composition)
to the dynamic pricing and learning problem.

###  1.3 Contributions and Organization

The following are the key contributions of our work:

  1. 1.

We propose novel acquisition functions cEI and cUCB for Bayesian Optimization
for function compositions. These acquisition functions are based on EI and UCB
acquisition functions for vanilla BO and are less compute intensive and faster
to run through each iteration as compared to other state of the art algorithms
for function composition.

  2. 2.

We assume independent GPs to model the constituent functions and this allows
for possible parallelization of the posterior update step.

  3. 3.

As a key contribution of this work, we propose to use BO based dynamic pricing
algorithms to optimize the revenue. To the best of our knowledge, we are the
first, to use BO for learning the optimal price when the demand functions are
expensive to evaluate or are black-box in nature.

  4. 4.

We consider various revenue maximization scenarios, obtain the revenue
function as a composition of price and demand and illustrate the utility of
our algorithms in each of these setting.

The following is the organization of the rest of the paper. Section 2 formally
describes the problem statement and Section 3 describes our proposed
algorithms; Section 4 details our experimental results ; and finally, we
conclude in Section 5.

##  2 Problem Description

We begin by describing the problem of BO for composite functions in subsection
2.1. In subsection 2.2 we describe the dynamic pricing problem and model the
revenue function as a function composition to which BO methods for composite
functions can be applied.

###  2.1 BO for Function Composition

We consider the problem of optimizing
$g(\textbf{x})=h(f_{1}(\textbf{x}),f_{2}(\textbf{x}),\ldots,f_{M}(\textbf{x}))$
where $g:\mathcal{X}\rightarrow\mathrm{R}$,
$f_{i}:\mathcal{X}\rightarrow\mathrm{R}$,
$h:\mathrm{R}^{M}\rightarrow\mathrm{R}$ and
$\mathcal{X}\subseteq\mathrm{R}^{d}$. We assume each $f_{i}$ is a black-box
expensive-to-evaluate continuous function and $h$ is known and cheap to
evaluate. The optimization problem that we consider is

$\max_{\textbf{x}\in\mathcal{X}}~{}h(f_{1}(\textbf{x}),f_{2}(\textbf{x}),\ldots,f_{M}(\textbf{x})).$
(1)

We want to solve Problem 1 in an iterative manner where in the $n^{\text{th}}$
iteration, we can use the previous observations
$\\{\textbf{x}_{i},f_{1}(\textbf{x}_{i}),\ldots,f_{M}(\textbf{x}_{i}\\})\\}^{n-1}_{i=1}$
to request a new observation
$\\{\textbf{x}_{n},f_{1}(\textbf{x}_{1}),\ldots,f_{M}(\textbf{x}_{n})\\}$.

A vanilla BO algorithm applied to this problem would first assume a prior GP
model on $g$, denoted by $\mathcal{GP}(\mu(\cdot),K(\cdot,\cdot))$ where $\mu$
and $K$ denote the mean and covariance function of the prior model. Given some
function evaluations, an updated posterior GP model is obtained. A suitable
acquisition function, such as EI or PI can be used, to identify the next query
point. For example, in the $n+1^{\text{th}}$ update round, one would first use
the $n$ available observations
$(g(\textbf{x}_{1}),g(\textbf{x}_{2}),\ldots,g(\textbf{x}_{n}))$ to update the
GP model to $\mathcal{GP}(\mu^{(n)}(\cdot),K^{(n)}(\cdot,\cdot))$ where
$\mu^{(n)}(\cdot)$ is the posterior mean function and $K^{(n)}(\cdot,\cdot)$
is the posterior covariance function, see [24] for more details. The
acquisition function then uses this posterior model to identify the next query
location $\textbf{x}_{n+1}$. In doing so, vanilla BO ignores the values of the
member functions in the composition $h$.

BO for composite function, on the other hand, takes advantage of the available
information about $h$, and its easy-to-compute nature. Astudillo and Frazier
[1] model the constituent functions of the composition by a single multi-
output function
$\textbf{f}(\textbf{x})=(f_{1}(\textbf{x}),\ldots,f_{M}(\textbf{x}))$ and then
model the uncertainty in $\textbf{f}(\textbf{x})$ using a multi-output
Gaussian process to optimize $h({\textbf{f}(\textbf{x})})$. Since the prior
over $f$ is modelled as a MOGP, the proposed method tries to capture the
correlations between different components of the multi-output function
$\textbf{f}(\textbf{x})$. Note that the proposed EI and PI-based acquisition
functions are required to be computed using Monte Carlo sampling. Furthermore,
a sample from the posterior distribution is obtained by first sampling an $n$
variate normal distribution, then scaling it by the lower Cholesky factor and
then centering it with the mean of the posterior GP. Two problems arise due to
this:  1. Such simulation based averaging approach increases the time
complexity of the procedure linearly with the number of samples taken for
averaging and  2. calculation of the lower Cholesky factor increases the
function’s time complexity cubically with the number of data points.  These
factors render the algorithm unsuitable, particularly for problems with large
number of member functions or for problems with large dimensions.

To alleviate these problems, in this work, we model the constituent functions
using independent GPs. This modelling approach allows us to train GPs for each
output independently and hence the posterior GP update can be parallelized. We
propose two acquisition functions, cEI which is based on the EI algorithm and
cUCB, which is based on the GP-UCB algorithm [28]. Our cEI acquisition
function is similar in spirit to the EI-CF acquisition function of [1] but is
less computationally intensive owing to the independent GP model. Since we
have independent one dimensional GP model for each constituent function,
sampling points from the posterior GP does not require computing the Cholesky
factor (and hence matrix inversion), something that is needed in the case of
high-dimensional GP’s of [1]. This greatly reduces the complexity of the MC
sampling steps of our algorithm (see section 3 for more details). However, the
cEI acquisition function still suffers from the drawback of requiring Monte
Carlo averaging. To alleviate this problem, we propose a UCB based acquisition
function that uses the current mean plus scaled variance of the posterior GP
at a point as a surrogate for the constituent function at that point. As shown
by Srinivas et al. [28], while the mean term in the surrogate guides
exploitation, it is the variance of the posterior GP at a point that allows
for suitable exploration. The scaling of the variance term is controlled in
such a way that it balances the trade off between exploration and
exploitation. In Section 4, we illustrate the utility of our method, first for
standard test functions and then as an application to dynamic pricing problem.
Our algorithms, especially the cUCB one, outperforms not only vanilla BO but
also those proposed in Astudillo and Frazier [1].

###  2.2 Bayesian Optimization for dynamic pricing

We consider Bayesian Optimization for two types of revenue optimization
problems. The first problem optimizes the revenue per customer where customers
are characterized by their willingness-to-pay distribution (which is unknown).
In the second problem, we assume a parametric demand model (the functional
form is assumed to be unknown) and optimizes the associated revenue.

In the first model, we assume that an arriving customer has an associated
random variable, $V$, with complimentary cumulative distribution function
$\bar{F}$, indicating its maximum willingness to pay for the item sold. For an
item on offer at a price $p$, an arriving customer purchases it with the
probability

$d(p):=\bar{F}(p)=\text{Pr}\\{V\geq p\\}.$ (2)

In this case, when the product is on offer at a price $p$, the revenue per
customer $r(p)$ is given by $r(p)=p\bar{F}(p)$. The revenue function is a
composition of the price and demand or purchase probability and we assume that
the distribution of the purchase probability i.e., $F$ is not known and also
expensive to estimate. One could perform a vanilla BO algorithm by having a GP
model on $r(p)$ itself. However to exploit the known nature of the revenue
function, we will apply our function composition method by instead having a GP
on $\bar{F}(p)$ and demonstrate its superiority over vanilla BO.

In the second model, we assume that the true demand $d(p)$ for a commodity at
price $p$ has a functional form. This forms the ground truth model that
governs the demand, but we assume that the functional form for this demand is
not known to the manager optimizing the revenue. In our experiments, we assume
linear, logit, Booth and Matyas functional forms for the demand (see section 4
for more details.) Along similar lines, one could build more sophisticated
demand models to account for external factors (such as supply chain issues,
customer demographics or inventory variables), something that we leave for
future explorations.

Note that we make some simplifying assumptions about the retail environment in
these two models and our experiments. We assume a non-competitive monopolistic
market with an unlimited supply of the product and no marginal cost of
production. However, these assumptions can easily be relaxed by changing the
ground truth demand model appropriately, which are used in the experiments to
reflect these aspects. The fact that we use a GP model as a surrogate for the
unknown demand model offers it the ability to model a diverse class of demand
functions under diverse problem settings. We do not discuss these aspects
further but focus on the following simple yet meaningful experimental examples
that one typically encounters in revenue management problems. In the
following, p denotes the price vector:

  1. 1.

Independent demand model: A retailer supplies its product to two different
regions whose customer markets behave independently from each other. Thus, the
same product has independent and different demand functions (and hence
different optimal prices) in different geographical regions and under such
black-box demand models for the two regions $(d_{1},d_{2})$. The retailer is
interested in finding the optimal prices, leading to the optimization of the
following function: $g(\textbf{p})=p_{1}d_{1}(p_{1})+p_{2}d_{2}(p_{2})$.

  2. 2.

Correlated demand model: Assume that a retailer supplies two products at
prices $p_{1}$ and $p_{2}$ and the demand for the two products is correlated
and influenced by the price for the other product. Such a scenario can be
modelled by a revenue function of the form
$g(\textbf{p})=p_{1}d_{1}(\textbf{p})+p_{2}d_{2}(\textbf{p})$. Consider the
example where the prices of business and economy class tickets can influence
the demand in each segment. Similarly, the demand for a particular dish in a
fast food chain might be influenced by the prices for other dishes.

  3. 3.

Identical price model:  In this case, the retailer is compliant with having a
uniform price across locations. However, the demand function across different
locations could be independent at each of these locations, leading to the
following objective function: $g(p)=pd_{1}(p)+pd_{2}(p)$. This scenario can be
used to model different demand functions for different population segments in
their age, gender, socio-economic background, etc.

##  3 Proposed Method

As discussed in Section 2, we propose that instead of having a single GP model
over $g$, we have $M$ different GP models over each constituent function in
the composition. Each prior GP model will be updated using GP regression
whenever the observations of the constituent functions are available. A
suitably designed acquisition function would then try to find the optimal
point when the constituent functions should all be evaluated at, in the next
iteration. For ease of notation, we use the shorthand
$h(\\{f_{i}(\textbf{x})\\})$ to denote
$h(f_{1}(\textbf{x}),\ldots,f_{M}(\textbf{x}))$ in the subsequent sections.

###  3.1 Statistical Model and GP regression

Let $f_{i}^{1:n},~{}i\in\\{1,2,\ldots,M\\}$ denote the function evaluations of
the member functions at locations
$\\{\textbf{x}_{1},\textbf{x}_{2},\ldots,\textbf{x}_{n}\\}$ denotes as
$\textbf{x}^{1:n}$. In the input space $\mathcal{X}\subset\mathbb{R}^{d}$, let
$\mathcal{GP}(\mu^{(n)}_{i},k^{(n)}_{i})$ be the posterior GP over the
function $f_{i}$ where $\mu^{(n)}_{i}:\mathcal{X}\rightarrow\mathbb{R}$ is the
posterior mean function,
$k^{(n)}_{i}:\mathcal{X}\times\mathcal{X}\rightarrow\mathcal{R}$ is the
positive semi-definite covariance function and the variance of the function is
denoted by $\sigma_{i}^{(n)}(\textbf{x})$. The superscript $n$ is used to
denote the fact that the posterior update accounts for $n$ function
evaluations made till now. For each such GP, the underlying prior is a
combination of a constant mean function $\mu_{i}\in\mathbb{R}$ and the squared
exponential function $k_{i}$

$k_{i}(\textbf{x},\textbf{x}^{\prime})=\sigma^{2}\exp\left(-\frac{(\textbf{x}-\textbf{x}^{\prime})^{T}(\textbf{x}-\textbf{x}^{\prime})}{2l_{i}^{2}}\right)$

The kernel matrix $K_{i}$ is then defined as

$\textbf{K}_{i}:=\begin{bmatrix}k_{i}(\textbf{x}_{1},\textbf{x}_{1})&k_{i}(\textbf{x}_{1},\textbf{x}_{2})&\dots&k_{i}(\textbf{x}_{1},\textbf{x}_{n})\\\
\vdots&\vdots&\vdots&\vdots\\\
k_{i}(\textbf{x}_{n},\textbf{x}_{1})&k_{i}(\textbf{x}_{n},\textbf{x}_{2})&\dots&k_{i}(\textbf{x}_{n},\textbf{x}_{n})\\\
\end{bmatrix}$

and with abuse of notation define $\textbf{K}=\textbf{K}+\lambda^{2}I$ (to
account for noise in the function evaluations). The posterior distribution on
the function $f_{i}(\textbf{x})$ at any input $\textbf{x}\in\mathcal{X}$ [24]
is given by

$P(f_{i}(\textbf{x})|\textbf{x}^{1:n},f_{i}^{1:n})=\mathcal{N}(\mu^{(n)}_{i}(\textbf{x}),\sigma_{i}^{(n)}(\textbf{x})+\lambda^{2}),~{}~{}~{}i\in\\{1,2,\ldots,M\\}\mbox{~{}where~{}}$

$\displaystyle\mu^{(n)}_{i}(\textbf{x})$
$\displaystyle=\mu^{(0)}+\textbf{k}_{i}^{T}\textbf{K}_{i}^{-1}(f_{i}^{1:n}-\mu_{i}(\textbf{x}^{1:n}))$
$\displaystyle\sigma_{i}^{(n)}(\textbf{x})$
$\displaystyle=k_{i}(\textbf{x},\textbf{x})-\textbf{k}_{i}^{T}\textbf{K}_{i}^{-1}\textbf{k}_{i}$
$\displaystyle\textbf{k}_{i}$
$\displaystyle=\left[k_{i}(\textbf{x},\textbf{x}_{1})\hskip 5.0pt\ldots\hskip
5.0ptk_{i}(\textbf{x},\textbf{x}_{n})\right].$

Algorithm 1 cEI: Composite BO using EI based acquisition function

1:$T\xleftarrow{}$ Budget of iterations

2:$h(\cdot),f_{1}(\cdot),\ldots,f_{M}(\cdot)\xleftarrow{}$ composition and
member functions

3:$\textbf{X}=\\{\textbf{x}_{1},\ldots,\textbf{x}_{s}\\}\xleftarrow{}s$
starting points

4:$\textbf{F}=\\{(f_{1}(\textbf{x}),\ldots,f_{M}(\textbf{x}))\\}_{\textbf{x}\in\textbf{X}}\xleftarrow{}$
function evaluations at starting points

5:for $n=s+1,\ldots,s+T$ do

6: for $i=1,\ldots,M$ do

7: Fit model $\mathcal{GP}(\mu_{i}^{(n)}(\cdot),K_{i}^{(n)}(\cdot,\cdot))$
using evaluations of $f_{i}$ at points in X

8: end for

9: Find new point $\textbf{x}_{n}$ by optimizing $\text{cEI}(\textbf{x},L)$
(defined below)

10: Get $(f_{1}(\textbf{x}_{n}),\ldots,f_{M}(\textbf{x}_{n}))$

11: Augment the data $(f_{1}(\textbf{x}_{n}),\ldots,f_{M}(\textbf{x}_{n}))$
into F and update X with $\textbf{x}_{n}$

12:end for

13:function cEI($\textbf{x},L$)

14: for $l=1,\ldots,L$ do

15: Draw $M$ samples $Z_{(l)}\sim\mathcal{N}_{M}(0_{M},I_{M})$

16: Compute
$\alpha^{(l)}:=\\{h(\\{\mu^{(n)}_{i}(\textbf{x})+\sigma_{i}^{(n)}(\textbf{x})Z^{(l)}_{i}\\})-g^{*}_{n}\\}^{+}$

17: end for

18: return $E_{n}(\textbf{x})$ = $\frac{1}{L}\Sigma_{l=1}^{L}\alpha^{(l)}$

19:end function

###  3.2 cEI and cUCB Acquisition Functions

For any fixed point $\textbf{x}\in\mathcal{X}$, we use the information about
the composition function $h$ to estimate $g$ by first estimating the value of
each member function at x. However, this is not a straightforward task and
needs to be performed in a way similar to the vanilla EI acquisition using
Monte Carlo sampling. We propose to use the following acquisition function,
that we call as cEI.

$E_{n}(\textbf{x})=\mathbb{E}_{n}\left[{h(\\{\mu^{(n)}_{i}(\textbf{x})+\sigma^{(n)}_{i}(\textbf{x})Z_{i}\\})}-g_{n}^{*}\right]^{+}$
(3)

where $Z$ is drawn from an $M$-variate normal distribution and $g_{n}^{*}$ is
the best value observed so far. This acquisition function is similar to EI as
we subtract the best observation, $g^{*}$, so far and only consider negative
terms to be 0. Assuming independent GPs over the functions allows constant
time computation of the variance at x. However, since each function $f_{i}$ is
being considered an independent variable with mean $\mu_{i}^{(n)}(\cdot)$ and
variance $\sigma_{i}^{(n)}(\cdot)$, the calculation of $E_{n}(\bm{x})$ does
not have a closed form and thus, the expectation needs to be evaluated
empirically with sampling. Algorithm 1 provides the complete procedure for
doing BO with this acquisition function.

To alleviate this complexity in estimating the acquisition function, we
propose a novel UCB-style acquisition function. This function estimates the
value of each member function using the GP priors over them and controls the
exploration and exploitation factor with the help of the hyperparameter
$\lambda_{n}$:

$U_{n}(\textbf{x})=h(\\{\mu^{(n)}_{i}(\textbf{x})+\beta_{n}\sigma^{(n)}_{i}(\textbf{x})\\})$
(4)

Algorithm 2 gives the complete details for using this acquisition function.
The user typically starts with a high value for $\beta$ to promote exploration
and reduces iteratively to exploit the low reget regions it found. For our
experiments, we start with $\beta=1$ and exponentially decay it in each
iteration by a factor of 0.99.

Algorithm 2 cUCB: Composite BO using UCB based acquisition function

1:$T\xleftarrow{}$ Budget of iterations

2:$h(\cdot),f_{1}(\cdot),\ldots,f_{M}(\cdot)\xleftarrow{}$ composition and
member functions

3:$\textbf{X}=\\{\textbf{x}_{1},\ldots,\textbf{x}_{s}\\}\xleftarrow{}s$
starting points

4:$\textbf{F}=\\{(f_{1}(\textbf{x}),\ldots,f_{M}(\textbf{x}))\\}_{\textbf{x}\in\textbf{X}}\xleftarrow{}$
function evaluations at starting points

5:$\beta\xleftarrow{}$ Exploration factor

6:for $n=s+1,\ldots,s+T$ do

7: for $i=1,\ldots,M$ do

8: Fit model $\mathcal{GP}(\mu_{i}^{(n)}(\cdot),K_{i}^{(n)}(\cdot,\cdot))$
using evaluations of $f_{i}$ at points in X

9: end for

10: Find new point $\textbf{x}_{n}$ suggested by the composition function
using Eq. 4

11: Get $(f_{1}(\textbf{x}_{n}),\ldots,f_{M}(\textbf{x}_{n}))$

12: Augment the data $(f_{1}(\textbf{x}_{n}),\ldots,f_{M}(\textbf{x}_{n}))$
into F and update X with $\textbf{x}_{n}$

13: Update $\beta$

14:end for

##  4 Experiments and Results

In this section, we compare the results of our cUCB and cEI algorithms with
Vanilla EI, Vanilla UCB and the state-of-the-art BO for Composite Functions
(BO-CF) [1] using HOGP [21] in terms of loss in regret and runtime of the
algorithms. We first compare our methods on 3 test functions and then move on
to show their applications to three different pricing scenarios. Our code is
available here.

Our algorithms are implemented with the help of the BoTorch framework [3] and
use the APIs provided by them to declare and fit the GP models. We assume
noiseless observations for our results in this section, and the same results
can be obtained when we add Gaussian noise to the problem with a fixed mean
and variance. We start with the same 10 initial random points and run our BO
algorithms for 70 iterations. We use a system with 96 Intel Xeon Gold 6226R
CPU @2.90GHz and 96GB of memory shared between the CPUs.

We compare the performance of different algorithms based on the log of mean
minimum regret till each iteration, averaged over 100 runs. In a single BO
run, the regret at iteration $i$ in the $k^{th}$ run is defined as
$l_{i}^{k}=g^{*}-g(\textbf{x}_{i})$ where $g^{*}$ is the global maximum of the
objective function. The minimum regret at iteration $i$ in the $k^{th}$ run is
defined as $m_{i}^{k}=\min_{1\leq j\leq i}l_{j}^{k}$ and the final metric at
iteration $i$ averaged across 100 runs is calculated as

$r_{i}=\log_{10}\left(\frac{1}{100}\sum_{k=1}^{100}m_{i}^{k}\right).$

###  4.1 Results on Test Functions

(a) Langermann function

(b) Dixon-Price function

Figure 1: Log regret for test functions with a composite nature

We first asses our algorithms on three standard test functions [29]:

####  4.1.1 Langermann Function:

To express this function as a function composition, we consider each outer
iteration of the Langermann function to be a separate constituent function,
that is,

$f_{i}(\textbf{x})=\exp\left(-\frac{1}{\pi}\sum\limits_{j=1}^{d}(x_{j}-A_{ij})^{2}\right)\cos\left(\pi\sum\limits_{j=1}^{d}(x_{j}-A_{ij})\right).$

The composition for this will be
$h(\\{f_{i}(\textbf{x})\\})=\sum_{i=1}^{m}c_{i}f_{i}(\textbf{x})$ with $d=2$,
$c=(1,2,5,2,3)$, $m=5$, $A=((3,5),(5,2),(2,1),(1,4),(7,9))$ and domain
$\mathcal{X}=[0,10]^{2}$. Note that the terms differ only in the columns of
hyperparameter $A$ for different member functions and thus, should have a high
covariance.

####  4.1.2 Dixon-Price Function

In this function, we take the term associated with each dimension of the input
to be a separate constituent, that is,

$f_{1}(\textbf{x})=(x_{1}-1)^{2},f_{i}(\textbf{x})=i(2x_{i}^{2}-x_{i-1})^{2}$

The composition for this function will be
$h(\\{f_{i}(\textbf{x})\\})=\sum_{i=1}^{d}f_{i}(\textbf{x})$ with $d=5$ and
domain $\mathcal{X}=[-10,10]^{d}$. Since only consecutive terms in this
function share one variable, the member functions do have a non-zero
covariance but it will not be as high as in the Langermann function.

####  4.1.3 Ackley Function

Here, we build a more complex composition function by considering the terms in
the exponents as the member functions, resulting in

$f_{1}(\textbf{x})=\sqrt{\frac{1}{d}\sum\limits_{i=1}^{d}x_{i}^{2}}\text{ and
}f_{2}(\textbf{x})=\frac{1}{d}\sum\limits_{i=1}^{d}\cos(cx_{i})$
$h(\textbf{x})=-a\exp(-bf_{1}(\textbf{x}))-\exp(f_{2}(\textbf{x}))+a+\exp(1)$

with $d=5$, $a=20$, $b-0.2$, $c=2\pi$ and domain
$\mathcal{X}=[-32.768,32.768]^{d}$.

####  4.1.4 Results

Figure 1(a), 1(b) and 2(a) compares the results of different algorithms.
Vanilla EI and UCB algorithms do not consider the composite nature of the
function while BO-CF and our methods use the composition defined above. Even
with the high covariance between the members in Langermann function, cUCB
outperforms BO-CF while the cEI algorithm has a similar performance level.
However, when that covariance reduces in the Dixon-Price function, the cEI
algorithm performs better than BO-CF while the cUCB algorithm significantly
outperforms it. Figure 2(a) shows that our algorithms work well with
complicated composition functions as well and both, cUCB and cEI, outperform
BO-CF.

###  4.2 Results for Demand Pricing Experiments

(a) Ackley function

(b) Independent demand model

Figure 2: Log of expected regret for pricing tasks

We now test our approach on the demand models discussed in Section 2.2.

####  4.2.1 Independent Demand Model:

Recall that in this model, we allow the price at each location to be different
and model the demand in a region to depend only on the price therein. We
consider 4 regions where each region has a parametric demand functions and
randomly chosen parameters. Particularly, we assume that two regions have a
logit demand function ($d(p)=\frac{e^{-z_{1}-z_{2}p}}{1+e^{-z_{1}-z_{2}p}}$)
with $z_{1}\in\left[1.0,2.0\right]$, $z_{2}\in\left[-1.0.1.0\right]$ and the
other two regions have a linear demand function ($d(p)=z_{1}-z_{2}p$) with
$z_{1}\in\left[0.75,1\right]$ and $z_{2}\in[2/3,0.75]$. The domain for this
model is $\mathcal{X}=[0,1]^{4}$ and the composition is The composition
function looks
$h(d_{1}(\textbf{p}),d_{2}(\textbf{p}),d_{3}(\textbf{p}),d_{4}(\textbf{p}))=\sum_{i=1}^{4}p_{i}d_{i}(p_{i})$.

####  4.2.2 Correlated Demand Model:

In this example, we assume that different products are for sale at different
prices, and there is a certain correlation between demand for different
products via their prices. We consider the case of 2 products where one of the
product has a demand function governed by the Matyas function [29]. We assume
that the demand for the second product is governed by the Booth function [29].
More specifically, the function composition and constituent functions are as
below, where the domain for the problem is $\mathcal{X}=[0,10]^{2}$:

$\displaystyle d_{1}(\textbf{p})$
$\displaystyle=8(100-\text{Matyas}(\textbf{p}))$ $\displaystyle
d_{2}(\textbf{p})$ $\displaystyle=1154-\text{Booth}(\textbf{p})$

where Matyas function is defined as
$\text{Matyas}(\textbf{x})=0.26(x_{1}^{2}+x_{2}^{2})-0.48p_{1}p_{2}$.
Similarly, Booth function is defined as
$\text{Booth}(\textbf{x})=(x_{1}+2x_{2}-7)^{2}+(2x_{1}+x_{2}-5)^{2}$. The
composition for this will be
$h(d_{1}(\textbf{p}),d_{2}(\textbf{p}))=p_{1}d_{1}(\textbf{p})+p_{2}d_{2}(\textbf{p})$.

####  4.2.3 Identical price Model:

In this example, we assume that a commodity is sold for same price at two
different regions but the willingness to pay variable for customers in the two
regions is different. We assume that the willingness to pay distribution in
one region follows exponential distribution with $\lambda=5.0$. n the other
region, this is assumed to be a gamma distribution with
$\alpha=10.0,\beta=10.0$ The resulting function composition is given by :
$h(d_{1}(p),d_{2}(p))=pd_{1}(p)+pd_{2}(p)$.

####  4.2.4 Results

Figures 2(b), 3(a) and 3(b) compare the results of these dynamic pricing
models for the different BO algorithms. Our algorithms perform well even in
higher dimensions of input and member functions with cUCB marginally
outperforming BO-CF in the first model. cUCB matches the minimum regret in the
second model and converges to it much faster than BO-CF. In the case of the
third model, having independent GP’s performs better than BO-CF with cEI.

(a) Correlated demand model

(b) Identical price model

Figure 3: Log of expected regret for pricing tasks Task | EI | UCB | cEI | cUCB | HOGP
---|---|---|---|---|---
Langermann function | 1.74 | 1.73 | 9.71 | 9.34 | 36.30
Dixon-Price function | 1.70 | 1.69 | 14.47 | 12.71 | 41.72
Ackley function | 1.63 | 1.55 | 11.56 | 11.37 | 47.91
Independent demand model | 2.05 | 2.03 | 7.85 | 7.12 | 19.12
Correlated demand model | 3.91 | 3.29 | 9.63 | 9.19 | 53.01
Identical price model | 2.55 | 2.38 | 7.27 | 7.81 | 24.37
Table 1: Run-time for 70 iterations across algorithms in seconds

###  4.3 Runtime Comparisons with State of the Art

Along with the performance of our algorithms being superior in terms of
regret, the methodology of training independent GPs is significantly faster in
terms of run time. As shown in Table 1, both of our algorithms are between 3
to 4 times faster than BO-CF using HOGP on average and their run time
increases linearly with the number of member functions in the composition when
compared to vanilla EI and UCB. By not having to compute the inverse matrix
for estimating the lower Cholesky factor of the covariance matrix, we gain
large improvements in run time. The elimination of inverse matrix computation
while estimating with the help of lower Cholesky factor of the covariance
matrix results in the large improvement in run time over BO-CF. Also note that
our UCB variant is marginally faster than the EI variant as well due to the
elimination of MC sampling in the process.

##  5 Conclusion

In this work, we have proposed EI and UCB based BO algorithms, namely cEI and
cUCB for optimizing functions with a composite nature. We further apply our
algorithms to the revenue maximization problem and test our methods on
different market scenarios. We show that our algorithms, particularly cUCB,
outperforms vanilla BO as well as the current state of the art BO-CF
algorithm. Our algorithms are computationally superior because they do not
require multiple Cholesky decompositions as required in the BO-CF algorithm.

As part of future work, we would like to provide theoretical bounds on
cumulative regret for the proposed algorithms. We would also like to see the
applicability of the proposed algorithms in hyper-parameter tuning for
optimizing F1 score. It would also be interesting to propose BO algorithms for
an extended model wherein the member functions can be probed independently
from each other at different costs.

## References

  * [1] Astudillo, R., Frazier, P.: Bayesian optimization of composite functions. In: Chaudhuri, K., Salakhutdinov, R. (eds.) Proceedings of the 36th International Conference on Machine Learning. Proceedings of Machine Learning Research, vol. 97, pp. 354–363. PMLR (09–15 Jun 2019), https://proceedings.mlr.press/v97/astudillo19a.html
  * [2] Babaioff, M., Dughmi, S., Kleinberg, R., Slivkins, A.: Dynamic pricing with limited supply. ACM Transactions on Economics and Computation 3(1), 1–26 (Mar 2015), https://doi.org/10.1145/2559152
  * [3] Balandat, M., Karrer, B., Jiang, D.R., Daulton, S., Letham, B., Wilson, A.G., Bakshy, E.: BoTorch: A Framework for Efficient Monte-Carlo Bayesian Optimization. In: Advances in Neural Information Processing Systems 33 (2020), http://arxiv.org/abs/1910.06403
  * [4] Barber, R.F., Sidky, E.Y.: Mocca: Mirrored convex/concave optimization for nonconvex composite functions. Journal of Machine Learning Research 17(144), 1–51 (2016), http://jmlr.org/papers/v17/15-583.html
  * [5] Bergstra, J., Yamins, D., Cox, D.: Making a science of model search: Hyperparameter optimization in hundreds of dimensions for vision architectures. In: Dasgupta, S., McAllester, D. (eds.) Proceedings of the 30th International Conference on Machine Learning. Proceedings of Machine Learning Research, vol. 28, pp. 115–123. PMLR, Atlanta, Georgia, USA (17–19 Jun 2013), https://proceedings.mlr.press/v28/bergstra13.html
  * [6] den Boer, A.V., Zwart, B.: Simultaneously learning and optimizing using controlled variance pricing. Management Science 60(3), 770–783 (Mar 2014), https://doi.org/10.1287/mnsc.2013.1788
  * [7] den Boer, A.V., Zwart, B.: Dynamic pricing and learning with finite inventories. Operations Research 63(4), 965–978 (2015), http://www.jstor.org/stable/24540425
  * [8] Brochu, E., Cora, V.M., de Freitas, N.: A tutorial on bayesian optimization of expensive cost functions, with application to active user modeling and hierarchical reinforcement learning (2010)
  * [9] Broder, J., Rusmevichientong, P.: Dynamic pricing under a general parametric choice model. Operations Research 60(4), 965–980 (2012), http://www.jstor.org/stable/23260288
  * [10] Candelieri, A.: A gentle introduction to bayesian optimization. In: 2021 Winter Simulation Conference (WSC). pp. 1–16 (2021). https://doi.org/10.1109/WSC52266.2021.9715413
  * [11] Chen, Y.M., Jain, D.C.: Dynamic monopoly pricing under a poisson-type uncertain demand. The Journal of Business 65(4), 593–614 (1992), http://www.jstor.org/stable/2353199
  * [12] Cheung, W.C., Simchi-Levi, D., Wang, H.: Technical note—dynamic pricing and demand learning with limited price experimentation. Operations Research 65(6), 1722–1731 (Dec 2017), https://doi.org/10.1287/opre.2017.1629
  * [13] Crapis, D., Ifrach, B., Maglaras, C., Scarsini, M.: Monopoly pricing in the presence of social learning. Management Science 63(11), 3586–3608 (Nov 2017). https://doi.org/10.1287/mnsc.2016.2526, https://doi.org/10.1287/mnsc.2016.2526
  * [14] den Boer, A.V.: Dynamic pricing and learning: Historical origins, current research, and new directions. Surveys in Operations Research and Management Science 20(1), 1–18 (2015), https://www.sciencedirect.com/science/article/pii/S1876735415000021
  * [15] Drusvyatskiy, D., Paquette, C.: Efficiency of minimizing compositions of convex functions and smooth maps. Mathematical Programming 178(1-2), 503–558 (Jul 2018), https://doi.org/10.1007/s10107-018-1311-3
  * [16] Elreedy, D., F. Atiya, A., I. Shaheen, S.: A novel active learning regression framework for balancing the exploration-exploitation trade-off. Entropy 21(7) (2019), https://www.mdpi.com/1099-4300/21/7/651
  * [17] Frazier, P.I.: A tutorial on bayesian optimization (2018)
  * [18] Harrison, J.M., Keskin, N.B., Zeevi, A.: Bayesian dynamic pricing policies: Learning and earning under a binary prior distribution. Management Science 58(3), 570–586 (Mar 2012), https://doi.org/10.1287/mnsc.1110.1426
  * [19] Injadat, M., Salo, F., Nassif, A.B., Essex, A., Shami, A.: Bayesian optimization with machine learning algorithms towards anomaly detection. In: 2018 IEEE Global Communications Conference (GLOBECOM). pp. 1–6 (2018). https://doi.org/10.1109/GLOCOM.2018.8647714
  * [20] Kuo, C.W., Ahn, H.S., Aydin, G.: Dynamic pricing of limited inventories when customers negotiate. Operations Research 59(4), 882–897 (2011), http://www.jstor.org/stable/23013153
  * [21] Maddox, W., Balandat, M., Wilson, A.G., Bakshy, E.: Bayesian optimization with high-dimensional outputs. In: Beygelzimer, A., Dauphin, Y., Liang, P., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems (2021), https://openreview.net/forum?id=vDo__0UwFNo
  * [22] Phillips, R.L.: Pricing and Revenue Optimization. Stanford University Press (May 2021), https://doi.org/10.1515/9781503614260
  * [23] Pyzer-Knapp, E.O.: Bayesian optimization for accelerated drug discovery. IBM Journal of Research and Development 62(6), 2:1–2:7 (2018). https://doi.org/10.1147/JRD.2018.2881731
  * [24] Rasmussen, C.E., Williams, C.K.I.: Gaussian Processes for Machine Learning. The MIT Press (2005). https://doi.org/10.7551/mitpress/3206.001.0001, https://doi.org/10.7551/mitpress/3206.001.0001
  * [25] Scotto Di Perrotolo, A.: A Theoretical Framework for Bayesian Optimization Convergence. Master’s thesis, KTH, Optimization and Systems Theory (2018), http://kth.diva-portal.org/smash/record.jsf?pid=diva2%3A1194281&dswid=6896
  * [26] Shapiro, A.: On a class of nonsmooth composite functions. Mathematics of Operations Research 28(4), 677–692 (2003), http://www.jstor.org/stable/4127005
  * [27] Snoek, J., Larochelle, H., Adams, R.P.: Practical bayesian optimization of machine learning algorithms. In: Pereira, F., Burges, C., Bottou, L., Weinberger, K. (eds.) Advances in Neural Information Processing Systems. vol. 25. Curran Associates, Inc. (2012), https://proceedings.neurips.cc/paper/2012/file/05311655a15b75fab86956663e1819cd-Paper.pdf
  * [28] Srinivas, N., Krause, A., Kakade, S.M., Seeger, M.W.: Information-theoretic regret bounds for gaussian process optimization in the bandit setting. IEEE Transactions on Information Theory 58(5), 3250–3265 (2012). https://doi.org/10.1109/TIT.2011.2182033
  * [29] Surjanovic, S., Bingham, D.: Virtual library of simulation experiments: Test functions and datasets. Retrieved February 7, 2023, from https://www.sfu.ca/~ssurjano
  * [30] Woodworth, B.E., Srebro, N.: Tight complexity bounds for optimizing composite objectives. In: Lee, D., Sugiyama, M., Luxburg, U., Guyon, I., Garnett, R. (eds.) Advances in Neural Information Processing Systems. vol. 29. Curran Associates, Inc. (2016), https://proceedings.neurips.cc/paper/2016/file/645098b086d2f9e1e0e939c27f9f2d6f-Paper.pdf
  * [31] Wu, J., Chen, X.Y., Zhang, H., Xiong, L.D., Lei, H., Deng, S.H.: Hyperparameter optimization for machine learning models based on bayesian optimizationb. Journal of Electronic Science and Technology 17(1), 26–40 (2019), https://www.sciencedirect.com/science/article/pii/S1674862X19300047


# LLplace: The 3D Indoor Scene Layout Generation and Editing via Large
Language Model

Yixuan Yang1,2 Junru Lu2 Zixiang Zhao3 Zhen Luo1 James J.Q. Yu4 Victor
Sanchez2 Feng Zheng1

1 Southern University of Science and Technology 2 University of Warwick
3Xi’an Jiao Tong University 4York University
Email<EMAIL_ADDRESS>Author.

###### Abstract

Designing 3D indoor layouts is a crucial task with significant applications in
virtual reality, interior design, and automated space planning. Existing
methods for 3D layout design either rely on diffusion models, which utilize
spatial relationship priors, or heavily leverage the inferential capabilities
of proprietary Large Language Models (LLMs) , which require extensive prompt
engineering and in-context exemplars via black-box trials. These methods often
face limitations in generalization and dynamic scene editing. In this paper,
we introduce LLplace, a novel 3D indoor scene layout designer based on
lightweight fine-tuned open-source LLM Llama3. LLplace circumvents the need
for spatial relationship priors and in-context exemplars, enabling efficient
and credible room layout generation based solely on user inputs specifying the
room type and desired objects. We curated a new dialogue dataset based on the
3D-Front dataset, expanding the original data volume and incorporating
dialogue data for adding and removing objects. This dataset can enhance the
LLM’s spatial understanding. Furthermore, through dialogue, LLplace activates
the LLM’s capability to understand 3D layouts and perform dynamic scene
editing, enabling the addition and removal of objects. Our approach
demonstrates that LLplace can effectively generate and edit 3D indoor layouts
interactively and outperform existing methods in delivering high-quality 3D
design solutions. Code and dataset will be released.

##  1 Introduction

The design and optimization of 3D indoor object layouts play a crucial role in
various applications, including interior design Feng et al. (2024);
Paschalidou et al. (2021), game design Deitke et al. (2022), automated space
planning Fu et al. (2024); Huang et al. (2023a), and robotics Yang et al.
(2024a); Chen et al. (2023). Effective and reasonable indoor layout design
enhances both the functionality and aesthetic appeal of living and working
spaces, directly impacting the quality of life and productivity of their
occupants. Despite significant advancements in the field of artificial
intelligence, specifically in natural language processing and computer vision,
the task of flexibly generating and dynamically editing 3D indoor layouts from
naive texts remains a complex challenge.

Existing methods for designing indoor scene layouts are primarily categorized
into two types. The first one is based on diffusion models Ho et al. (2020),
which utilize these models along with various spatial feature priors to
generate 3D layouts. Representative approaches in this category include
DiffuScene Tang et al. (2023) and InstructScene Lin and Mu (2024). The second
category relies on the inferential capabilities of existing LLMs (e.g., GPT-4
Achiam et al. (2023)), using numerous prompts to generate corresponding 3D
layouts, such as LayoutGPT Feng et al. (2024) and Holodeck Yang et al.
(2024b). DiffuScene Tang et al. (2023) extracts the multidimensional features
of objects in space and uses diffusion to achieve the self-denosing learning
and generation of 3D spatial layouts. In contrast, InstructScene Lin and Mu
(2024) leverages the positional relationships between objects as conditions,
constructing a graph model where each node represents an object, then uses
graph diffusion to generate the layout. On the other hand, LLM-based methods
differ from diffusion-based models, as they utilize inherent language
understanding and generation capabilities to interpret textual prompts and
translate them into spatial arrangements. Holodeck requires specific spatial
relationships between objects as prompts to generate the room layout, while
LayoutGPT first retrieves relevant room layouts from a well-crafted database
and then use these as in-context exemplars to guide the LLMs in generating the
targeting layout. In summary, above existing approaches have obvious
drawbacks. Firstly, most layout generation models rely on spatial relationship
priors or examples as model inputs to guide in generation. If users do not
provide these relationships or if the system cannot retrieve accurate
examples, these models cannot achieve convincing results. Herein, these prior-
inspired strategies significantly constrain the model’s generalization
capability when meets newly different scenarios, where high-quality priors or
exemplars are expensive. Secondly, most current LLM-based layout models solely
supports one-time static layout generation, while can not perform dynamic
scene editing. This does not align with the interactive nature intended for
LLMs. Therefore, we are particularly interested in exploring the potential of
LLMs as a dynamic 3D scene layout designer that do not rely on strong priors
or pre-prepared in-context exemplars.

Figure 1: Generated results of 3D indoor scenes from LLplace and compared with
_LayoutGPT_ and _GPT-4o_. And editing results from LLplace and compared with
_GPT-4o_.

Consequently, in this paper, we introduce a novel 3D indoor scene layout
designer, LLplace (Large Language Model for Indoor Placement). We first
carefully design a format-friendly meta prompt template towards 3D indoor
layout design, then reconstruct regular 3D-Front dataset Fu et al. (2021a) for
static scene generation and dynamic scenario editing in the format of multi-
turn conversations. This is to ensure compatibility with the interactive
routines of LLMs. Specifically, we opt to fine-tune a SOTA open-source LLM,
Llama3 Touvron et al. (2023).We employ LoRA Hu et al. (2021) for parameter-
efficient fine-tuning. In our design pipeline, we first specify the user input
as the room type and descriptions of objects within the room. We then retrieve
3D assets and corresponding bounding boxes from the 3D-Front dataset using the
object descriptions. Subsequently, we convert the user inputs and
corresponding bounding boxes of the corresponding objects into a JSON format
that the LLM can accept, which is like the general data process in the LLM
area Liu et al. (2023); Lu et al. (2023). The entire transcription is finally
completed after we embed the user request JSON with our meta prompt template.
The overall pipeline is not only used for the construction of training data,
but also for the execution of inference. In accordance with the input JSON
format, we also design to use JSON to standardize the labels of the training
data. The “JSON-in” and “JSON-out” schema is beneficial for the coupling of
semi-structured natural language requests and auxiliary structured
programming. Based on the retrieved 3D assets and their bounding boxes, we ask
the LLM to report its design containing the coordinates and rotation angles of
the objects in the room. We go beyond traditional static 3D indoor layout
generation by also considering dynamic scene editing. We develop the
aforementioned instructions and labels into dialogues, adding an additional
round of editing requests, such as adding or removing objects. The LLM then
reasonably modifies its further output accordingly. In addition, we are able
to refactor the user’s input JSON and LLM’s output JSON at each turn of the
conversation into spatial 3D bounding box layouts, which can then be rendered
into a series of 3D representations .

As shown in Figure 1, we compare LLplace with LayoutGPT Feng et al. (2024) and
the latest GPT-4o model with our meta prompt template. The scenes generated by
LLplace are more reasonable compared to the other two models without
overlapping and wrong rotation problems. In scene editing, LLplace can
understand the existing 3D scene and add objects to the correct positions.
This demonstrates that the dynamic understanding and editing capabilities of
the LLplace designer are not present in the existing LLM-based approaches. The
main contribution of the LLplace can be summarized as follows:

  * •

We introduce a novel 3D indoor scene layout designer, LLplace, which is based
on a fine-tuned open-source LLM model. This designer does not require the use
of spatial relationship priors or in-context exemplars. Instead, it
efficiently generates credible room layouts based solely on user inputs
specifying the room type and the objects to be placed.

  * •

We curate a new dialogue dataset based on the 3D-Front dataset, which not only
expands the original data volume but also includes dialogue data for adding
and removing objects, enhancing the spatial understanding capabilities of the
LLM towards the real physical world.

  * •

By fine-tuned with this dialogue data, LLplace ensures that the LLM can
statically generate 3D layouts. Also, it activates the LLM’s capability to
understand and generate 3D layouts via chatting, enabling the dynamic addition
and removal of objects within the spatial layout.

##  2 Related Work

Models for 3D indoor scene layout can be broadly classified into three
categories: traditional methods using prior knowledge, generative models for
scene generation, and LLM-based methods.

Traditional 3D Indoor Scene Design. Early approaches to 3D indoor scene design
used autoregressive models that required supervision with 2D bounding boxes or
various visual maps Ritchie et al. (2019); Luo et al. (2020); Yang et al.
(2021b). And Purkait et al. (2020); Gao et al. (2023); Yang et al. (2021a) use
variational auto-encoder (VAE) Kingma and Welling (2013) to model the
distribution of objects to generate indoor scenes. SceneFormer Wang et al.
(2021) introduced the use of transformers to add furniture to scenes, marking
a significant innovation in the field. Unlike previous methods that relied on
separate models to predict different object attributes, ATISS Paschalidou et
al. (2021) demonstrated that a single transformer model could generate more
realistic and efficient arrangements. However, these traditional models often
cannot use textual instructions to specify scene inputs and requirements.

Generative Models for 3D Indoor Scene Design. Using diffusion models for
indoor scene design has become increasingly popular Tang et al. (2023); Huang
et al. (2023b); Fang et al. (2023); Lin and Mu (2024). DiffuScene Tang et al.
(2023) extracts the features of various objects and uses a diffusion model to
generate the characteristics of indoor scenes. Similarly, Ctrl-room Fang et
al. (2023) generates bounding boxes and then employs ControlNet Zhang et al.
(2023) to create panoramas, which are converted into textured meshes.
InstructScene Lin and Mu (2024) takes a different approach by utilizing
relationships between objects as priors and applying graph diffusion for scene
generation. Despite their strengths, diffusion-based models face challenges in
achieving real-time interactivity and understanding existing scene layouts for
further editing. These limitations make it difficult for diffusion models to
facilitate dynamic scene modifications, which is a capability that shows
potential in LLM-based models.

LLM-Based Methods for 3D Indoor Scene Design. LLM-based methods leverage the
inferential capabilities of LLMs to design 3D indoor layouts. These models use
extensive prompts to generate corresponding layouts. For instance, Holodeck
Yang et al. (2024b) requires users to specify spatial relationships between
objects as prompts to generate room layouts. Similarly, I-Design Çelen et al.
(2024) first generates a graph of relationships, which is then used to create
2D design plans. Aguina-Kang et al. (2024) represents an entire scene using a
program, also requiring numerous prompts and spatial relationships to assist
in generation. LayoutGPT Feng et al. (2024), uses user inputs to retrieve
relevant room layouts from a database and employs these as in-context
exemplars to guide the GPT model in generating new layouts. While these
methods demonstrate the capability of LLMs in layout design, they still face
challenges in terms of flexibility due to their reliance on predefined
examples. And also these methods do not utilize the inherent conversational
capabilities of LLMs to further edit scenes after generation. This limits
their interactivity and adaptability, as they cannot dynamically adjust
layouts based on user dialogue.

##  3 Method

The overall pipeline of LLplace is illustrated in Fig. 2. In section 3.1, we
define the problem, highlighting our goal of not only generating room layouts
but also enabling the LLM to understand layout distributions and perform
layout editing. Next, in section 3.2, we present our approach,  proposing
three strategies for model expansion. Then we detail how to define the input
and output for the LLM model in section 3.2.1 and explain how to construct
effective meta instruction prompts to guide layout generation and editing in
section 3.2.2. Finally, in section 3.2.3 we describe the process of
constructing dialogue data that allows for layout generation and scene
editing, based on the existing 3D-Front dataset.

###  3.1 Problem Formulation

To support LLplace for designing indoor 3D scenes, the system imposes three
requirements on user instruction $\mathcal{I}$: (1) The type of the room
$\bm{T}$, (2) The specific descriptions $\bm{D}_{1\sim N}$ of $\bm{N}$ objects
that to be placed in the room, and (3) The specific quantity $\bm{Q}_{n}$ of
the $\bm{n}$-th object. $\bm{T}$ helps the designer to understand the overall
objective of the room layout, clarifying whether the user intends to design a
bedroom, living room, or another type of space. Descriptions of the objects
$\bm{D}_{1\sim N}$ enable our designer to retrieve the most suitable items
from an existing 3D database. Together with the specified quantities
$\bm{Q}_{n}$ of each item, these descriptions allow the designer to design a
practical room layout. Consequently, the user input can be structured as
$\mathcal{I}=\left\\{\bm{T},\left[\bm{Q}_{n},\bm{D}_{n}\right]_{1\sim
N}\right\\}$.

Based on the user’s description $\bm{D}_{1\sim N}$, we employ a retrieval
module $\bm{R}$ to perform text-to-3D searches within an aligned text-3D
database $\bm{B}$. In addition to the mesh structures, each object in the
database has been transformed into the 3D representation and also includes
bounding box annotations, denoted as $\bm{bbox}_{1\sim M}\in\bm{B}$, where
$\bm{M}$ is a far larger number compared with typical object quantity
$\bm{Q}_{n}$ in an user’s normal request. The retrieved 3D information
$\bm{bbox}_{1\sim N}$ of all requested $\bm{N}$ objects serves as the core of
conditional prompt for the LLplace. The LLM is tasked with generating the
central coordinate $\bm{c}_{1\sim N}$, and rotation angle data $\bm{r}_{1\sim
N}$. Besides, we annotate the meta prompt template of static generation as
$\bm{P}_{gen}$, which is a fixed text wrapper of sorting retrieved
$\bm{bbox}_{1\sim N}$ into a fluent text instruction, and then guiding the LLM
for effective design. The required input and output of LLplace can be denoted
as follows:

$\bm{bbox}_{1\sim N}=\bm{R}(\bm{D}_{1\sim N},\bm{B})$ (1)
$\\{\mathcal{I},[\bm{c}_{n},\bm{r}_{n}]_{1\sim
N}\\}=LLM[\bm{P}_{gen}(\mathcal{I},\bm{bbox}_{1\sim N})]$ (2)

For a single room, there are $\bm{N}$ objects to be placed, which is
formulated as $\\{\bm{bbox}_{n},\bm{c}_{n},\bm{r}_{n}\\}\in Room$. Each
triplet $\\{\bm{bbox}_{n},\bm{c}_{n},\bm{r}_{n}\\}$ uniquely determines the
position of $\bm{n}$-th object in 3D space. Particularly,
$\bm{bbox}_{n}=\\{\bm{h}_{n},\bm{w}_{n},\bm{d}_{n}\\}$, which represents the
bounding box size with height, width, and depth. And
$\bm{c}_{n}=\\{\bm{x}_{n},\bm{y}_{n},\bm{z}_{n}\\}$, which indicates the 3D
coordinate center and represents the centroid coordinates. And $\bm{r}_{n}$ is
the rotation angle. Moreover, as previously mentioned, our LLplace support
further scene editing. Let $\bm{E}$ denotes either adding ($\bm{+}$) or
removing ($\bm{-}$) operation, and
$\mathcal{E}=\left\\{\bm{E},\left[\bm{Q}_{k},\bm{D}_{k}\right]_{1\sim
K}\right\\}$ denotes the new editing request over $\bm{K}$ targeting objects:

$\bm{bbox}_{1\sim K}=\bm{R}(\bm{D}_{1\sim K},\bm{B}|\bm{E}:=\bm{+})$ (3)
$\\{\mathcal{I}\pm\mathcal{E},[\bm{c}_{n\pm k},\bm{r}_{n\pm k}]_{1\sim N\pm
K}\\}=LLM[\bm{P}_{edit}(\\{\mathcal{I},[\bm{c}_{n},\bm{r}_{n}]_{1\sim
N}\\},\mathcal{E},\bm{bbox}_{1\sim K|E:=+})]$ (4)

If the $\bm{k}$-th object is to be removed, the user is allowed to provide a
new description $\bm{D}_{k}$ and the removing request $\bm{E}:=\bm{-}$. To add
objects, newly bounding box $\bm{bbox}_{1\sim K}$ will be additionally
retrieved. The LLM then interactively modifies the existing layout, using the
meta prompt template of editing $\bm{P}_{edit}$ as the text wrapper similar to
the static generation stage. Subsequently, we parse the output
$\\{\mathcal{I},[\bm{c}_{n},\bm{r}_{n}]_{1\sim N}\\}$ of static generation and
the output $\\{\mathcal{I}\pm\mathcal{E},[\bm{c}_{n\pm k},\bm{r}_{n\pm
k}]_{1\sim N\pm K}\\}$ of dynamic editing into layouts, and then render them
into 3D representations $\bm{S}_{3D}^{gen}$ and $\bm{S}_{3D}^{edit}$,
respectively.

Figure 2: The pipeline of LLplace. First, start from the left upper corner, we
extract the room type and the user’s desired objects from the user input.
Then, we retrieve 3D objects and corresponding bounding boxes. Next, we wrap
the user input, bounding box information, and meta prompt $\bm{P}_{gen}$ into
an LLM instruction, as shown in the middle upper box. Using the LoRA fine-
tuned Llama3 model, we obtain the LLM output (the right upper box), which
includes the center coordinates and rotation angles of the objects. We then
combine this output with the input information to convert it into a 3D layout
and render it into a 3D scene (the left bottom corner). To edit the generated
3D scene layout, we combine the previous layout, user input, and edit prompt
$\bm{P}_{edit}$ into a new instruction. The fine-tuned Llama3 model is then
applied to generate the new scene, illustrated at the right bottom.

###  3.2 LLplace

In this section, we present details of building LLplace. We employ the SOTA
open-source model Llama3111https://huggingface.co/meta-llama/Meta-
Llama-3-8B-Instruct as a prior foundation. Building on this, we utilize Low-
Rank Adaptation (LoRA) for fine-tuning, aiming to stimulate and refine the
model’s abilities in 3D world spatial reasoning and configuration parameter-
effectively. We propose the following steps to foster the LLM’s understanding,
generating and editing of 3D spaces: (1) Define Input and Output Format.
Defining the input and output stream of the LLM is a primary task, as
appropriate inputs and outputs can significantly aid the model in more
effectively generating and understanding 3D spatial features; (2) Establish
Meta Prompt Template for Object Placement. Design instructive meta prompt
templates as language wrapper to guide and give reasonable constraints to the
LLM for appropriate object placements; (3) Construct Dialogue-based Training
Data. Modify and augment the existing indoor scene layout dataset to enhance
the LLM’s understanding of spatial positions, and leverage its chatting
talents for further editing functionality of objects through conversational
instructions.

####  3.2.1 Define Input and Output Format

We begin by discussing the input and output specifications of the LLM model
during the layout generation and scene editing process. As mentioned in
Section 3.1, we can use the spatial feature triplet
$\\{\bm{bbox},\bm{c},\bm{r}\\}$ to represent an object in a 3D space, where
bounding box $\bm{bbox}=\\{\bm{h},\bm{w},\bm{d}\\}$, location coordinates
$\bm{c}=\\{\bm{x},\bm{y},\bm{z}\\}$, in specific. In order to provide a
cornerstone for the LLM, when we retrieve an appropriate 3D object, we
determine the $\bm{bbox}$ as an intrinsic feature that is already inherent in
each 3D object, whereas $\bm{c}$ and $\bm{r}$ represent two other properties
that can be freely designed in space. Therefore, in LLplace, we merge the
$\bm{bbox}$ feature as the conditional information with user’s description
$\bm{D}$, wrapped through carefully designed meta prompt templates $\bm{P}$,
to feed into the LLM for the inference of the spatial information $\bm{c}$ and
$\bm{r}$. The left upper corner of Figure 2 illustrates the text-3D retrieval
and instruction packing, resulting in the complete LLM’s instruction within
the orange box at the middle upper part, including the step-by-step task
description and formalized user inputs. We standardize the user textual inputs
with a series of special delimiters and formatted JSON. The room type $\bm{T}$
is wrapped with [Task Room Type] and [/Task Room Type]. And the descriptions
of requested objects $[\bm{D}_{n},\bm{Q}_{n}]_{1\sim N}$ and retrieved
$\bm{bbox}_{1\sim N}$ is correspondingly placed in JSON, attached with [Task
Objects & Bounding Box Size] and [/Task Objects & Bounding Box Size]. We aim
for the model to infer from the user’s request and generate the 3D coordinates
$\bm{c}_{1\sim N}$ and rotation angles $\bm{r}_{1\sim N}$ of objects in the
room space. As demonstrated in the right upper of Figure 2, we guide the model
to also report design plans with JSON format, sorting given priors and
inferred attributes as key-value pairs, and suggesting with special delimiters
[Task Output] and [/Task Output] for easy termination. The “JSON-in” and
“JSON-out” schema enhances the stability of following transcription from the
text output into 3D layout (left bottom corner of Figure 2).

After generating the indoor scene layout, if the resulting scene is
unsatisfactory, the model’s dynamic editing capabilities allow for further
modifications. The input and output text for layout editing are similar to
those used in the generation process, as shown in the bottom of Figure 2. To
add an object $\bm{k}$ to the indoor scene, an additional object description
$\bm{D}_{k}$ is used to retrieve the appropriate 3D object and its bounding
box information $\bm{bbox}_{k}$. The above information, is then formatted into
JSON and enclosed within the special delimiters [Add Objects] and [/Add
Objects]. On the contrary, for deleting objects, the designer does not need to
search for and describe the exact 3D object features. We allow users to
describe the objects using plain natural language. We then convert these
descriptions $\bm{D}_{1\sim K}$ into the format required for removing the
objects and also placing them within special task indicating delimiters
[Delete Objects] and [/Delete Objects].

####  3.2.2 Establish Meta Prompt Template for Object Placement.

Incorporating well-defined prompt instructions into the methodology of LLMs
profoundly influences their capacity to generate and understand spatial
layouts. Except for the aforementioned “JSON-in” and “JSON-out” schema and
tailored component delimiters, we propose several key task statements to guide
the generation of layouts. Here’s how these guidelines are structured, as
briefly demonstrated in the middle top of Figure 2 and completely reported in
the appendix A.3:

  * •

(1) Placement at Room Edges: The most important constraint we introduce at
first is to encourage the placement of objects at the edges of the room
whenever possible. This not only prevents the layout from being too
concentrated in the center of the room but also enhances the perception of
space, making the room appear larger and more open. This strategy is vital for
maximizing the utility of space while maintaining an aesthetically pleasing
arrangement, leading to human-preferred design.

  * •

(2) Avoiding Overlap in Bounding Boxes: It is crucial to explicitly state in
the prompts that the bounding boxes of the generated objects should not
overlap. This ensures that the generated layout maintains both functionality
and visual appeal. By preventing physical interference between objects, we
enhance the usability and aesthetic quality of the space.

  * •

(3) Alignment of Objects: To maintain order and symmetry in the layout, the
prompt should encourage the alignment of objects. This alignment is essential
for aesthetic consistency and functional design, contributing to a harmonious
and efficient environment.

  * •

(4) Setting the Center of 3D Space: To anchor the model’s understanding of
space, we define the center of the room as coordinates (0, 0, 0).

Additionally, to familiarize the model with JSON inputs and outputs without
biasing its generative process, we provide an in-context example that
illustrates the format for layout generation. It is worth noting that this
embedded example is a fixed format illustration, which is not retrieved
through any existing large scale layout database (e.g., LayoutGPT). To enable
the model to perform further layout edits, we incorporate additional task
statements that guide the editing process. Complete prompt templates for
generation and editing, which facilitate targeted adjustments and enhancements
to the generated layouts, are detailed extensively in the appendix A.3.

####  3.2.3 Construct Dialogue-based Training Data

We apply the 3D-Front Fu et al. (2021a) dataset following the previous works
InstructScene, DiffuScene, and LayoutGPT Lin and Mu (2024); Tang et al.
(2023); Feng et al. (2024). We tend to reconstruct the 3D-Front dataset into
two turns of dialogue, involving a first turn of static generation and a
second turn of dynamic editing. We fully leverage the defined input and out
format in section 3.2.1 and established meta prompt templates in section 3.2.2
.

We report our dataset construction algorithm in Alg. 1. In specific, we begin
with extracted objects $[\bm{Q}_{n},\bm{D}_{n}]_{1\sim N}$ according to the
design request, along with their attributes, shown as line 5 to 9. We then
pre-generate complete design input $\bm{input}$ and label $\bm{label}$ with
line 10 and 11. Afterward, we incorporate randomness for editing
functionality, and prepare a corrupted subset ($\bm{input}_{N-K}$,
$\bm{label}_{N-K}$) using line 12 to 16. For creating addition editing data,
we use the subset input $\bm{input}_{N-K}$ and addition objects
$\bm{obj}_{1\sim K}$. As for removing data, we use the complete set
$\bm{input}$ and removing objects $\bm{obj}_{1\sim K}$. Simultaneously, we
comprehensively modify the inputs & outputs of both generation (the first
turn) and editing (the second turn) to contribute a fluent two-round dialogue
serving as our training set, as shown from line 17 to 25. Particularly, the
<|eot_id |> is the end of the turn token of the Llama3 model, which means the
end of one turn conversation. Although dialogue data is quite common in LLM
tasks, we are the pioneers in creating a dialogue dataset specifically for
indoor scene design. By engaging in realistic dialogue scenarios, the model
can learn to respond dynamically to user requests, simulating real-world
interior design consultations.

Finally, each single data sample is annotated with the names of objects in the
room, their corresponding 3D data, spatial coordinates, the size of each
object’s bounding box, and their rotation angles. The descriptions of the
objects are written by GPT-4V and cross-validated by human experts.

Algorithm 1 Construction of the dialogue data from original 3D-front data.

0: The 3D-front data $\bm{OD}$,generation prompt $\bm{P}_{gen}$, and editing
prompt $\bm{P}_{edit}$

0: Construct the dialogue-based generation and editing data with add and
remove operations.

1: Define the room type $\bm{T}$, objects quantity $\bm{Q}$ and objects
description $\bm{D}$

2: Define empty sets $\bm{L}_{gen}$ and $\bm{L}_{edit}$ for collecting
generation data and editing data, respectively

3: for each original data in $\bm{OD}$ do

4: Extract design request
$\mathcal{I}=\left\\{\bm{T},\left[\bm{Q}_{n},\bm{D}_{n}\right]_{1\sim
N}\right\\}$

5: for object in $[\bm{Q}_{n},\bm{D}_{n}]_{1\sim N}$ do

6: $\bm{obj}=[\bm{Q}_{n},\bm{D}_{n}]$

7: Extract each object’s bounding box $\bm{bbox}_{n}$

8: Extract each object’s placement coordinates $\bm{c}_{n}$ and rotation angle
$\bm{r}_{n}$

9: end for

10: Generate the $\bm{input}=\bm{P}_{gen}[\mathcal{I},\bm{bbox}_{1\sim
N}]+\textbf{<|eot\\_id|>}$

11: Generate the $\bm{label}=[\mathcal{I},[\bm{c}_{n},\bm{r}_{n}]_{1\sim N}]$

12: if $n>4$ then

13: Randomly select editable objects $\bm{obj}_{1\sim
K}=[\bm{Q}_{k},\bm{D}_{k}]_{1\sim K}$ from $\bm{input}$ with probability 0.4

14: Ensure essential items (table, chair, sofa, bed, lamp) are not selected

15: Remove $\bm{obj}_{1\sim K}$ and its attributes from $\bm{input}$ and
$\bm{label}$ to get $\bm{input}_{N-K}$ and $\bm{label}_{N-K}$

16: end if

17: if Generate editing data for adding objects then

18: Generate the
$\bm{input}_{edit}=\bm{P}_{edit}(\bm{label}_{N-K},[\bm{obj},\bm{bbox}]_{1\sim
K}|add=True)+\textbf{<|eot\\_id|>}$

19: Generate the $\bm{label}_{edit}=\bm{label}$

20: Update the $\bm{input}:=\bm{input}_{N-K}$ and the
$\bm{label}:=\bm{label}_{N-K}$

21: end if

22: if Generate editing data for removing objects then

23: Generate the
$\bm{input}_{edit}=\bm{P}_{edit}(\bm{label},[\bm{obj},\bm{bbox}]_{1\sim
K}|remove=True)+\textbf{<|eot\\_id|>}$

24: Generate the $\bm{label}_{edit}=\bm{label}_{N-K}$

25: end if

26: Add ($\bm{input}$, $\bm{label}$) data pair to generation set
$\bm{L}_{gen}$

27: Add ($\bm{input}_{edit}$, $\bm{label}_{edit}$) data pair to generation set
$\bm{L}_{edit}$

28: end for

##  4 Experiments

###  4.1 Experiments Setup

Dataset Setup. Our constructed dataset contains a total of 5,754 original
entries. Through the process of random removal and addition, we create an
additional 2,944 removal editing instructions and 3,100 addition editing
instructions. We use the same room test set as ATISS Paschalidou et al. (2021)
and LayoutGPT Feng et al. (2024), consisting of 423 bedroom entries and 53
living room entries. We also use 43 bedroom editing entries and 33 living room
editing entries in the test dataset. We split the rest of the 11,246 data with
566 as the eval set and other data as the training set.

Table 1: The quantitative performance between LLplace and LayoutGPT.

Method | Room Type | FID ↓ | OOR ↓ | GPT-4o Func. ↑ | GPT-4o Layout. ↑ | GPT-4o Aes. ↑
---|---|---|---|---|---|---
LayoutGPT Feng et al. (2024) | Bedroom | 133.7 | 0.078 | 8.0 | 7.5 | 7.2
Livingroom | 171.6 | 0.487 | 7.2 | 7.8 | 7.2
Avg. | 152.7 | 0.283 | 7.6 | 7.65 | 7.2
LLplace | Bedroom | 95.8 | 0.056 | 8.4 | 8.0 | 7.5
Livingroom | 157.8 | 0.472 | 7.6 | 7.6 | 7.2
Avg. | 126.8 | 0.264 | 8.0 | 7.8 | 7.35

Figure 3: The showcases of LLplace in generating the layout of the bedroom and
the living room.

Training Setup. In our training process, due to resource constraints, we adopt
the latest open-source model Llama3-8B-Instruct as the base LLM and use LoRA
Hu et al. (2021) to fine-tune Llama3, for training the LLM’s 3D layout
generation and editing understanding capabilities. Specifically, we set the
LoRA alpha value to 32, the LoRA r value to 8, and the LoRA dropout rate to
0.05. Additionally, we set the learning rate to 1e-4 and use a cosine
scheduler for optimization, training for 20 epochs. All the experiments are
conducted on four NVIDIA A100 X 40G GPUs. It takes 18 hours for entire
training.

Evaluation Metrics. Existing methods do not provide a completely unified set
of evaluation metrics. Therefore, in this paper, we introduce three evaluation
metrics. The first is the FID metric, as used in LayoutGPT Feng et al. (2024),
to assess the consistency between the rendered and real scenes. Next, we
calculate the Object Overlap Rate (OOR) of bounding boxes in the scene to
evaluate the rationality of the generated scene layout. Finally, we use the
GPT-4o model to assess the quality of our rendered results. We use a prompt
template to make GPT-4o evaluate the generated layouts from three
perspectives: Functionality and Activity-based Alignment, Layout and
Furniture, and Aesthetics of the Room’s Layout. Each aspect has a marking
score varying from 0 to 10. To maintain consistency in rendering, we apply the
open-source simple 3D viz renderer for all renderings.

###  4.2 Experiment Results

Quantitative Results. From the OOR values, it is evident that our bounding box
position predictions yield a more reasonable distribution. Despite the OOR
values indicating that the layout generation quality for living rooms still
lags behind that of bedrooms, compared with LayoutGPT, which requires
expensive high-quality in-context exemplars, we lead 2.2% and 1.5% OOR values
in the bedroom and living room scenarios, respectively. The FID values further
demonstrate that our method can produce higher-quality scenes. LLplace
achieves the best generation results for bedrooms, and for living rooms,
obtaining up to 25.9 absolute improvements on average, compared with
LayoutGPT. Furthermore, the evaluation results using GPT-4o show that our
model performs better in rendered scenes for Bedrooms and has a similar
performance to LayoutGPT in living Rooms, across all three evaluation aspects.
In terms of functionality, GPT-4o considers our generated room layouts to be
more practical, marking with a higher 8.0 average score. In terms of layout
rationality and aesthetics, our model still leads LayoutGPT with 0.15 absolute
improvements for both perspectives.

Figure 4: The qualitative results of the LLplace in scene editing.

Qualitative Reports. As shown in Fig. 3, the LLplace can generate reasonable
layouts and understand the general 3D relationship between objects of the
indoor scene. For example, Llplace accurately understand the following spatial
relations: “ _the wardrobe should be on the side of the bed_ ”, “ _chairs
should be placed around the table_ ”, and “ _the TV stand should be placed in
front of the bed or the couch_ ”. We also provide further comparison analysis
of cases presented in Figure 1 in Appendix A.4, across our LLplace and other
strong baselines.

Scene Editing Results. In Figure 4, we demonstrate the scene editing
capabilities of LLplace. As shown in the figure, we can add objects to an
existing scene through language instructions. For example, we added a tall
bookshelf to the living room. LLplace can analyze the relationships between
existing objects and place the additional object in a suitable position. If
the user wants to delete an object, they can do so using simple descriptive
keywords as well. For instance, “ _a TV stand_ ” and “ _one chair_ ” are
removed from the scene at the first row. This proves that LLplace, trained
with dialogue data, possesses scene understanding and editing capabilities
that current models cannot achieve. We also test the FID and OOR scores of
editing random 20 bedroom scenes and 20 living room scenes. As shown in Table
3, LLplace maintains a relatively stable performance even after editing,
losing 16.6 FID average score, which is still better than the 152.7 FID score
of LayoutGPT in Table 1.

###  4.3 Ablation Study

We apply random 50 bedroom scenes to conduct the ablation study. We test the
generation results without adding key task instruction introduced in section
3.2.2 and evaluate the object addition capability of the model trained only
with the original data, excluding dialogue data. We also report the FID and
OOR values . As shown in the upper half of Table 3, the removal of task
instruction leads to a significant performance degradation. Nevertheless,
involving our carefully designed “JSON-in/out” schema and dialogue data during
the training promises a comparable performance compared with former strong
baselines. And the model trained with dialogue data can better understand the
scene and place the added objects in appropriate locations, as reported in the
second row of Table 3.

Table 2: The FID ↓ and OOR ↓ performance
after adding objects in the scenes by LLplace.

Method | Room | FID ↓ | OOR ↓
---|---|---|---
Generation | Bedroom | 95.8 | 0.056
Livingroom | 157.8 | 0.472
Avg. | 126.8 | 0.264
Editing | Bedroom | 114.7 | 0.060
Livingroom | 172.4 | 0.525
Avg. | 143.4 | 0.293

Table 3: The ablation studies of generation without task instruction and
editing without training.

Method |  | FID ↓ | OOR ↓
---|---|---|---
Task Ins.(TI.) | with TI. | 95.8 | 0.056
w/o TI. | 131.5 | 0.103
Dialogue Editing | with training | 114.7 | 0.060
w/o training | 142.1 | 0.072

##  5 Conclusion

In this paper, we propose LLplace, a novel 3D indoor designer. LLplace
utilizes LoRA to fine-tune the Llama3-8B-Instruct LLM to enable both room
layout generation and scene editing through dialogue. Specifically, our model
does not rely on expensive in-context exemplars and spatial relationship
priors as existing approaches insist. Instead, we develop general prompt
templates, then follow the mainstream paradigm of LLMs to extend the 3D-Front
dataset into a dialogue dataset containing one turn of generation and the
other turn of further editing, which manages to foster LLplace with both
static generation and dynamic editing capabilities. Experimental results
demonstrate that LLplace outperforms existing LLM-based indoor scene design
methods across various metrics.

## References

  * Achiam et al. [2023] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al.  Gpt-4 technical report.  _arXiv preprint arXiv:2303.08774_ , 2023.
  * Aguina-Kang et al. [2024] Rio Aguina-Kang, Maxim Gumin, Do Heon Han, Stewart Morris, Seung Jean Yoo, Aditya Ganeshan, R Kenny Jones, Qiuhong Anna Wei, Kailiang Fu, and Daniel Ritchie.  Open-universe indoor scene generation using llm program synthesis and uncurated object databases.  _arXiv preprint arXiv:2403.09675_ , 2024.
  * Çelen et al. [2024] Ata Çelen, Guo Han, Konrad Schindler, Luc Van Gool, Iro Armeni, Anton Obukhov, and Xi Wang.  I-design: Personalized llm interior designer.  _arXiv preprint arXiv:2404.02838_ , 2024.
  * Chen et al. [2023] Sijin Chen, Xin Chen, Chi Zhang, Mingsheng Li, Gang Yu, Hao Fei, Hongyuan Zhu, Jiayuan Fan, and Tao Chen.  Ll3da: Visual interactive instruction tuning for omni-3d understanding, reasoning, and planning.  _arXiv preprint arXiv:2311.18651_ , 2023.
  * Deitke et al. [2022] Matt Deitke, Eli VanderBilt, Alvaro Herrasti, Luca Weihs, Kiana Ehsani, Jordi Salvador, Winson Han, Eric Kolve, Aniruddha Kembhavi, and Roozbeh Mottaghi.  Procthor: Large-scale embodied ai using procedural generation.  _Advances in Neural Information Processing Systems_ , 35:5982–5994, 2022.
  * Fang et al. [2023] Chuan Fang, Xiaotao Hu, Kunming Luo, and Ping Tan.  Ctrl-room: Controllable text-to-3d room meshes generation with layout constraints.  _arXiv preprint arXiv:2310.03602_ , 2023.
  * Feng et al. [2024] Weixi Feng, Wanrong Zhu, Tsu-jui Fu, Varun Jampani, Arjun Akula, Xuehai He, Sugato Basu, Xin Eric Wang, and William Yang Wang.  Layoutgpt: Compositional visual planning and generation with large language models.  _Advances in Neural Information Processing Systems_ , 36, 2024.
  * Fu et al. [2021a] Huan Fu, Bowen Cai, Lin Gao, Ling-Xiao Zhang, Jiaming Wang, Cao Li, Qixun Zeng, Chengyue Sun, Rongfei Jia, Binqiang Zhao, et al.  3d-front: 3d furnished rooms with layouts and semantics.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 10933–10942, 2021a.
  * Fu et al. [2021b] Huan Fu, Rongfei Jia, Lin Gao, Mingming Gong, Binqiang Zhao, Steve Maybank, and Dacheng Tao.  3d-future: 3d furniture shape with texture.  _International Journal of Computer Vision_ , 129:3313–3337, 2021b.
  * Fu et al. [2024] Rao Fu, Jingyu Liu, Xilun Chen, Yixin Nie, and Wenhan Xiong.  Scene-llm: Extending language model for 3d visual understanding and reasoning.  _arXiv preprint arXiv:2403.11401_ , 2024.
  * Gao et al. [2023] Lin Gao, Jia-Mu Sun, Kaichun Mo, Yu-Kun Lai, Leonidas J Guibas, and Jie Yang.  Scenehgn: Hierarchical graph networks for 3d indoor scene generation with fine-grained geometry.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 2023.
  * Ho et al. [2020] Jonathan Ho, Ajay Jain, and Pieter Abbeel.  Denoising diffusion probabilistic models.  _Advances in neural information processing systems_ , 33:6840–6851, 2020.
  * Hu et al. [2021] Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen.  Lora: Low-rank adaptation of large language models.  _arXiv preprint arXiv:2106.09685_ , 2021.
  * Huang et al. [2023a] Jiangyong Huang, Silong Yong, Xiaojian Ma, Xiongkun Linghu, Puhao Li, Yan Wang, Qing Li, Song-Chun Zhu, Baoxiong Jia, and Siyuan Huang.  An embodied generalist agent in 3d world.  _arXiv preprint arXiv:2311.12871_ , 2023a.
  * Huang et al. [2023b] Siyuan Huang, Zan Wang, Puhao Li, Baoxiong Jia, Tengyu Liu, Yixin Zhu, Wei Liang, and Song-Chun Zhu.  Diffusion-based generation, optimization, and planning in 3d scenes.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 16750–16761, 2023b.
  * Kingma and Welling [2013] Diederik P Kingma and Max Welling.  Auto-encoding variational bayes.  _arXiv preprint arXiv:1312.6114_ , 2013.
  * Lin and Mu [2024] Chenguo Lin and Yadong Mu.  Instructscene: Instruction-driven 3d indoor scene synthesis with semantic graph prior.  _arXiv preprint arXiv:2402.04717_ , 2024.
  * Liu et al. [2023] Xiao Liu, Xuanyu Lei, Shengyuan Wang, Yue Huang, Zhuoer Feng, Bosi Wen, Jiale Cheng, Pei Ke, Yifan Xu, Weng Lam Tam, et al.  Alignbench: Benchmarking chinese alignment of large language models.  _arXiv preprint arXiv:2311.18743_ , 2023.
  * Lu et al. [2023] Junru Lu, Siyu An, Mingbao Lin, Gabriele Pergola, Yulan He, Di Yin, Xing Sun, and Yunsheng Wu.  Memochat: Tuning llms to use memos for consistent long-range open-domain conversation.  _arXiv preprint arXiv:2308.08239_ , 2023.
  * Luo et al. [2020] Andrew Luo, Zhoutong Zhang, Jiajun Wu, and Joshua B Tenenbaum.  End-to-end optimization of scene layout.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 3754–3763, 2020.
  * Paschalidou et al. [2021] Despoina Paschalidou, Amlan Kar, Maria Shugrina, Karsten Kreis, Andreas Geiger, and Sanja Fidler.  Atiss: Autoregressive transformers for indoor scene synthesis.  _Advances in Neural Information Processing Systems_ , 34:12013–12026, 2021.
  * Purkait et al. [2020] Pulak Purkait, Christopher Zach, and Ian Reid.  Sg-vae: Scene grammar variational autoencoder to generate new indoor scenes.  In _European Conference on Computer Vision_ , pages 155–171. Springer, 2020.
  * Ritchie et al. [2019] Daniel Ritchie, Kai Wang, and Yu-an Lin.  Fast and flexible indoor scene synthesis via deep convolutional generative models.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 6182–6190, 2019.
  * Tang et al. [2023] Jiapeng Tang, Yinyu Nie, Lev Markhasin, Angela Dai, Justus Thies, and Matthias Nießner.  Diffuscene: Scene graph denoising diffusion probabilistic model for generative indoor scene synthesis.  _arXiv preprint arXiv:2303.14207_ , 2023.
  * Touvron et al. [2023] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al.  Llama: Open and efficient foundation language models.  _arXiv preprint arXiv:2302.13971_ , 2023.
  * Wang et al. [2021] Xinpeng Wang, Chandan Yeshwanth, and Matthias Nießner.  Sceneformer: Indoor scene generation with transformers.  In _2021 International Conference on 3D Vision (3DV)_ , pages 106–115. IEEE, 2021.
  * Yang et al. [2021a] Haitao Yang, Zaiwei Zhang, Siming Yan, Haibin Huang, Chongyang Ma, Yi Zheng, Chandrajit Bajaj, and Qixing Huang.  Scene synthesis via uncertainty-driven attribute synchronization.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 5630–5640, 2021a.
  * Yang et al. [2021b] Ming-Jia Yang, Yu-Xiao Guo, Bin Zhou, and Xin Tong.  Indoor scene generation from a collection of semantic-segmented depth images.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 15203–15212, 2021b.
  * Yang et al. [2024a] Yandan Yang, Baoxiong Jia, Peiyuan Zhi, and Siyuan Huang.  Physcene: Physically interactable 3d scene synthesis for embodied ai.  _arXiv preprint arXiv:2404.09465_ , 2024a.
  * Yang et al. [2024b] Yue Yang, Fan-Yun Sun, Luca Weihs, Eli VanderBilt, Alvaro Herrasti, Winson Han, Jiajun Wu, Nick Haber, Ranjay Krishna, Lingjie Liu, et al.  Holodeck: Language guided generation of 3d embodied ai environments.  In _The IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR 2024)_ , volume 30, pages 20–25. IEEE/CVF, 2024b.
  * Zhang et al. [2023] Lvmin Zhang, Anyi Rao, and Maneesh Agrawala.  Adding conditional control to text-to-image diffusion models.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 3836–3847, 2023.

##  Appendix A Appendix

###  A.1 Details of the Evaluation Metrics

Object Overlap Rate (OOR). The object overlap rate (OOR) metric quantifies the
spatial intersection between a set of 3D bounding boxes. It is calculated by
determining the volume of intersection between each pair of bounding boxes and
then dividing this volume by the smaller volume of the two boxes. This metric
provides a value indicating the degree of overlap, where a value of 0
indicates no overlap and a value of 1 indicates complete embedding of the
smaller box within the larger one. We use the OOR metric to evaluate whether
objects within a 3D scene overlap.

GPT-4o Evaluation. We use an evaluation method similar to that mentioned in
I-Design Çelen et al. [2024], where GPT-4o acts as the evaluator, scoring our
generated room layouts with a maximum score of 10. The prompt template we used
is shown in Table 7.

###  A.2 Objects Retrieval

During inference, we chose to perform object retrieval from the 3D-Future
dataset Fu et al. [2021b]. We use the item description data annotated by
GPT-4V as mentioned in InstuctScene Lin and Mu [2024], and each retrieval was
conducted through text matching. This is because our method does not require
the specific features of 3D objects for training or inference; thus, we opt
for the most efficient and convenient text-matching retrieval method. Each
retrieval yields the file path corresponding to the 3D asset and the bounding
box dimensions of the object, allowing us to include it in the input for scene
generation or editing.

###  A.3 The Prompt Template

In this section, we present all our prompt templates, including the meta
prompt template, the add prompt, and the remove prompt templates. We show the
meta prompt template, the removal prompt template, and the addition prompt
template in Table 4, Table 6, and Table 5, respectively.

As shown in Table 4, our complete meta prompt contains a total of 8 rules.
Besides the four rules mentioned in the main text, the remaining four further
standardize the placement rules for our 3D scene, such as the placement of
chairs and the format of the LLM’s output. In the task instruction Table 6, 5,
we demonstrate in the table how the instructions for editing should be
defined.

###  A.4 More Qualitative Results

In Figure 5, we illustrate more generated results of the bedroom and living
room. These examples demonstrate that our method achieves reasonable results
in generating 3D indoor scenes.

And we compare more generation results with LayoutGPT and GPT-4o+meta prompt
in Figure 6. As shown in Figure 6, our model produces a more reasonable
distribution of 3D objects and outputs reasonable rotation angles. LayoutGPT,
on the other hand, fails to produce occasional overlaps between objects and
issue of object rotation angles, which is a drawback of heavily relying with
high-quality examples. Besides, although GPT-4o does not produce significant
overlap issues, it lacks vertical adaptation on the indoor scene task,
resulting in less plausible room layouts.

Figure 5: The more showcases of LLplace in generating the 3D scene layouts.
Figure 6: The comparison results of the LLplace, LayoutGPT, and GPT-4o + our
meta prompt.

###  A.5 Limitations

Although our method extends the LLM-based 3D scene layout generation task to
allow for scene editing through dialogue, there are still several limitations
that need to be addressed. First, due to resource constraints, we can only
fine-tune lightweight large language models using LoRA. While this leverages
the existing knowledge base of the LLM, the inability to fully fine-tune means
that the LLM’s full potential cannot be activated. Second, our model is
limited to a token length of 2048, restricting it to once or twice turn scene
edits conversation in training and inference. Although we can process the new
output to the whole layout JSON for the next turn conversation, we hope to
apply more length tokens to train and inference. In the future, we can
consider expanding the dataset to construct a larger-scale scene editing
dataset. Additionally, we found that even after data cleaning, some overlap
issues remain in the 3D-Front dataset. Therefore, constructing a larger and
cleaner dataset would further help in advancing the 3D scene layout task.

Table 4: Meta Prompt Template for generation task. Meta Prompt Template for
generation task
---
You are a skilled room layout designer. Your task is to arrange [Objects]
within a given [Room Type] effectively. Follow these guidance to complete your
design:
(1) Extract the [Room Type], [Objects], and [Bounding Box Size] from the
provided JSON data.
(2) Analyze the spatial relationships among [Objects] within the specified
[Room Type]. Pay special attention to avoiding overlap and consider other
spatial factors like accessibility and aesthetics.
(3) Determine and design the precise location of all [Objects] ensuring that
their bounding boxes do not overlap and that the layout is functional and
visually appealing.
(4) I prefer objects to be placed at the edge (the most important constraint)
of the room if possible which makes the room look more spacious.
(5) The objects are usually *aligned*.
(6) Chairs must be placed near to the table/desk and face to the table/desk.
(7) The last design output token is the [/Task Output] and only one.
(8) Report your design with detailed 3D space coordinates and rotation angles
for each object in JSON format, as follows:
{
"object": "object",
"coordinates": [
{
"x": x,
"y": y,
"z": z
}
],
"rotate":[
{
"angle": r
}
]
}
The centroid of the room is {"x": 0.00, "y": 0.00, "z": 0.00"}.
First carefully read this example:

    [Example Room Type]
    Bedroom
    [/Example Room Type]

    [Example Objects and Bounding Box Size]
    /* A fixed example is put here to show the input format*/
    [/Example Objects and Bounding Box Size]

    [Example Output]
    /* A fixed example is put here to show the output format*/
    [/Example Output]

Now, please proceed with the design task as outlined and provide only the JSON
formatted output of your design:

    [Task Room Type]
    /*Input room type*/
    [/Task Room Type]

    [Task Objects & Bounding Box Size]
    /* The JSON format input of objects description
    and bounding box size*/
    [/Task Objects & Bounding Box Size]

Table 5: Addition Prompt Template for LLplace. Addition prompt template in
dialogue data.
---
Following the before layout generation, I need you to add some objects to the
[Task Output] JSON and final output JSON in [Added Output]. Consider the whole
scene layout and design a new place for new objects. The add objects format
is:

    [Add Objects]
    /*Insert the JSON format objects description and
    corresponding bounding box information here.*/
    [/Add Objects]

And the [Added Output] JSON has the same format as the [Task Output] JSON.
This means the output will end at [/Added Output].
Table 6: Removal Prompt Template for LLplace. Removal prompt template in
dialogue data.
---
Following the before layout generation, I need you to delete some objects from
the [Task Output] JSON and give a new output [Deleted Output]. The delete
objects should be formatted as follows:

    [Delete Objects]
    /*Insert the JSON format objects description here.*/
    [/Delete Objects]

And the [Deleted Output] JSON has the same format as the [Task Output] JSON.
This means the output will end at [Deleted Output].
Table 7: GPT-4o Prompt Template for Evaluation. GPT-4o Prompt Template for
Evaluation
---
Give a grade from 0 to 10 to the following room renders based on how well they
correspond together to the user preference (in triple backquotes) in the
following aspects:
\- Functionality and Activity-based Alignment
\- Layout and furniture
\- Aesthetics of the room’s layout
The user preferences:
/*Add the user preferences here.*/
Return the results in the following JSON format:
“{example_json}”