# Learning an Inventory Control Policy with General Inventory Arrival Dynamics

Sohrab Andaz Amazon, New York, NY. Correspondence to<EMAIL_ADDRESS>Carson Eisenach11footnotemark: 1 Dhruv Madeka11footnotemark: 1 Kari
Torkkola11footnotemark: 1 Randy Jia AfreshWork done while at Amazon. Dean
Foster11footnotemark: 1 Sham Kakade11footnotemark: 1 Harvard University,
Cambridge, MA.

###### Abstract

In this paper we address the problem of learning and backtesting inventory
control policies in the presence of general arrival dynamics – which we term
as a quantity-over-time arrivals model (QOT). We also allow for order
quantities to be modified as a post-processing step to meet vendor constraints
such as order minimum and batch size constraints – a common practice in real
supply chains. To the best of our knowledge this is the first work to handle
either arbitrary arrival dynamics or an arbitrary downstream post-processing
of order quantities. Building upon recent work (Madeka et al., 2022) we
similarly formulate the periodic review inventory control problem as an
exogenous decision process, where most of the state is outside the control of
the agent. Madeka et al. (2022) show how to construct a simulator that replays
historic data to solve this class of problem. In our case, we incorporate a
deep generative model for the arrivals process as part of the history replay.
By formulating the problem as an exogenous decision process, we can apply
results from Madeka et al. (2022) to obtain a reduction to supervised
learning. Finally, we show via simulation studies that this approach yields
statistically significant improvements in profitability over production
baselines. Using data from an ongoing real-world A/B test, we show that Gen-
QOT generalizes well to off-policy data.

##  1 Introduction

The periodic review inventory control problem is that of determining how much
inventory to hold in order to maximize revenue. This problem has been studied
extensively in the operations research literature, and is often formulated as
a Markov decision process (Porteus, 2002). Some complexities involved include
stochastic demands with unknown seasonality, lost sales, stochastic vendor
lead times, multiple shipments per order, unreliable replenishment, and order
quantity restrictions. Classical approaches are typically able to handle only
a subset of these complexities due to the curse of dimensionality and the fact
that closed form solutions are not available as the problem setting increases
in complexity. In fact, base stock policies (which are optimal for simplified
settings and often used in practice) have been shown to perform worse than
constant order policies in the presence of lost sales and stochastic demand
(Zipkin, 2008).

Madeka et al. (2022) recently introduced the first Deep Reinforcement Learning
periodic review inventory system which was able to handle many of the
aforementioned challenges. By formulating inventory control as an exogenous
decision process (Sinclair et al., 2023), Madeka et al. (2022) demonstrate a
reduction in complexity of the learning problem to that of supervised
learning. They also show how censored (unobserved) historic data can be used
for policy learning and backtesting by constructing a simulator that replays
historic data rather than assuming a parametric form in order to simulate the
future.

One of the assumptions Madeka et al. (2022) does maintain from the classical
inventory control literature, however is the structure of inventory arrivals
given an order quantity provided to the vendor. The authors handle the case of
stochastic lead times (with unknown distribution), but in real-world settings
there are several additional complexities that can arise. First, a single
order placed to a vendor may arrive in multiple shipments at different future
periods111To the best of our knowledge, no existing work considers this
setting.. Figure 1 shows prototypical distributions of number of shipments and
inter-arrival times of purchase orders made by a large e-retailer.

(a) Number of distinct shipments per order

(b) Inter-arrival times for the same order

Figure 1: Orders often arrive in multiple shipments, and spread out over
multiple time periods.

Second, the supply may be unreliable and vendors may only partially fill
orders that they receive – for example an order for 100 units of an item may
result in only 75 units being supplied. This may occur for multiple reasons,
including that the vendor itself is out of stock. In the literature the
proportion of the original order quantity retailer ultimately receives is
referred to as the yield or fill rate. Figure 2(a) shows a prototypical
distribution of yields at a large e-retailer.

(a) Observed distribution of supplier fill rates

(b) Overall yields, including any post-processing

Figure 2: Example of yields – multiplicative proportion of order quantity that
is received – for purchase orders at a large e-retailer, both before and after
any post-processor is applied to the order quantity.

Finally, the order quantity dictated by the policy may not meet the
requirements of the vendor and may need to be rounded before a purchase order
is sent: for example minimum order quantities or batch size restrictions222For
example, order quantities may need to be a multiple of case sizes or other
unit of packaging are quite common (Zhu et al., 2015). The impact of such
restrictions is difficult to analyze, is not well understood, and heuristic
rules are typically employed (Robb and Silver, 1998; Zhu et al., 2015; Zhou et
al., 2007). In real-world settings, this may be performed as a secondary post-
processing step after the optimal order quantity has been determined – for
example by rounding up to the minimum order quantity (Kiesmüller et al., 2011;
Zhu, 2022). Figure 2(b) shows the distribution of “end-to-end” yields –
including both supply uncertainty and any post-processing steps to meet batch
ordering or minimum order quantity constraints applied by ordering systems –
for replenishment decisions at a large e-retailer. Note how in many cases, the
“end-to-end” yield can be greater than the original order quantity due to the
presence of a post-processing step.

#### Our Contributions and Organization

In this paper we address the problem of learning and backtesting inventory
control policies in the presence of general arrival dynamics (which we term as
a quantity-over-time model or QOT). We also allow for order quantities to be
modified as a post-processing step to meet vendor constraints such as order
minimum and batch size constraints. To the best of our knowledge this is the
first work to handle either arbitrary arrival dynamics or an arbitrary
downstream post-processing of order quantities – a common practice in real
supply chains. Figure 3 illustrates the difference in cumulative arrivals over
time under the different types of arrival dynamics.

Figure 3: Cumulative arrivals over time for a single purchase order under
differing dynamics.

The remainder of this paper is organized as follows: in Section 3 we formulate
our problem as an exogenous interactive decision process and leverage results
from Madeka et al. (2022) to demonstrate a reduction to supervised learning.
We also describe our approach to modeling the arrival dynamics for use as part
of a simulator that replays historic data (Madeka et al., 2022). Next, in
Section 4.1 we evaluate the performance of the learned dynamics model on “on-
policy” data – that is, data generated by the same policy that generated the
data used to fit the model. Then, Section 4.2 we demonstrate via backtests the
impact a realistic arrivals model has on policy performance. Finally, in
Section 4.3, results from a large real-world A/B test in the supply chain of a
large e-retailer show that (1) the RL policy learned using our approach can
outperform classic approaches to periodic review inventory control, and (2)
our learned dynamics model generalizes well to “off-policy” data which
validates our assumption that we have an accurate forecast of transitions
under the learned policy.

##  2 Related Work

###  2.1 Forecasting and Generative Modeling

All work which considers stochastic vendor lead times implicitly requires a
forecast of lead times (even if the lead time distribution is assumed to be
stationary). We have not found much study of forecasting lead times
specifically, but approaches from the probabilistic time series forecasting
community can be used to forecast lead times. There is an extensive body of
work which has successfully applied deep learning to time series forecasting,
including in the supply chain for forecasting demand (Nascimento et al., 2019;
Yu et al., 2017; Gasparin et al., 2019; Mukhoty et al., 2019; Wen et al.,
2017; Salinas et al., 2020; Wen and Torkkola, 2019; Eisenach et al., 2020;
Madeka et al., 2018).

#### Deep Generative Modeling

A number of deep generative techniques have been developed to estimate the
likelihood of observations in training data and generate new samples from the
underlying data distribution. These include generative adversarial networks
(Goodfellow et al., 2014), variational auto-encoders (Kingma and Welling,
2013, 2019), and autoregressive models. Autoregressive models have been used
successfully in image generation (Van Oord et al., 2016), NLP (Brown et al.,
2020; Devlin et al., 2019), and time-series forecasting (Madeka et al., 2018;
Eisenach et al., 2020; Januschowski et al., 2022; Wen et al., 2017; Salinas et
al., 2020; Lim et al., 2019). Our work employs autoregressive modeling by
decomposing the full problem of estimating the joint distribution of arrivals
into the simpler problem of merely predicting the next arrival in a sequence
given the previously realized shipments.

###  2.2 Reinforcement Learning and Exogenous Sequential Decision Problems

Reinforcement learning has been applied to many sequential decision-making
problems including games and simulated physics models where many simulations
are possible (Silver et al., 2016; Szepesvári, 2010; Mnih et al., 2013; Sutton
and Barto, 2020; Schulman et al., 2017; Mnih et al., 2016). In general, one
can require exponentially many samples (in the horizon of the problem) to
learn a control policy.

Recently, Madeka et al. (2022) and Sinclair et al. (2023) considered a class
of problems called exogenous interactive decision processes wherein the state
consists of a stochastic exogenous process (independent of the control) and an
endogenous component that is governed by a known transition function $f$ of
both the previous endogenous state and the exogenous process. In these cases,
Madeka et al. (2022); Sinclair et al. (2023) prove a reduction in sample
complexity to that of supervised learning – an exponential improvement over
the general RL setting.

###  2.3 Periodic Review Inventory Systems

Inventory control systems have been studied extensively in the literature
under a variety of conditions (see Porteus (2002) for a comprehensive
overview). The simplest form is the newsvendor, which solves a myopic problem
(Arrow et al., 1958). Many extensions exist (Nahmias, 1979; Arrow et al.,
1958), and the optimal policy in many variants takes the form of a base stock
policy which, informally, consists of an optimal inventory level and then
orders up to that level. However, outside the restrictive conditions under
which optimal policies can be derived, Zipkin (2008) showed that even constant
order policies can be better than base stock policies.

#### RL for Inventory Control

More recently, several authors have applied reinforcement learning to solve
multi-period inventory control problems (Giannoccaro and Pontrandolfo, 2002;
Das et al., 1999; Balaji et al., 2019; Madeka et al., 2022). We adopt the
modeling approach of Madeka et al. (2022), and similarly treat the periodic
review inventory control problem under general arrival dynamics as an
exogenous decision process. Also following Madeka et al. (2022), we build a
differentiable simulator (Suh et al., 2022; Hu et al., 2019; Ingraham et al.,
2018; Clavera et al., 2020) using historical supply chain data, and
successfully train and backtest an RL agent to achieve improved real world
performance over traditional methods.

#### Lead Times and Supply Reliability

There has been extensive work in the literature on how to handle stochastic
lead times in periodic review inventory systems (Porteus, 2002; Song and
Zipkin, 1996; Maggiar et al., 2022; Nahmias, 1979; Kaplan, 1970). In addition,
unreliable supply has been studied in a variety of settings. One form this
often takes is a stochastic yield (Maddah and Jaber, 2008; Li et al., 2004;
Henig and Gerchak, 1990; Gerchak et al., 1988; Maggiar et al., 2022;
Bollapragada and Morton, 1999), for example via some portion of the supply
being defective. A common heuristic is to simply adjust the order quantity for
the mean yield (Bollapragada and Morton, 1999). The second standard
formulation is to have the fill rate level be determined by the number of
units the vendor has available, where vendors fill up to the amount of units
they have available (Dada et al., 2007). The first setting is more common as
it is more tractable to find analytic solutions. We are unaware of any work
that considers the impact of a varying number of shipments per order placed
with the supplier.

In this work, we take the following approach: we assume that the vendor fills
orders up to their capacity (which is exogenous to the retailer’s
replenishment decisions), but we add an arrival shares process which encodes
how the filled amount arrives over future time periods.

#### Minimum Order Quantities, Maximum Order Quantities and Batch Ordering

A minimum order quantity or maximum order quantity constraint means that the
supplier will reject all orders under or over that amount, respectively. For
minimum order quantity constraints, we only consider the setting where the
retailer can also choose to order 0 (i.e. not to order) as otherwise the
minimum quantity constraint is unimportant. Similarly, a batch ordering
requirement means that the supplier only accepts order quantities that are an
integer multiple of some specified batch quantity. Maximum constraints have
been studied extensively in the literature and typically affect the policy a
in benign way (Chan and Muckstadt, 1999; Federgruen and Zipkin, 1986). On the
other hand, minimum order quantities (Fisher and Raman, 1996; Zhou et al.,
2007; Zhao and Katehakis, 2006; Shen et al., 2019) and batch requirements
(Veinott, 1965; Zhu et al., 2015) – although widely adopted by suppliers in
practice (Zhu, 2022) – are relatively unstudied in the literature as they
present significant difficulties in deriving the optimal order quantity. For
minimum order quantities, even in highly simplified settings, optimal policies
are only partially characterized and are too complicated to implement in
practice (Zhao and Katehakis, 2006). Instead, retailers use base stock
policies with a heuristic rounding before placing the order (Zhou et al.,
2007; Kiesmüller et al., 2011).

##  3 Mathematical Formulation and Methodology

In this section, we follow the Interactive Decision Process (IDP) formulation
of Madeka et al. (2022), borrowing most of the conventions and notation,
except we define and treat the “lead time” process differently. At a high
level, a central planner is trying to determine how many units of inventory to
order at every time step $t=1,2,...,T$, in order to satisfy demands $D_{t}$.
The goal is to maximize profits by balancing having enough inventory on hand
to satisfy demand (we assume a lost sales customer model), with the cost of
holding too much inventory.

The dynamics we are most concerned with is how order quantities selected by
the policy evolve into future arrivals at the retailer’s warehouse. The
standard formulation in the literature is the vendor lead time (VLT) arrivals
model, whereupon placing an inventory order decision $a_{t}$ at time $t$, a
single quantity $v_{t}$ is drawn from an exogenous lead time distribution, and
the entire order arrives $v_{t}$ time steps later at time $t+v_{t}$. In the
case of stochastic yields, there are two approaches in the literature: either
the yield is a random multiplicative factor multiplied times the order
quantity, or the vendor has a stochastic supply and fills up to the amount of
their supply. Where we depart from Madeka et al. (2022) is that our
formulation allows that

  1. 1.

inventory can arrive in multiple shipments for a single order,

  2. 2.

yields can be stochastic, and

  3. 3.

there can be a downstream system that applies a heuristic order quantity
rounding to satisfy batch, minimum and maximum quantity constraints.

We propose a novel quantity over time (QOT) arrivals model, which generalizes
all the settings described above. In the QOT arrivals model, we assume that
orders can arrive in multiple shipments over time, and the total arriving
quantity may not necessarily sum up to the order quantity placed. At every
time $t$, the vendor has allocated a supply $U_{t}$ that denotes the maximum
number of units it can send (regardless the amount we order), which will
arrive over from the current week up to $L$ weeks in the future according to
an exogenous arrival shares vector $(\rho_{t,0},...,\rho_{t,L})$ where
$\sum_{l}\rho_{t,l}=1$. That is, the arrivals at lead time $j$ from order
$a_{t}$ is equal to $\min(U_{t},a_{t})\rho_{t,j}$. We denote the arrival
quantity as $o_{t,j}:=\min(U_{t},a_{t})\rho_{t,j}$. Here, we implicitly assume
that orders necessarily arrive after $L$ time steps.

We also allow for the existence of an order quantity post-processor $f_{p}$
that is arbitrary (but known) that modifies order quantities before they are
sent to the supplier – e.g. to ensure they meet any vendor constraints.

###  3.1 Mathematical notation

Denote by $\mathbb{R}$, $\mathbb{R}_{\geq 0}$, $\mathbb{Z}$, and
$\mathbb{Z}_{\geq 0}$ the set of reals, non-negative reals, integers, and non-
negative integers, respectively. Let $[\;\cdot\;]$ refer to the set of
positive integers up to the argument, i.e.
$[\;\cdot\;]=\\{x\in\mathbb{Z}\,|\,1\leq x\leq\cdot\;\\}$. The inventory
management problem seeks to find the optimal inventory level for each product
$i$ in the set of retailer’s products, which we denote by $\mathcal{A}$. We
assume our exogenous random variables are defined on a canonical probability
space $(\Omega,\mathcal{F},\mathbb{P})$, and policies are parameterized by
$\theta$ in some parameter set $\Theta$. We use $\mathbb{E}^{\mathbb{P}}$ to
denote an expectation operator of a random variable with respect to some
probability measure $\mathbb{P}$. Let $||X,Y||_{TV}$ denote the total
variation distance between two probability measures $X$ and $Y$.

###  3.2 IDP Construction

Our IDP is governed by external (exogenous) processes, a control process,
inventory evolution dynamics, and a reward function. To succinctly describe
our process, we focus on just one product $i\in\mathcal{A}$, though we note
that decisions can be made jointly for every product.

##### External Processes

At every time step $t$, for product $i$, we assume that there is a random
demand process $D_{t}^{i}\in[0,\infty)$ that corresponds to customer demand
during time $t$ for product $i$. We also assume that the random variables
$p_{t}^{i}\in[0,\infty)$ and $c_{t}^{i}\in[0,\infty)$ are the random variables
corresponding to selling price and purchase cost. We also assume any
constraints the vendor imposes on the retailer’s orders
$\bm{M}^{i}_{t}\in\mathbb{R}^{d_{v}}$ – such as minimum order quantities and
batch sizes – are exogenous to the ordering decisions. The supply
$U^{i}_{t}\in[0,\infty)$ corresponds to the maximum amount of inventory the
vendor is able to send. Finally, the arrival shares process
$\bm{\rho}^{i}_{t}:=\\{\rho^{i}_{t,j}\\}_{j=0}^{L}$ describes the arrivals
over the next $L$ time steps from an order placed at the current time $t$ –
note that $\sum_{j=0}^{L}\rho^{i}_{t,j}=1$ and $\rho^{i}_{t,j}>0$ for all $i$,
$t$, and $j$. Our exogenous state vector for product $i$ at time $t$ is all of
this information:

$s^{i}_{t}=(D_{t}^{i},p_{t}^{i},c_{t}^{i},U^{i}_{t},\bm{M}^{i}_{t},\bm{\rho}^{i}_{t}).$

To allow for the most general formulation possible, we consider policies that
can leverage the history of all products. In our implementation, however, we
learn a policy that only uses the history of that product and for our
learnability results in Section 3.3 we will assume independence of the
processes between products. Therefore, we will define the history

$H_{t}:=\\{(s_{1}^{i},...,s_{t-1}^{i})\\}_{i=1}^{|\mathcal{A}|}$

as the joint history vector of the external processes for all the products up
to time $t$.

##### Control Processes

Our control process will involve picking actions for each product jointly from
a set of all possible actions $\mathbb{A}:=\mathbb{R}_{\geq
0}^{|\mathcal{A}|}$. For product $i$, the action taken is denoted by
$a_{t}^{i}\in\mathbb{R}_{\geq 0}$, the order quantity for product $i$. For a
class of policies parameterized by $\theta$, we can define the actions as

$a_{t}^{i}=\pi_{\theta,t}^{i}(H_{t}).$

We characterize the set of these policies as
$\Pi=\\{\pi_{\theta,t}^{i}|\theta\in\Theta,i\in\mathcal{A},t\in[0,T]\\}.$

##### Order Quantity Constraints

We allow for the existence of an order quantity post-processer
$f_{p}:\mathbb{R}_{\geq 0}\times\mathbb{R}^{d_{v}}\rightarrow\mathbb{R}_{\geq
0}$ that may modify the order quantity – for example to satisfy the
constraints given by $\bm{M}^{i}_{t}$. The final order quantity requested from
the vendor is denoted as
$\widetilde{a}^{i}_{t}:=f_{p}(a_{t}^{i},\bm{M}^{i}_{t})$. Note that we do not
impose a requirement that any constraints encoded in $\bm{M}_{t}^{i}$ be
satisfied, merely that we permit the endogenous portion of the state’s
evolution to depend on such a function as they appear so often in real supply
chains.

##### Inventory Evolution Dynamics

We assume that the implicit endogenous inventory state follows standard
inventory dynamics and conventions. Inventory arrives at the beginning of the
time period, so the inventory state transition function is equal to the order
arrivals at the beginning of the week minus the demand fulfilled over the
course of the week. Both demand and arrivals may be censored due to having
lower inventory on-hand or vendor having low supply, respectively. The amount
arriving, according to our model of arrivals is:

$I^{i}_{t_{-}}=I^{i}_{t-1}+\sum_{j=0}^{L}\min(U^{i}_{t-j},\widetilde{a}^{i}_{t-j})\rho^{i}_{t-j,j},$
(3.1)

where $I_{t}^{i}$ is the inventory at the end of time $t$, and $I_{t_{-}}^{i}$
is the inventory at the beginning of time $t$, after arrivals but before
demand is fulfilled. Then, at the end of time $t$, the inventory position is:

$I^{i}_{t}=\min(I^{i}_{t_{-}}-D^{i}_{t},0).$

##### Reward Function

The reward at time $t$ for product $i$ is defined as the selling price times
the total fulfilled demand, less the total cost associated with any newly
ordered inventory (that will be charged by the vendor upon delivery):

$R_{t}^{i}=p_{t}^{i}\min(D^{i}_{t},I^{i}_{t_{-}})-c_{t}^{i}\min(U^{i}_{t},\widetilde{a}_{t}^{i}).$
(3.2)

Note that the cost charged is the realized order quantity, which is the
standard practice in the literature.

We will write $R_{t}(H_{t},\theta)$ to emphasize that the reward is a function
only of the exogenous $H_{t}$ and the policy parameters $\theta$. Recall that
selling price and buying cost are determined exogenously. We assume all
rewards $R_{t}^{i}\in[R^{min},R^{max}]$, and assume a multiplicative discount
factor of $\gamma\in[0,1]$ representing the opportunity cost of reward to the
business. Again, we make the dependence on the policy explicit by writing
$R_{t}^{i}(\theta)$. The objective is to select the best policy (i.e., best
$\theta\in\Theta$) to maximize the total discounted reward across all
products, expressed as the following optimization problem:

$\displaystyle\max_{\theta}$
$\displaystyle~{}\mathbb{E}_{\mathbb{P}}[\sum_{i\in\mathcal{A}}\sum_{t\in[0,T]}\gamma^{t}R_{t}^{i}(\theta)]$
(3.3) subject to: $\displaystyle I^{i}_{0}$ $\displaystyle=k^{i}$
$\displaystyle a^{i}_{t}$ $\displaystyle=\pi_{\theta,t}^{i}(H_{t})$
$\displaystyle\widetilde{a}^{i}_{t}$
$\displaystyle=f_{p}^{i}(a^{i}_{t},H_{t})$ $\displaystyle I^{i}_{t_{-}}$
$\displaystyle=I^{i}_{t-1}+\sum_{j=0}^{L}\min(U^{i}_{t-j},\widetilde{a}^{i}_{t-j})\rho^{i}_{t-j,j},$
(3.4) $\displaystyle I^{i}_{t}$
$\displaystyle=\min(I^{i}_{t_{-}}-D^{i}_{t},0).$

Here, $\mathbb{P}$ denotes the joint distribution over the exogenous
processes. The inventory $I_{0}^{i}$ is initialized at $k_{i}$, a known
quantity _a priori_.

###  3.3 Learnability

#### Learning Objective

For the policy to be efficiently learnable, we need to restrict the policy for
product $i$ at time $t$ to be a function only of the history of item $i$,
$H^{i}_{t}:=\\{(s_{1}^{i},...,s_{t-1}^{i})\\}$, and the learnable parameter
$\theta$ is shared by all item’s policies. The reward is therefore now a
function $R_{t}(H^{i}_{t},\theta)$ of only the history of item $i$ and the
parameter $\theta$. The learning objective then becomes

$J_{T}(\theta):=\mathbb{E}[\sum_{i\in\mathcal{A}}\sum_{t\in[0,T]}\gamma^{t}R_{t}^{i}(\theta)],$

which we estimate via simulation with the objective

$\widehat{J}_{T}(\theta):=\sum_{i\in\mathcal{A}}\sum_{t\in[0,T]}\gamma^{t}R_{t}^{i}(\theta).$

This is clearly an unbiased estimate of $J_{T}$ as the historical data $H_{T}$
is exogenous to the choice of policy.

#### Learnability

Our problem formulation fits under the framework described in Madeka et al.
(2022), with additional exogenous variables $U_{t}$, $\bm{\rho}_{t}$, and
$\bm{M}_{t}$ as part of the external state process. Hence, assuming full
observability of these processes, we can accurately simulate the value of any
policy. This follows immediately from Theorem 2333The addition of the post-
processor $f_{p}$ does not impact the result. of Madeka et al. (2022) as we
assume that the supply, arrival shares, and vendor constraint processes are
exogenous.

In reality, one does not fully observe the supply of the vendors ($U^{i}_{t}$)
or the arrival shares ($\bm{\rho}^{i}_{t}$). The supply of the vendor
$U_{t}^{i}$ is only observed historically at times when the vendor did not
fully fill the order – in which case, $U_{t}^{i}=\sum_{j=0}^{L}o_{t}^{j}$. The
arrival shares $\bm{\rho}^{i}_{t}$ are fully observed historically whenever an
order is placed and the vendor sends at least one unit.

To proceed we require a way to obtain these missing values. First, for the
supply $U_{t}^{i}$, the retailer could collect this data by asking vendors to
share how many units they are able to supply444This problem is little studied
in the literature, but in some scenarios it is known that the supplier is
always at least as well off if they share capacity information (Bao, 2006).
Another approach is to treat this as a missing data problem, and then use
additional exogenous observed context $x_{t}^{i}\in\mathbb{R}^{D}$ that is
available at time $t$ for product $i$, to forecast these unobserved
components. The observed context history is denoted as
$X_{T}^{i}:=(x_{1}^{i},\dots,x_{T}^{i})$. The latter is similar to the
uncensoring of demand in Madeka et al. (2022).

######  Assumption 3.1 (Accurate Forecast of Supply and Arrival Shares).

Let
$H^{i}_{T,F}:=(U^{i}_{1},\bm{\rho}^{i}_{1},\dots,U^{i}_{T},\bm{\rho}^{i}_{T})$
denote the history of the unobserved exogenous supply and arrival shares
processes through time $T$. Likewise, denote the observed components of the
exogenous history $H^{i}_{T}$ as $H^{i}_{T,O}$. Now, we can consider the
distributions
$\mathbb{P}_{F}^{i}:=\mathbb{P}(H^{i}_{T,F}|H^{i}_{T,O},X^{i}_{T})$ and
$\widehat{\mathbb{P}}_{F}^{i}:=\widehat{\mathbb{P}}(H^{i}_{T,F}|H^{i}_{T,O},X^{i}_{T})$.
If

$\frac{1}{|\mathcal{A}|}\sum_{i\in\mathcal{A}}||\widehat{\mathbb{P}}_{F}^{i},\mathbb{P}_{F}^{i}||_{TV}\leq\epsilon_{F},$

we call $\widehat{\mathbb{P}}_{F}^{i}$ an accurate forecast of
$\mathbb{P}_{F}^{i}$.

Under 3.1, it follows from Theorem 322footnotemark: 2 of Madeka et al. (2022)
that the inventory control problem with general arrivals is efficiently
learnable in the case where we do not observe the supply and arrival shares
processes. In practice, we may choose to forecast arrivals instead of the
supply and arrival shares processes – see Remark 3.2 below.

######  Remark 3.2 (Forecasting Arrivals).

Note that the dynamics (3.4) and reward function (3.2) depend only on the
arrivals
$o_{t,j}^{i}:=\min(U^{i}_{t},f_{p}(a_{t}^{i},\bm{M}^{i}_{t}))\rho^{i}_{t,j}$,
so we forecast arrivals conditional on the action $a_{t}^{i}$ rather than the
supply and fill processes for the purposes of constructing our simulator from
historic data.

###  3.4 Modeling arrivals with Gen-QOT

Having established that our problem of interest is efficiently learnable, we
proceed with describing the QOT model and then evaluating the model. Per
remark 3.2, we forecast the arrival sequence directly rather than the supply
and arrival processes. Formally, we forecast the distribution

$p(o^{i}_{t,0},\dots,o^{i}_{t,L}|H_{t,O}^{i},X_{t}^{i}).$

The model is trained to minimize log-likelihood of the forecasted
distribution. See Appendix D for a complete description of the model and
training objective. It is worth emphasizing that modeling arrivals directly
allows us to treat $f_{p}$ as part of the forecast of the arrivals process,
which will be advantageous in Section 3.5 when we describe how to make the
simulator differentiable.

Figure 4 shows a set of sample paths generated from our Gen-QOT model
alongside a set of real order-quantity normalized inventory arrivals.

Figure 4: A set of 256 real and simulated sample paths of order-quantity
normalized inventory arrivals.

A natural question then arises: how should one measure the quality of the
generated sample paths? The simplest thing would be to compare against other
methods for forecasting vendor lead times (this will be referred to as
Criterion 1). In addition, we might want to check that the generated paths
satisfy several desirable properties that we anticipate may be relevant to the
inventory control problem:

  1. 1.

Criterion 2 – Does Gen-QOT predict the right amount of cumulative inventory
$l$ weeks after an order is placed?

  2. 2.

Criterion 3 – Does Gen-QOT predict receiving zero inventory for the correct
orders?

  3. 3.

Criterion 4 – Does Gen-QOT predict correctly whether there is an arrival in
the first week after an order is placed?

Accordingly, we propose four methods for evaluating the quality of the
dynamics model:

  1. 1.

Criterion 1 – Obtain the empirical distribution predicted by Gen-QOT and
evaluate against with standard accuracy metrics such as CRPS and quantile
loss.

  2. 2.

Criterion 2 – Regress the actual cumulative fill rate on the mean predicted
cumulative fill rate for each week.

  3. 3.

Criterion 3 and 4 – Generate sequences and use as a classifier, then construct
a classifier calibration plots for generated sequences.

In Section 4.1 we fit the Gen-QOT model to historic data and perform
evaluations against the 4 criterion listed above.

###  3.5 Simulator Construction

Our simulator is constructed similarly to Madeka et al. (2022) aside from the
change to the inventory transition dynamics. To handle the general arrival
dynamics, the first approach is to estimate the partially observed
$H^{i}_{t,F}$ and directly implement (3.4) since $f_{p}$ is known.
Alternatively, following Remark 3.2, we could forecast arrivals and simply
sample that distribution at each step. The issue with both these approaches
are that the simulator is no longer path-wise differentiable and if possible,
we would prefer to leverage the fact that most of our dynamics are
differentiable and thus the exact gradient can be computed analytically.

The approach we take in our empirical work is the following: first, given the
action $a_{t}$ and the exogenous $H_{t,O}^{i}$, $X_{t}^{i}$, sample the
estimated forward model

$\widehat{o}^{i}_{t,0},\dots,\widehat{o}^{i}_{t,L}\sim\widehat{p}(\cdot|H_{t,O}^{i},X_{t}^{i},a_{t}).$
(3.5)

The sampled arrivals can then be converted into a sampled sequence of partial
fills $\bar{\alpha}^{i}_{t,l}=\frac{\widehat{o}^{i}_{t,l}}{a^{i}_{t}}$ by
rescaling by the action $a_{t}^{i}$. The inventory update in (3.4) becomes

$I^{i}_{t_{-}}=I^{i}_{t-1}+\sum_{j=0}^{L}\bar{\alpha}^{i}_{t-j,j}a^{i}_{t-j},$
(3.6)

This is similar in spirit to the approach in Clavera et al. (2020), except
they use the re-parametrization trick and then differentiate through the
forward model, passing a noise process as input.

##  4 Empirical Results

In this section we present some empirical results. First, in Section 4.1 we
evaluate the performance of the fitted Gen-QOT under the four criterion
discussed in Section 3.4. Then in Section 4.2 we demonstrate through backtests
against historical data in a simulator based on the Gen-QOT model the
difference in performance between baseline policies, an RL trained in a
simulator with Gen-QOT dynamics and an RL trained in dynamics with a classical
vendor lead-time model. Finally, in Section 4.3 we show recent results from a
real-world A/B test of the RL policy in the US store of a large e-retailer.

###  4.1 Evaluating the Gen-QOT Model

First, we evaluate our proposed Gen-QOT model as we require an accurate
forecast in order for the policy backtest to be valid. The Gen-QOT model
architecture and training objective can be found in Appendix D.

#### Training and Evaluation Data

We train Gen-QOT on purchase orders from 250K products from the US store of a
large e-retailer from 2017-05-13 to 2019-03-23 and holdout 100K actions from
2019-03-23 to 2020-03-21 to evaluate model performance. This time period
allows us to judge both in-time and out-of-time time generalization. The
features used in our model can be found in Appendix C.

#### Results

##### VLT Forecasting

Because many inventory control systems rely on simplified optimization models
that assume only random lead-time, we evaluate the Gen-QOT model against one
that directly predicts quantiles of the vendor lead-time distribution. The
architecture used is similar to Gen-QOT, but we replace the recurrent neural-
net decoder with a simple multi-layer perceptron.555More precisely, the
architecture is MQCNN which achieve SOTA performance on probabilistic
forecasting tasks that use very similar data – see “MQ_CNN_wave” in Figure 3
of Wen et al. (2017).

To produce a forecast of the vendor lead time from Gen-QOT, samples are
generated from Gen-QOT to obtain an empirical distribution from which
quantiles can be determined. Table 1 shows the backtest results on data from
2019-03-23 to 2020-03-21, showing that Gen-QOT is competitive with traditional
vendor lead time forecasting approaches. See Appendix A for definitions of the
CRPS and Quantile Loss metrics used.

Table 1: Backtest of generative model versus direct quantile forecast (lower values are better); 95% confidence intervals are on the performance gap between the two models. | Model |
---|---|---
Metric | Direct Prediction | Gen-QOT | 95% CI
CRPS | 100.00 | 101.61 | [-0.23, 3.46]
P10 QL | 100.00 | 99.92 | [-1.91, 1.75]
P30 QL | 100.00 | 101.24 | [-0.57, 3.04]
P50 QL | 100.00 | 102.41 | [0.38, 4.44]
P70 QL | 100.00 | 103.21 | [0.89, 5.54]
P90 QL | 100.00 | 102.22 | [0.50, 4.96]

See Section B.1 for an ablation study across different candidate architectures
for Gen-QOT comparing their performance under the CRPS and Quantile Loss
metrics.

##### Calibration and Classifier Metrics

Next we check the calibration of the cumulative receives forecast that can be
inferred from Gen-QOT – see Section 3.5 for how we define calibration. Table 2
shows the calibration of Gen-QOT’s forecasted distributions of cumulative
arrivals. Most coefficients are close to one, showing that Gen-QOT predicts
the cumulative quantity of inventory received for a specific order over time
reasonably well, although it is not perfectly calibrated (coefficient of 1).

Table 2: Calibration of cumulative inventory fill rate predicted $k$ weeks after submitting an order and actuals. | In Time Holdout | Out of Time Holdout
---|---|---
Weeks After Order | Estimate | 95% CI | Estimate | 95% CI
1 | 1.0577 | [1.056, 1.060] | 1.0559 | [1.055, 1.057]
2 | 1.1393 | [1.138, 1.141] | 1.1529 | [1.151, 1.154]
3 | 1.108 | [1.107, 1.109] | 1.1311 | [1.130, 1.132]
4 | 1.0866 | [1.086, 1.087] | 1.1147 | [1.114, 1.115]
5 | 1.0789 | [1.078, 1.079] | 1.1094 | [1.109, 1.110]
6 | 1.0745 | [1.074, 1.075] | 1.1021 | [1.102, 1.103]
7 | 1.0694 | [1.069, 1.070] | 1.0995 | [1.099, 1.100]
8 | 1.063 | [1.063, 1.063] | 1.0953 | [1.095, 1.096]
9 | 1.0538 | [1.053, 1.054] | 1.0895 | [1.089, 1.090]

Figure 5 shows this on samples of actual vs. predicted mean for normalized
cumulative fill rate at the end of the fourth week on the held out orders from
the training period and the test period. The results on the out-of-time hold
out set are very similar to those in the training period.

(a)

(b)

Figure 5: Residual calibration plot for cumulative fill rate – residuals are
plotted against predicted values in the main figure, while the original plot
is shown in an inner figure.

Rge classifier calibration plots are produced by discretizing and binning the
predicted probabilities of the event we are interested in, and then estimating
the mean of the actual classifications for each bin. Ideally, the average
actual will be equal to the mean of the predicted probabilities in each bin,
and this point will fall along the 45∘ line.

(a)

(b)

Figure 6: Residual calibration plots for Gen-QOT – residuals are plotted
against predicted values in the main figure, while the original plot is shown
in an inner figure.

Figure 6 shows the calibration of Gen-QOT at predicting whether a purchasing
action will yield zero inventory for both the in-time holdout (Figure 6(a))
and out-of-time holdout (Figure 6(b)). In both cases, points generally follow
the ideal calibration line, with some slight and expected degradation in the
out-of-time holdout.

###  4.2 Backtest of Inventory Control Policies

Having demonstrated in Section 4.1 that we have a dynamics model that achieves
good accuracy on our dataset, the next thing we want to do is use our
generative model to backtest various policies and measure their performance.

#### Data

We use a similar dataset to that used to fit the Gen-QOT model. The training
period is data from 2017-05-13 to 2019-03-23 and the backtest period is
2019-03-23 to 2020-03-21. The features used for the policy and the simulator
can be found in Appendix C.

#### Policies and Training

Because Madeka et al. (2022) performed an exhaustive evaluation of various
policies, we consider only a base stock baseline and two RL policies – one
trained on a gym that uses the inventory arrival dynamics from Madeka et al.
(2022) and one trained on gym with Gen-QOT arrival dynamics. These will be
referred to as Newsvendor, VLT-DirectBP, and QOT-DirectBP, respectively. The
policy networks consist of a Wavenet encoder (van den Oord et al., 2016) and
MLP decoder.

Following Madeka et al. (2022) we implement a differentiable simulator in
PyTorch. All algorithms are trained using a single p3dn.24xlarge EC2 instance.
We use the DirectBackprop algorithm to train the VLT-DirectBP and QOT-DirectBP
agents.

#### Results

Table 3 summarizes the changes in the sum of discounted reward of both
policies relative to a baseline policy. QOT-DirectBP outperforms VLT-DirectBP
when evaluated under the QOT transition dynamics. While it is unsurprising
that the policy trained on the simulator (with Gen-QOT) used in evaluation
performs best, this does underscore the impact of a “Sim2Real” gap – in this
case “reality” is the QOT simulator – as the overall performance gain is 8%.
Both policies still outperform the Newsvendor baseline, which is unsurprising
given the results in Madeka et al. (2022).

Table 3: Comparison of RL policies in a backtest using Gen-QOT. 95% confidence intervals are on the difference from baseline.  Policy | Discounted Reward | 95% CI
---|---|---
Newsvendor Baseline | 100.00% | –
VLT-DirectBP | 109.64% | [8.83% , 10.45%]
QOT-DirectBP |  117.81% | [16.92% , 18.69%]

In addition to cumulative discounted reward, we can also consider the
distribution of period-wise statistics. Figure 7. What is interesting is that
QOT-DirectBP selects larger order quantities, which likely reflects the fact
that Gen-QOT captures stochastic yields. We also note that QOT-DirectBP has
higher mean and median reward than other policies.

(a)

(b)

Figure 7: Distribution of per-period statistics for order quantity and rewards
in the backtest period (2019-03 to 2020-03).

###  4.3 Real World A/B Test

Finally, we ran a randomized control trial of the QOT-DirectBP agent in the US
store of a large e-retailer lasting several months and covering thousands of
products. The control arm used the existing production system (a base stock
policy). The trial is ongoing and we do not yet have final results on metrics
treatment effect on profit. Below we use data from the in-progress trial to
evaluate the performance of Gen-QOT off-policy.

#### Gen-QOT Performance “off-policy”

We use data from the Treatment and Control arms of the A/B test described
above to validate our assumption on the forecast accuracy of the QOT model. We
already know from Section 4.1 that the QOT model generalizes well out of
sample and forward in time. But the question remains: does it generalize well
out-of-sample and off-policy? This is critical because in order for our
inventory control backtest to be accurate, we required 3.1.

To validate our assumption that QOT does generalize off-policy, we check the
forecast errors on the treatment arm versus the control arm. In Table 4 we see
that the difference in forecast performance on-policy versus off-policy in the
actual supply chain is not statistically significant.

| Change (Control - Treatment)
---|---
Quantile | Mean | 95% CI
P10 | -0.17% | [-3.06%, 5.49%]
P30 | 0.45% | [-4.25%, 5.29%]
P50 | 0.47% | [-4.53%, 5.86%]
P70 | 0.30% | [-4.80%, 6.88%]
P90 | -0.56% | [-6.21%, 7.00%]
P98 | -6.26% | [-7.68%, 9.61%]
(a) Quantile losses of VLT forecasts produced by Gen-QOT

| Change (Control - Treatment)
---|---
$k$ | Mean | 95% CI
1 | 1.45% | [-8.41%, 11.33%]
2 | -0.88% | [-7.87%, 6.09%]
3 | -0.87% | [-6.25%, 4.51%]
4 | -1.96% | [-6.81%, 2.88%]
5 | -2.19% | [-6.87%, 2.48%]
6 | -1.09% | [-5.23%, 3.04%]
7 | -0.08% | [-3.94%, 3.78%]
8 | 0.54% | [-2.84%, 3.94%]
9 | 0.85% | [-1.76%, 3.47%]
(b) Calibration of cumulative inventory fill rate predicted $k$ weeks after
ordering

Table 4: Comparison of metrics on-policy (control arm) versus off-policy
(treatment arm) on data from real world A/B test.

##  5 Conclusion

We extended existing work on periodic review inventory control systems to the
case where inventory replenishments can arrive in multiple shipments over
time. We also allow for learning an inventory control policy in the case where
the retailer uses a post-processor to adjust order quantities suggested by the
control policy before submitting them to the supplier – a common practice used
to ensure order quantities meet minimum order and batch sizing requirements.
This is the first work to handle either of the two aforementioned
complexities. We then performed extensive empirical evaluation to show the
viability of the approach. We also validated that our learned dynamic model
generalizes well off-policy by backtesting it on data from a real-world A/B
test of our RL agent.

Interesting, and important, directions of future work include further
exploration of the minimal assumptions needed in order for the inventory
control problem to be efficiently learnable. It is also of interest to better
understand the precise conditions needed on the forward dynamics model (Gen-
QOT) in order for the theoretical results to hold. This may in turn give
insight into what evaluations practitioners should perform on the models they
incorporate into a simulator.

## References

  * Abbas et al. (2023) Abbas, Z., Zhao, R., Modayil, J., White, A. and Machado, M. C. (2023).  Loss of plasticity in continual deep reinforcement learning.  arXiv:2303.07507.
  * Alvo et al. (2023) Alvo, M., Russo, D. and Kanoria, Y. (2023).  Neural inventory control in networks via hindsight differentiable policy optimization.  arXiv:2306.11246.
  * Arrow et al. (1958) Arrow, K. J., Karlin, S., Scarf, H. E. et al. (1958).  Studies in the mathematical theory of inventory and production.  Stanford University Press.
  * Augenblick and Rabin (2019) Augenblick, N. and Rabin, M. (2019).  Belief movement, uncertainty reduction, and rational updating.  Tech. rep., Haas School of Business, University of California, Berkeley.
  * Balaji et al. (2019) Balaji, B., Bell-Masterson, J., Bilgin, E., Damianou, A., Garcia, P. M., Jain, A., Luo, R., Maggiar, A., Narayanaswamy, B. and Ye, C. (2019).  Orl: Reinforcement learning benchmarks for online stochastic optimization problems.  arXiv:1911.10641.
  * Bao (2006) Bao, Y. (2006).  Supply chain competition.  Tech. rep., UNSW Sydney. [PDF]
  * Bollapragada and Morton (1999) Bollapragada, S. and Morton, T. E. (1999).  Myopic heuristics for the random yield problem.  Operations Research 47 713–722.
  * Brown et al. (2020) Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford, A., Sutskever, I. and Amodei, D. (2020).  Language models are few-shot learners.  In Advances in Neural Information Processing Systems (H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan and H. Lin, eds.), vol. 33. Curran Associates, Inc.
  * Chan and Muckstadt (1999) Chan, E. and Muckstadt, J. (1999).  The effects of load smoothing on inventory levels in a capacitated production and inventory system.  Tech. rep., Cornell University Operations Research and Industrial Engineering. [PDF]
  * Clavera et al. (2020) Clavera, I., Fu, V. and Abbeel, P. (2020).  Model-augmented actor-critic: Backpropagating through paths.  In ICLR.
  * Dada et al. (2007) Dada, M., Petruzzi, N. C. and Schwarz, L. B. (2007).  A newsvendor’s procurement problem when suppliers are unreliable.  Manufacturing & Service Operations Management 9 9–32.
  * Das et al. (1999) Das, T. K., Gosavi, A., Mahadevan, S. and Marchalleck, N. (1999).  Solving semi-markov decision problems using average reward reinforcement learning.  Management Science 45 560–574.
  * Dawid (1982) Dawid, A. (1982).  The well calibrated bayesian.  Journal of the American Statistical Association 77 605–613.
  * Devlin et al. (2019) Devlin, J., Chang, M.-W., Lee, K. and Toutanova, K. (2019).  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding.  In NAACL-HLT.
  * Efroni et al. (2022a) Efroni, Y., Foster, D. J., Misra, D., Krishnamurthy, A. and Langford, J. (2022a).  Sample-efficient reinforcement learning in the presence of exogenous information.  arXiv:2206.04282.
  * Efroni et al. (2022b) Efroni, Y., Kakade, S., Krishnamurthy, A. and Zhang, C. (2022b).  Sparsity in partially controllable linear systems.  In International Conference on Machine Learning. PMLR.
  * Eisenach et al. (2020) Eisenach, C., Patel, Y. and Madeka, D. (2020).  MQTransformer: Multi-Horizon Forecasts with Context Dependent and Feedback-Aware Attention.  arXiv:2009.14799.
  * Federgruen and Zipkin (1986) Federgruen, A. and Zipkin, P. (1986).  An inventory model with limited production capacity and uncertain demands i. the average-cost criterion.  Mathematics of Operations Research 11 193–207.
  * Fisher and Raman (1996) Fisher, M. and Raman, A. (1996).  Reducing the cost of demand uncertainty through accurate response to early sales.  Operations research 44 87–99.
  * Foster and Stine (2021) Foster, D. and Stine, R. (2021).  Threshold Martingales and the Evolution of Forecasts.  arXiv:2105.06834.
  * Foster and Vohra (1997) Foster, D. P. and Vohra, R. V. (1997).  Calibrated learning and correlated equilibrium.  Games and Economic Behavior 21 40–55.
  * Gasparin et al. (2019) Gasparin, A., Lukovic, S. and Alippi, C. (2019).  Deep Learning for Time Series Forecasting: The Electric Load Case.  arXiv:1907.09207.
  * Gerchak et al. (1988) Gerchak, Y., Vickson, R. G. and Parlar, M. (1988).  Periodic review production models with variable yield and uncertain demand.  Iie Transactions 20 144–150.
  * Giannoccaro and Pontrandolfo (2002) Giannoccaro, I. and Pontrandolfo, P. (2002).  Inventory management in supply chains: a reinforcement learning approach.  International Journal of Production Economics 78 153–161.
  * Gijsbrechts et al. (2022) Gijsbrechts, J., Boute, R. N., Van Mieghem, J. A. and Zhang, D. J. (2022).  Can deep reinforcement learning improve inventory management? performance on lost sales, dual-sourcing, and multi-echelon problems.  Manufacturing & Service Operations Management 24 1349–1368.
  * Goodfellow et al. (2014) Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A. and Bengio, Y. (2014).  Generative adversarial nets.  In Advances in Neural Information Processing Systems (Z. Ghahramani, M. Welling, C. Cortes, N. Lawrence and K. Weinberger, eds.), vol. 27. Curran Associates, Inc.
  * Graves (2012) Graves, A. (2012).  Sequence transduction with recurrent neural networks.  arXiv:1211.3711.
  * Graves (2013) Graves, A. (2013).  Generating sequences with recurrent neural networks.  arXiv:1308.0850.
  * Henig and Gerchak (1990) Henig, M. and Gerchak, Y. (1990).  The structure of periodic review policies in the presence of random yield.  Operations Research 38 634–643.
  * Hu et al. (2019) Hu, Y., Liu, J., Spielberg, A., Tenenbaum, J. B., Freeman, W. T., Wu, J., Rus, D. and Matusik, W. (2019).  Chainqueen: A real-time differentiable physical simulator for soft robotics.  In 2019 International conference on robotics and automation (ICRA). IEEE.
  * Ingraham et al. (2018) Ingraham, J., Riesselman, A., Sander, C. and Marks, D. (2018).  Learning protein structure with a differentiable simulator.  In International Conference on Learning Representations.
  * Januschowski et al. (2022) Januschowski, T., Wang, Y., Torkkola, K., Erkkilä, T., Hasson, H. and Gasthaus, J. (2022).  Forecasting with trees.  International Journal of Forecasting 38 1473–1481.  Special Issue: M5 competition.
  * Kaplan (1970) Kaplan, R. S. (1970).  A dynamic inventory model with stochastic lead times.  Management Science 16 491–507.
  * Kiesmüller et al. (2011) Kiesmüller, G. P., Kok, D. and Dabia, S. (2011).  Single item inventory control under periodic review and a minimum order quantity.  International Journal of Production Economics 133 280–285.
  * Kingma and Welling (2013) Kingma, D. P. and Welling, M. (2013).  Auto-encoding variational bayes.  arXiv:1312.6114.
  * Kingma and Welling (2019) Kingma, D. P. and Welling, M. (2019).  An introduction to variational autoencoders.  arXiv:1906.02691.
  * Li et al. (2004) Li, Z., Xu, S. H. and Hayya, J. (2004).  A periodic-review inventory system with supply interruptions.  Probability in the Engineering and Informational Sciences 18 33–53.
  * Lim et al. (2019) Lim, B., Arik, S. O., Loeff, N. and Pfister, T. (2019).  Temporal Fusion Transformers for Interpretable Multi-horizon Time Series Forecasting.  arXiv:1912.09363.
  * Maddah and Jaber (2008) Maddah, B. and Jaber, M. Y. (2008).  Economic order quantity for items with imperfect quality: Revisited.  International Journal of Production Economics 112 808–815.  Special Section on RFID: Technology, Applications, and Impact on Business Operations.
  * Madeka et al. (2018) Madeka, D., Swiniarski, L., Foster, D., Razoumov, L., Torkkola, K. and Wen, R. (2018).  Sample path generation for probabilistic demand forecasting.  In KDD 2018 Workshop on Mining and Learning from Time Series.
  * Madeka et al. (2022) Madeka, D., Torkkola, K., Eisenach, C., Foster, D. and Luo, A. (2022).  Deep inventory management.  arXiv:2210.03137.
  * Maggiar et al. (2022) Maggiar, A., Song, I. and Muharremoglu, A. (2022).  Multi-echelon inventory management for a non-stationary capacitated distribution network.  Tech. rep. [PDF]
  * Mnih et al. (2016) Mnih, V., Badia, A. P., Mirza, M., Graves, A., Lillicrap, T. P., Harley, T., Silver, D. and Kavukcuoglu, K. (2016).  Asynchronous methods for deep reinforcement learning.  arXiv:1602.01783.
  * Mnih et al. (2013) Mnih, V., Kavukcuoglu, K., Silver, D., Graves, A., Antonoglou, I., Wierstra, D. and Riedmiller, M. (2013).  Playing atari with deep reinforcement learning.  arXiv:1312.5602.
  * Mousa et al. (2023) Mousa, M., van de Berg, D., Kotecha, N., del Rio-Chanona, E. A. and Mowbray, M. (2023).  An analysis of multi-agent reinforcement learning for decentralized inventory control systems.  arXiv:2307.11432.
  * Mukhoty et al. (2019) Mukhoty, B. P., Maurya, V. and Shukla, S. K. (2019).  Sequence to sequence deep learning models for solar irradiation forecasting.  In IEEE Milan PowerTech.
  * Nahmias (1979) Nahmias, S. (1979).  Simple approximations for a variety of dynamic leadtime lost-sales inventory models.  Operations Research 27 904–924.
  * Nascimento et al. (2019) Nascimento, R. C., Souto, Y. M., Ogasawara, E., Porto, F. and Bezerra, E. (2019).  STConvS2S: Spatiotemporal Convolutional Sequence to Sequence Network for weather forecasting.  arXiv:1912.00134.
  * Parmas et al. (2023) Parmas, P., Seno, T. and Aoki, Y. (2023).  Model-based reinforcement learning with scalable composite policy gradient estimators.  In ICML.
  * Porteus (2002) Porteus, E. L. (2002).  Foundations of stochastic inventory theory.  Stanford University Press.
  * Qi et al. (2023) Qi, M., Shi, Y., Qi, Y., Ma, C., Yuan, R., Wu, D. and Shen, Z.-J. (2023).  A practical end-to-end inventory management model with deep learning.  Management Science 69 759–773.
  * Robb and Silver (1998) Robb, D. J. and Silver, E. A. (1998).  Inventory management with periodic ordering and minimum order quantities.  Journal of the Operational Research Society 49 1085–1094.
  * Salinas et al. (2020) Salinas, D., Flunkert, V., Gasthaus, J. and Januschowski, T. (2020).  Deepar: Probabilistic forecasting with autoregressive recurrent networks.  International Journal of Forecasting 36 1181–1191.
  * Schulman et al. (2017) Schulman, J., Wolski, F., Dhariwal, P., Radford, A. and Klimov, O. (2017).  Proximal policy optimization algorithms.  arXiv:1707.06347.
  * Shen et al. (2019) Shen, H., Tian, T. and Zhu, H. (2019).  A two-echelon inventory system with a minimum order quantity requirement.  Sustainability 11.
  * Silver et al. (2016) Silver, D., Huang, A., Maddison, C. J., Guez, A., Sifre, L., Van Den Driessche, G., Schrittwieser, J., Antonoglou, I., Panneershelvam, V., Lanctot, M. et al. (2016).  Mastering the game of go with deep neural networks and tree search.  Nature 529 484–489.
  * Sinclair et al. (2023) Sinclair, S. R., Vieira Frujeri, F., Cheng, C.-A., Marshall, L., Barbalho, H. D. O., Li, J., Neville, J., Menache, I. and Swaminathan, A. (2023).  Hindsight learning for MDPs with exogenous inputs.  In Proceedings of the 40th International Conference on Machine Learning, vol. 202 of Proceedings of Machine Learning Research. PMLR.
  * Song and Zipkin (1996) Song, J.-S. and Zipkin, P. H. (1996).  Inventory control with information about supply conditions.  Management science 42 1409–1419.
  * Suh et al. (2022) Suh, H. J., Simchowitz, M., Zhang, K. and Tedrake, R. (2022).  Do differentiable simulators give better policy gradients?  In International Conference on Machine Learning. PMLR.
  * Sundermeyer et al. (2010) Sundermeyer, M., Schluter, R. and Ney, H. (2010).  LSTM Neural Networks for Language Modeling.  In INTERSPEECH.
  * Sutton and Barto (2020) Sutton, R. and Barto, A. (2020).  Reinforcement Learning: An Introduction.  The MIT Press.
  * Szepesvári (2010) Szepesvári, C. (2010).  Algorithms for Reinforcement Learning.  Synthesis Lectures on Artificial Intelligence and Machine Learning, Morgan & Claypool Publishers.
  * Taleb (2018) Taleb, N. N. (2018).  Election predictions as martingales: an arbitrage approach.  Quantitative Finance 18 1–5.
  * Taleb and Madeka (2019) Taleb, N. N. and Madeka, D. (2019).  All roads lead to quantitative finance.  Quantitative Finance 19 1775–1776.
  * THOMAS (2023) THOMAS, J. D. (2023).  Towards cooperative marl in industrial domains .
  * van den Oord et al. (2016) van den Oord, A., Dieleman, S., Zen, H., Simonyan, K., Vinyals, O., Graves, A., Kalchbrenner, N., Senior, A. and Kavukcuoglu, K. (2016).  Wavenet: A generative model for raw audio.  arXiv:1609.03499.
  * Van Oord et al. (2016) Van Oord, A., Kalchbrenner, N. and Kavukcuoglu, K. (2016).  Pixel recurrent neural networks.  In International conference on machine learning. PMLR.
  * Veinott (1965) Veinott, A. F. (1965).  The optimal inventory policy for batch ordering.  Operations Research 13 424–432.
  * Wen and Torkkola (2019) Wen, R. and Torkkola, K. (2019).  Deep Generative Quantile-Copula Models for Probabilistic Forecasting.  In ICML Time Series Workshop.
  * Wen et al. (2017) Wen, R., Torkkola, K., Narayanaswamy, B. and Madeka, D. (2017).  A multi-horizon quantile recurrent forecaster.  In NIPS Time Series Workshop.
  * Williams and Zipser (1989) Williams, R. J. and Zipser, D. (1989).  A learning algorithm for continually running fully recurrent neural networks.  Neural Computation 1 270–280.
  * Yu et al. (2017) Yu, R., Zheng, S., Anandkumar, A. and Yue, Y. (2017).  Long-term Forecasting using Higher Order Tensor RNNs.  arXiv:1711.00073.
  * Zhao et al. (2023) Zhao, H., Tang, W. and Yao, D. D. (2023).  Policy optimization for continuous reinforcement learning.  arXiv:2305.18901.
  * Zhao and Katehakis (2006) Zhao, Y. and Katehakis, M. N. (2006).  On the structure of optimal ordering policies for stochastic inventory systems with minimum order quantity.  Probability in the Engineering and Informational Sciences 20 257–270.
  * Zhou et al. (2007) Zhou, B., Zhao, Y. and Katehakis, M. N. (2007).  Effective control policies for stochastic inventory systems with a minimum order quantity and linear costs.  International Journal of Production Economics 106 523–531.
  * Zhu (2022) Zhu, H. (2022).  A simple heuristic policy for stochastic inventory systems with both minimum and maximum order quantity requirements.  Annals of Operations Research 309 347–363.
  * Zhu et al. (2015) Zhu, H., Liu, X. and Chen, Y. F. (2015).  Effective inventory control policies with a minimum order quantity and batch ordering.  International Journal of Production Economics 168 21–30.
  * Zipkin (2008) Zipkin, P. (2008).  Old and new methods for lost-sales inventory systems.  Operations research 56 1256–1263.

##  Appendix A Gen-QOT Metrics

In this section we define the metrics used to evaluate Gen-QOT.

###  A.1 Vendor Lead Time Forecasting Metrics

For the purposes of defining the vendor lead time metric, assume we have a set
of historic purchase orders. Each sample $i\in{\mathcal{S}}$ consist of a
sequence of arrivals
$\\{(\kappa^{i}_{1},l^{i}_{1}),\dots,(\kappa^{i}_{J},l^{i}_{J})\\}$ where $j$
denotes the sequence index with some maximum sequence length $J$,
$\kappa^{i}_{j}$ denotes the arrival quantity, and $l^{i}_{j}$ denotes the
lead-time of the arrival relative to some forecast creation time, which can be
the order date or any day after.

The quantile loss of a forecast at quantile $q$ is defined as
$\mathrm{QL}_{q}(x,y)=(1-q)(x-y)^{+}+q(y-x)^{+}$. For a fixed quantile $q$,
the quantile loss metric evaluated in this paper is defined as

$\mathrm{QL}_{q}:=\frac{\sum_{i\in\mathcal{S}}\sum_{j=1}^{J}\kappa^{i}_{j}\mathrm{QL}_{q}(l^{i}_{j},\widehat{l}^{i,q})}{\sum_{i\in\mathcal{S}}\sum_{j=1}^{J}\kappa^{i}_{j}}$
(A.1)

Next, to compute the CRPS, we assume access to a quantile estimate of the
lead-time distribution for each sample, $\widehat{l}^{i,q}$ over a set of
quantiles $\mathcal{Q}:=\\{0.01,...,0.99\\}$. The CRPS is computed
approximately by averaging over the quantile losses for each quantile in
$q\in\mathcal{Q}$.

$\mathrm{CRPS}:=\frac{\sum_{i\in\mathcal{S}}\sum_{j=1}^{J}\kappa^{i}_{j}\frac{1}{|\mathcal{Q}|}\sum_{q\in\mathcal{Q}}\mathrm{QL}_{q}(l^{i}_{j},\widehat{l}^{i,q})}{\sum_{i\in\mathcal{S}}\sum_{j=1}^{J}\kappa^{i}_{j}}$
(A.2)

###  A.2 Calibration Metrics

A forecast of the probabilities of a sequence of events $E_{1},E_{2},...$
where $E_{t}\in\\{0,1\\}$ – is said to be calibrated if whenever a forecast
$p$ of $E_{t}=1$ is made, the empirical probabilities are $\approx p$. Because
the probability is real valued, the interval $[0,1]$ is split into bins in
order to get empirical probabilities whenever the forecast was $p$.

In the case of estimating the mean of a random variable (such as percent fill
after $l$ weeks), we define calibration as the regression coefficient of a
simple linear regression of the actual value given the predicted value. If a
forecast is well calibrated, this coefficient should be 1.

See Foster and Vohra (1997); Dawid (1982) for a more in-depth discussion of
calibrated forecasts.

##  Appendix B Additional Numerical Results for Gen-QOT

###  B.1 Neural Architecture Ablation

Given the broad set of neural architectures available for fitting sequence-to-
sequence problems, we test a set of different encoder and decoder neural
networks classes. Specifically we tested multi-layer perceptron vs causal-
convolution encoder, and recurrent neural network vs transformer decoder. In
the end, we trained four models on data from sequences of arrivals from 10MM
orders from 2017 and 2018 and tested on arrivals from 50K orders from 2019. To
assess prediction quality, we rely on negative-log-likelihood of next token
prediction, as well as unit weighted quantile-loss of cumulative quantity
arrivals at 1, 4, and 9 weeks since order was placed.

Table 5: Results of ablation analysis for multiple metrics. Model | Metric
---|---
| QL of Cumulative Quantity of Arrivals: Week1
| P10 | P30 | P50 | P70 | P90
MLP-RNN | 100.00 | 100.00 | 100.00 | 100.00 | 100.00
CNN-RNN | 70.15 | 98.60 | 89.15 | 96.67 | 100.42
CNN-Transformer | 68.54 | 95.79 | 91.76 | 98.81 | 110.10
| QL of Cumulative Quantity of Arrivals: Week4
| P10 | P30 | P50 | P70 | P90
MLP-RNN | 100.00 | 100.00 | 100.00 | 100.00 | 100.00
CNN-RNN | 89.43 | 95.90 | 97.48 | 100.00 | 104.94
CNN-Transformer | 29.27 | 39.85 | 52.04 | 66.75 | 80.86
| QL of Cumulative Quantity of Arrivals: Week9
| P10 | P30 | P50 | P70 | P90
MLP-RNN | 100.00 | 100.00 | 100.00 | 100.00 | 100.00
CNN-RNN | 100.79 | 100.58 | 100.82 | 101.70 | 102.01
CNN-Transformer | 90.24 | 92.24 | 96.29 | 99.72 | 104.03
| Negative Log-Likelihood of Next Token Prediction
MLP-RNN | 100.00
CNN-RNN | 93.76
CNN-Transformer | 94.07

##  Appendix C Featurization

#### Buying Agent

Below are the features provided to the RL policy – they are the same as Madeka
et al. (2022). Specifically, the state at time $t$ for product $i$ contains:

  1. 1.

The current inventory level $I^{i}_{(t-1)}$

  2. 2.

Previous actions $a^{i}_{u}$ that have been taken for all $u<t$

  3. 3.

Demand time series features

    1. (a)

Historical availability corrected demand

    2. (b)

Distance to public holidays

    3. (c)

Historical website glance views data

  4. 4.

Static product features

    1. (a)

Product group

    2. (b)

Text-based features from the product description

    3. (c)

Brand

  5. 5.

Economics of the product - (price, cost etc.)

#### Gen-QOT Model

The exogenous context and other information provided to Gen-QOT at time $t$
for product $i$ contains:

  1. 1.

Current action $a_{t}^{i}$

  2. 2.

Previous actions $a^{i}_{u}$ that have been taken for all $u<t$

  3. 3.

Time series features

    1. (a)

Distance to public holidays

    2. (b)

Previous arrivals for all times $u<t$

    3. (c)

Vendor constraints (minimum order quantities, batch sizes, etc.)

  4. 4.

Static product features

    1. (a)

Product group

    2. (b)

Brand

    3. (c)

Vendor

##  Appendix D The Gen-QOT Model

In this section, we describe our novel arrivals prediction model. First,
denote historical covariates $x^{i}_{t}$ for each product $i$ at each time $t$
that will be used to estimate the distribution of state transition
probabilities. The vector $x^{i}_{t}$ can be thought of as the observational
historical data that contains at minimum, information such as the time-series
of historical orders and arrivals. Other information that can be incorporated
includes vendor and product attributes, existing geographic inventory
allocation, and the distance to various holidays that may affect the ability
for vendors and logistics providers to reliably fulfill and ship inventory.
See Appendix C for the list of features we used. Also denote the actual
arrivals of inventory as $\\{o^{i}_{t,j}\\}_{j=0}^{L}$, where
$o^{i}_{t,j}\in\mathbb{R}_{\geq 0}$. Recall $L$ is the maximum possible lead
time for an arrival.

The Gen-QOT model solves the problem of predicting the joint distribution of
inventory arrivals for an action $a^{i}_{t}$ at time $t$. To model the
distribution of arrival sequences $\\{o^{i}_{t,j}\\}_{j=1}^{L}$, we consider
the distribution over partial fill rates666These are the same as the
$\bar{\alpha}_{t,j}^{i}$ in Section 3.5. instead:

$\alpha^{i}_{t,j}:=\begin{cases}\frac{o^{i}_{t,j}}{a^{i}_{t}}&a^{i}_{t}\neq
0\\\ 0&a^{i}_{t}=0.\\\ \end{cases}$

Our goal then is to produce a generative model from which we can sample
sequences of partial fill rates $\\{\alpha^{i}_{t,j}\\}_{j=1}^{L}$. Note that
these partial fill rates do not need to sum to 1 as this is modeling the “end-
to-end” yield (see Figure 2(b)).

Next, observe that an equivalent formulation is to model a sequence of tuples
of partial fill rates and time since the last non-zero arrival:
$\\{(k^{i}_{t,s},\widetilde{\alpha}^{i}_{t,s})\\}_{s\in\mathbb{Z}_{\geq 0}}$,
where $\widetilde{\alpha}^{i}_{t,s}$ denotes the proportion of $a^{i}_{t}$ in
the $s^{th}$ arrival and $k^{i}_{t,s}$ denotes the number of periods since the
previous non-zero arrival. By convention the first arrival is measured as an
offset from $t-1$.

###  D.1 Probabilistic Model

At a high-level, the methodology we employ to predict a sequence of arrivals
is to construct a grid over quantity and time that can be used to bin
individual arrivals into distinct arrival classes. Our model then produce a
categorical distribution over these classes conditioned on previous arrivals
in the sequence, akin to generative sequence modeling in NLP. Once the
sequence of classes has been sampled, we can map it back to the original
quantity of interest using the function $V$, defined in Section D.1.2.

####  D.1.1 Model the distribution by binning

We rely on binning to avoid making parametric assumptions about the
distributions of arrivals. To model the proportion of requested inventory in
each arrival and number weeks since last arrival, we can bin these pairs into
classes. We define a sequence of $N$ grid points $\tau^{(1)},\dots,\tau^{(N)}$
over time periods since last arrival and $M$ grid points
$p^{(1)},\dots,p^{(M)}$ for the proportions, such that

$\displaystyle\tau^{(n)}\leq\tau^{(m)}~{}~{}\forall n<m,$ $\displaystyle
p^{(n)}\leq p^{(m)}~{}~{}\forall n<m,$ $\displaystyle\tau^{(1)}=p^{(1)}=0,$
$\displaystyle\tau^{(N)}=\tau_{max}$ $\displaystyle p^{(M)}=p_{max}$

where $\tau_{max}$ and $p_{max}$ are both large constants. For each
$n,m\in[N-1]\times[M-1]$, define the set

$Z_{n,m}:=\\{(k,\alpha):\tau^{(n)}\leq k<\tau^{(n+1)}\text{ and
}p^{(m)}\leq\alpha<p^{(m+1)}\\}.$

By construction, $\mathcal{Z}:=\\{Z_{n,m}:(n,m)\in[N-1]\times[M-1]\,\\}$ is a
partition of $\\{u\in\mathbb{Z}\,|\,0\leq
u<\tau_{max}\;\\}\times\\{v\in\mathbb{R}\,|\,0\leq v<p_{max}\;\\}$, the space
of all possible pairs $(k^{i}_{s},\widetilde{\alpha}^{i}_{t,s})$.

Next we denote the index pair $(n,m)$ that corresponds to a specific arrival
class $Z_{n,m}$ by the random vector $\zeta$. For example $\zeta=(1,2)$ is a
reference to the arrival class $Z_{1,2}$. Additionally, we add a special index
pair denoting the end of an arrivals sequence, which we can signify with
$\cancel{\zeta}$. This $\cancel{\zeta}$ can be thought of as a special arrival
that designates that the sequence of arrivals from an action has terminated.
In practice, we can use $(0,0)$ to represent the value of $\cancel{\zeta}$.
Given this construction, we have $\zeta$ taking values in
$\widetilde{\mathcal{Z}}:=\\{[N-1]\times[M-1])\cup\\{(0,0)\\}\\}$

Our objective is now to estimate the joint probability of a sequence of index
pairs $\zeta^{i}_{t,0},\zeta^{i}_{t,1}\dots$ corresponding to a sequence of
arrival classes given some $a^{i}_{t}$. Gen-QOT specifically estimates

$P(\zeta^{i}_{t,0},\zeta^{i}_{t,1},...|x^{i}_{t},a^{i}_{t,0})=\prod_{j}P(\zeta^{i}_{t,j}|\zeta^{i}_{t,j-1},\zeta^{i}_{t,j-2},\dots,\zeta^{i}_{t,0},x^{i}_{t},a^{i}_{t})$
(D.1)

and decomposes this joint probability into the product of conditional
probabilities.

####  D.1.2 Generating samples of inventory arrivals: mapping from classes to
actual arrivals

Sampling this joint probability distribution gives a sequence of arrival
classes $(\zeta^{i}_{t,0},\zeta^{i}_{t,1},\dots)$. To obtain a sample path of
inventory arrivals for an action $a^{i}_{t}$, we use the function
$V:\widetilde{\mathcal{Z}}\rightarrow\mathbb{R}_{>0}\times\mathbb{Z}_{>0}$ to
map each element $\zeta^{i}_{t,s}=(n,m)$ of the sampled sequences of arrival
classes back to a tuple of partial fill rates and time-since-last-arrival.
This function is implemented by mapping each of the tuples $(n,m)$ to a
representative element $(\bar{k}_{n,m}$, $\bar{\alpha}_{n,m})$ in the
corresponding $Z_{i,j}$. Here, $\bar{k}_{n,m}$ and $\bar{\alpha}_{n,m}$ denote
the mean of the respective quantities observed in the data in each arrival
class bin, for every $n,m$. However, we note that other quantities can be
used, e.g. the center of the bin of each arrival class
$(\frac{\tau^{(n)}+\tau^{(n+1)}}{2},\frac{p^{(i)}+p^{(i+1)}}{2})$.

The sampled sequence of arrival class indexes,
$(\zeta^{i}_{t,0},\zeta^{i}_{t,1},\dots)$, can be transformed back into the
original quantity of interest $o^{i}_{t,j}$ by substituting each
$\zeta^{i}_{t,s}$ with the tuple $(\bar{k}_{n,m},\bar{\alpha}_{n,m})$ to
recover a sequence of estimated periods since last arrival and proportion of
inventory, $(k^{i}_{t,s},\widetilde{\alpha}^{i}_{t,s})$. By cumulatively
summing the periods, and multiplying each $\widetilde{\alpha}^{i}_{t,s}$ by
$a^{i}_{t}$, we immediately recover every $o^{i}_{t,j}$.

####  D.1.3 Example of arrival sequence transformation

Inventory arrivals to arrival class sequence: As an example, let
$a^{i}_{t}=10$, and the true arrival sequence be $\langle 0,3,5,0,4\rangle$.
Additionally, we can construct a series of grid-points $\tau^{(l)}=l-1$ and
$p^{(m)}=0.2\cdot(m-1)$ for all $l\in[L]$ and $m\in[M]$ where $L=4$ and $M=6$.

Then we can map from the original sequence of arrivals over time
$\\{o^{i}_{t,j}\\}$ to tuples of partial fill rates and time since last
arrival $\\{(k^{i}_{s},\widetilde{\alpha}^{i}_{t,s})\\}$ by normalizing by
action and computing the time since last arrival.

$\langle
0,3,5,0,4\rangle~{}\longrightarrow~{}\langle(2,0.3),(1,0.5),(2,0.4)\rangle$

Using the grid-points and $(0,0)$ as a reference for $\cancel{\zeta}$, we can
transform the sequence of $\\{(k^{i}_{s},\widetilde{\alpha}^{i}_{t,s})\\}$
into a sequence of $\zeta$’s by binning the tuples of time since last arrival
and inventory percentile. For example the tuple $(2,0.3)$ is placed in the bin
with coordinates $(2,2)$ because it is $2$ time periods since the start of the
sample and $0.3$ falls in the second bin that sits between $0.2$ and $0.4$.
The rest of the sequence is transformed as follows:

$\langle(2,0.3),(1,0.5),(2,0.4)\rangle~{}\longrightarrow~{}\langle(2,2),(1,3),(2,3),(0,0)\rangle$

Arrival class sequence to inventory arrivals: We can reverse this example by
imagining Gen-QOT generated the sequence
$\langle(2,2),(1,3),(2,3),(0,0)\rangle$. We can replace each $Z_{n,m}$ with
the representative element corresponding to the middle of the bin
$(\bar{k}_{n,m},\bar{\alpha}_{n,m})=(l,0.2\cdot m-0.1)$. Substituting into the
sequence we get

$\langle(2,2),(1,3),(2,3),(0,0)\rangle~{}\longrightarrow~{}\langle(2,0.3),(1,0.5),(2,0.5)\rangle$

Then we can cumulatively sum the time since last arrival and multiply by the
action to recover sampled sequence of arrivals over time, $\\{o^{i}_{t,j}\\}$

$\langle(2,0.3),(1,0.5),(2,0.5)\rangle~{}\longrightarrow~{}\langle
0,3,5,0,5\rangle$

In this example we see two crucial features of the probabilistic model
utilized by Gen-QOT. Firstly, the sum of arrival quantities over actual and
sampled arrival quantities do not need to be equal to the action $a^{i}_{t}$.
Secondly, the structure of the grid and choice of representative unit can
induce error in the estimated sample if not carefully chosen.

###  D.2 Neural Architecture and Loss

Following canonical work in generative modeling for language (Sundermeyer et
al., 2010; Graves, 2013), our work uses recurrent neural-networks to generate
sequences of arrival classes. Additionally, following van den Oord et al.
(2016); Wen et al. (2017) we rely on stacked and dilated temporal convolutions
to learn a useful embedding of historical time series data. Merging the
architectures together, Gen-QOT is implemented using a encoder-decoder style
architecture with a dilated temporal convolution encoder and recurrent
decoder. Full model hyperparameters along with arrival class definitions can
be found in Section D.3. Additionally, a richer comparison of various model
architectures can be found in Section B.1.

Figure 8: Visualization of Gen-QOT model architecture. The model structure
uses a classic encoder decoder architecture with a dilated causal convolution
encoder and standard recurrent decoder. The diagram above demonstrates how
samples are generated the Gen-QOT model during inference, where $<SOO>$ is a
vector of zeros

The network is optimized to maximize the likelihood of generated samples by
being trained to minimize cross-entropy loss. Allowing $y$ to be a matrix of
indicators across all arrival classes $Z_{n,m}$ and $\widehat{y}$ to be the
matrix of probabilities for each arrival class, then the loss for a single
prediction can be written as

$J(y,\widehat{y}\,)=-\sum^{N-1}_{n=1}\sum^{M-1}_{m=1}y_{n,m}\log(\,\widehat{y}_{n,m}\,)$

where the sum runs over all arrival classes. Finally, the network is trained
using the teacher-forcing algorithm (Williams and Zipser, 1989), where during
training the model learns to predict the next token in a sequence given the
actual trailing sub-sequence. During inference, strategies like beam-search
(Graves, 2012) can be used to find the highest likelihood sequence of arrival
classes. For our work, we implemented a simplified inference algorithm that
samples the predicted distribution of arrival classes to generate a trailing
sub-sequence that is used to predict subsequent token.

###  D.3 Gen-QOT Training Hyperparameters

Table 6: Training hyperparameters for Gen-QOT Hyper-parameter | Value
---|---
Epochs | $500$
Learning Rate | $1\times 10^{-4}$
Optimizer | Adam
Number of Convolution Layers | $5$
Number of Convolution Channels | $32$
Number of Recurrent Layers | $2$
Convolution Dilations | $[1,2,4,8,16]$
Recurrent Decoder Size | $512$
Multi-layer Perception Size | $512$
Activation | ReLU