Question:
You are a highly experienced, conscientious, and fair academic reviewer, please help me review this paper. The review should be organized into nine sections:
1. Summary: A summary of the paper in 100-150 words.
2. Strengths/Weaknesses/Questions: The Strengths/Weaknesses/Questions of paper, which should be listed in bullet points, with each point supported by specific examples from the article where possible.
3. Soundness/Contribution/Presentation: Rate the paper's Soundness/Contribution/Presentation, and match this score to the corresponding description from the list below and provide the result. The possible scores and their descriptions are:
    1 poor
    2 fair
    3 good
    4 excellent
4. Rating: Give this paper an appropriate rating, match this rating to the corresponding description from the list below and provide the result. The possible Ratings and their descriptions are:
    1 strong reject
    2 reject, significant issues present
    3 reject, not good enough
    4 possibly reject, but has redeeming facets
    5 marginally below the acceptance threshold
    6 marginally above the acceptance threshold
    7 accept, but needs minor improvements
    8 accept, good paper
    9 strong accept, excellent work
    10 strong accept, should be highlighted at the conference
5. Paper Decision: It must include the Decision itself(Accept or Reject) and the reasons for this decision, based on the criteria of originality, methodological soundness, significance of results, and clarity and logic of presentation.

Here is the template for a review format, you must follow this format to output your review result:
**Summary:**
Summary content

**Strengths:**
- Strength 1
- Strength 2
- ...

**Weaknesses:**
- Weakness 1
- Weakness 2
- ...

**Questions:**
- Question 1
- Question 2
- ...

**Soundness:**
Soundness result

**Presentation:**
Presentation result

**Contribution:**
Contribution result

**Rating:**
Rating result

**Paper Decision:**
- Decision: Accept/Reject
- Reasons: reasons content

Please ensure your feedback is objective and constructive. The paper is as follows:

# Time-uniform confidence bands for the CDF under nonstationarity

Paul Mineiro

Microsoft Research

pmineiro@microsoft.com &Steve Howard

The Voleon Group

steve@stevehoward.org

###### Abstract

Estimation of a complete univariate distribution from a sequence of observations is a useful primitive for both manual and automated decision making. This problem has received extensive attention in the i.i.d. setting, but the arbitrary data dependent setting remains largely unaddressed. We present computationally felicitious time-uniform and value-uniform bounds on the CDF of the running averaged conditional distribution of a sequence of real-valued random variables. Consistent with known impossibility results, our CDF bounds are always valid but sometimes trivial when the instance is too hard, and we give an instance-dependent convergence guarantee. The importance-weighted extension is appropriate for estimating complete counterfactual distributions of rewards given data from a randomized experiment, e.g., from an A/B test or a contextual bandit.

## 1 Introduction

What would have happened if I had acted differently? Although as old as time itself, successful companies have recently embraced this question via offline estimation of counterfactual outcomes using data from existing randomized experiments or contextual bandits. The problem is important in diverse domains such as software testing (Lindon et al., 2022; Wang and Chapman, 2022), portfolio management (Liu, 2021), and medicine (Shen et al., 2022). These experiments are run in the real (digital) world, which is rich enough to demand non-asymptotic statistical techniques under non-parametric and non-stationary (i.e., not i.i.d.) models. Although existing methods apply for estimating _average_ outcomes in this general setting (either under the observed distribution or counterfactual ones), estimating a complete _distribution_ of outcomes is heretofore only possible with additional assumptions: see Table 1 for a summary and Section 5 for complete discussion of related work.

To fix ideas, we briefly describe an application from Lindon et al. (2022) in the context of canary testing: rolling out changes in an online service to a small, random subset of users in order to detect accidental performance regressions while minimizing effect on overall user experience. The metric of interest measures latency for fetching content from the service. It is common to look beyond the mean of the latency distribution and especially to check for regressions in upper quantiles. As such, the authors choose to estimate bounds on the entire CDF of this latency metric under both the control and treatment arms and check for a statistically significant differences at any point in the CDF. The hope is to detect regressions as soon as possible, often within seconds or minutes, so the authors employ a sequential method which allows an automated system to continuously update the CDF bounds as data accumulates and to stop as soon as a significant regression is detected. Statistically, this translates into the requirement of confidence bands for the CDF which are both uniform over time (valid after every update) and uniform over values (so we can check for regressions at any quantile). We seek such bounds whose statistical validity is guaranteed under a minimum of assumptions.

Intriguingly, this problem is provably impossible in the general data dependent setting (Rakhlin et al., 2015). Consequently, our bounds always achieve non-asymptotic coverage, but may converge to zero width slowly or not at all, depending on the hardness of the instance. We call this design principle AVAST (\(\Delta\)lways \(\Delta\)laid \(\Delta\)nd Sometimes \(\mathsf{Trivial}\)).

#### Contributions

1. In Section 3.2 we provide a time- and value-uniform upper bound on the CDF of the averaged historical conditional distribution of a discrete-time real-valued random process. Consistent with the lack of sequential uniform convergence of linear threshold functions (Rakhlin et al., 2015), the bounds are \(\underline{\text{Always}}\)\(\underline{\text{Valid}}\) (see Theorem 3.1) And \(\underline{\text{Sometimes}}\)\(\underline{\text{Trivial}}\), i.e., the width guarantee is instance-dependent (see Theorem 3.3) and may not converge to zero width in the infinite data limit. When the data generating process is smooth with respect to the uniform distribution on the unit interval, the bound width adapts to the unknown smoothness parameter, following the framework of smoothed online learning (Rakhlin et al., 2011; Haghtalab et al., 2020, 2022b, 20, 2022).
2. In Section 3.3 we extend the previous technique to distributions with support over the entire real line, and further to distributions with a known countably infinite or unknown nowhere dense set of discrete jumps; with analogous instance-dependent guarantees.
3. In Section 3.4 we extend the previous techniques to importance-weighted random variables, achieving our ultimate goal of estimating a complete counterfactual distribution of outcomes.

We exhibit our techniques in various simulations in Section 4. Computationally our procedures have comparable cost to point estimation of the empirical CDF, as the empirical CDF is a sufficient statistic.

## 2 Problem Setting

Let \(\left(\Omega,\mathcal{F},\left\{\mathcal{F}_{t}\right\}_{t\in\mathbb{N}}, \mathbb{P}\right)\) be a probability space equipped with a discrete-time filtration, on which let \(X_{t}\) be an adapted, real-valued random process. Let \(\mathbb{E}_{t}\left[\cdot\right]\doteq\mathbb{E}\left[\cdot|\mathcal{F}_{t}\right]\). The quantity of interest is the (random) map \(\overline{\text{CDF}}_{t}:\mathbb{R}\rightarrow\left[0,1\right]\), the CDF of the averaged historical conditional distribution at time \(t\):

\[\overline{\text{CDF}}_{t}(v)\doteq\frac{1}{t}\sum_{s\in t}\mathbb{E}_{s-1} \left[1_{X_{s}\leq v}\right]. \tag{1}\]

We desire simultaneously time- and value-uniform bounds which hold with high probability, i.e., adapted sequences of maps \(L_{t},U_{t}:\mathbb{R}\rightarrow\left[0,1\right]\) satisfying

\[\mathbb{P}\left(\tfrac{\text{\tiny{freq}}\mathbb{N}}{\text{\tiny{freq}}\mathbb{ N}}:L_{t}(v)\leq\overline{\text{CDF}}_{t}(v)\leq U_{t}(v)\right)\geq 1-2\alpha. \tag{2}\]

Note that the estimand \(\overline{\text{CDF}}_{t}\) is changing at each time step as we incorporate the conditional distribution of the latest observation into our historical average. The maps \(L_{t},U_{t}\) provide a sequence of confidence bands which contain this sequence of changing CDFs uniformly over time with high probability.

In the i.i.d. setting, Equation (1) is deterministic and independent of \(t\), reducing to the CDF of the (unknown) generating distribution. In this setting, the classic results of Glivenko (1933) and

\begin{table}
\begin{tabular}{l|c c c|c c|c} \hline \hline Reference & Time- & Non- & Non- & Non- & Counter- & \(w_{\max}\)- \\  & Uniform? & stationary? & asymptotic? & parametric? & factual? & free? \\ \hline HR22 & ✓ & & ✓ & ✓ & & N/A \\ HLLA21 & & & ✓ & ✓ & ✓ & \\ UnO21, [IID] & & & ✓ & ✓ & ✓ & ✓ \\ UnO21, [NS] & & ✓ & & & ✓ & ✓ \\ WS22, [84] & ✓ & & ✓ & ✓ & ✓ & ✓ \\ this paper & ✓ & ✓ & ✓ & ✓ & ✓ & ✓ \\ \hline \hline \end{tabular}
\end{table}
Table 1: Comparison to prior art for quantile-uniform CDF estimation. _Time-uniform_: a sequence of confidence bands whose coverage holds uniformly over time with high probability. _Nonstationary_: does not require an i.i.d. assumption. _Non-asymptotic_: guarantees hold at all sample sizes. _Non-parametric_: guarantees apply over an infinite-dimensional class of distributions. _Counterfactual_: method applies to estimating a distribution other than the one which generated the observed data, via importance weighting. \(w_{\max}\)_-free_: guarantee applies without a bound on the maximum importance weight. See Section 5 for details.

Cantelli (1933) established uniform convergence of linear threshold functions; subsequently the Dvoretzky-Kiefer-Wolfowitz (DKW) inequality characterized fixed-time and value-uniform convergence rates (Dvoretzky et al., 1956; Massart, 1990); extended later to simultaneously time- and value-uniform bounds (Howard and Ramdas, 2022). The latter result guarantees an \(O(t^{-1}\log(\log(t)))\) confidence interval width, matching the limit imposed by the Law of the Iterated Logarithm.

AVAST principleIn contrast, under arbitrary data dependence, linear threshold functions are not sequentially uniformly convergent, i.e., the averaged historical empirical CDF does not necessarily converge uniformly to the CDF of the averaged historical conditional distribution (Rakhlin et al., 2015). Consequently, additional assumptions are required to provide a guarantee that the confidence width decays to zero. In this paper we design bounds that are Always Valid And Sometimes Trivial, i.e., under worst-case data generation, \(\sup_{v}\lvert U_{t}(v)-L_{t}(v)\rvert=O(1)\) as \(t\to\infty\). Fortunately our bounds are also equipped with an instance-dependent width guarantee based upon the smoothness of the distribution to a reference measure qua Definition 3.2.

Additional NotationLet \(X_{a:b}=\{X_{s}\}_{s=a}^{b}\) denote a contiguous subsequence of a random process. Let \(\mathbb{P}_{t}\) denote the average historical conditional distribution, defined as a (random) distribution over the sample space \(\mathbb{R}\) by \(\mathbb{P}_{t}(A)\doteq t^{-1}\sum_{s\leqslant t}\mathbb{E}_{s-1}\left[1_{X_{ s}\in A}\right]\) for a Borel subset \(A\) (note \(\mathbb{P}_{t}\) represents the entire historical average while \(\mathbb{E}_{t}\) corresponds to a single conditional distribution).

## 3 Derivations

### High Level Design

Our approaches work as reductions, achieving the value- and time-uniform guarantee of Equation (2) by combining bounds \(\Lambda_{t},\Xi_{t}\) that satisfy a time-uniform guarantee at any fixed value \(\rho\),

\[\mathbb{P}\left(\forall t\in\mathbb{N}:\Lambda_{t}(\rho)\leqslant\overline{ \operatorname{CDF}}_{t}(\rho)\leqslant\Xi_{t}(\rho)\right)\geqslant 1-\delta( \rho). \tag{3}\]

The bounds \(\Lambda_{t},\Xi_{t}\) are tools for estimating a sequence of _scalars_, in this case \((\overline{\operatorname{CDF}}_{t}(\rho))_{t=1}^{\infty}\) for a fixed value \(\rho\). We show how to extend such tools to the more difficult problem of estimating a sequence of (cumulative distribution) _functions_.

There are multiple existing approaches to obtaining the guarantee of Equation (3): we provide a self-contained introduction in Appendix A. For ease of exposition, we will only discuss how to construct

Figure 1: Visualization of Algorithm 1. The values of interest are uncountably infinite; the algorithm allocates probability to maintain upper bounds on a countably infinite set of points \(\rho\) at different resolution levels via the monotonicity of \(\overline{\operatorname{CDF}}_{t}(v)\). As resolution increases, the value \(\rho\) better approximates \(v\), but the allocated probability decreases; the algorithm chooses the tightest of available bounds. Shaded nodes would be consulted for an upper bound for \(v=5/17\).

a time- and value-uniform upper bound by combining fixed-value, time-uniform upper bounds, and defer the analogous lower bound construction to Appendix B.2. Our approach is to compose these fixed-value bounds into a value-uniform bound by taking a union bound over a particular collection of values, leveraging monotonicity of the CDF.

Quantile vs Value SpaceIn the i.i.d. setting, a value-uniform guarantee can be obtained by taking a careful union bound over the unique value associated with each quantile (Howard and Ramdas, 2022). This "quantile space" approach has advantages, e.g., variance based discretization and covariance to monotonic transformations. However, under arbitrary data dependence, the value associated with each quantile can change. Therefore we proceed in "value space". See Appendix A.1 for more details.

### On the Unit Interval

Algorithm 1, visualized in Figure 1, constructs an upper bound on Equation (1) which, while valid for all values, is designed for random variables ranging over the unit interval. For a given value \(v\), it searches over upper bounds on the CDF evaluated at a decreasing sequence of values \(\rho_{1}\geq\rho_{2}\geq\cdots\geq v\) and exploits monotonicity of \(\overline{\mathrm{CDF}}_{t}(v)\). That is, at each level \(d=1,2,\dots\), we construct a discretizing grid of size \(\epsilon(d)\) over the unit interval, and construct a time-uniform upper bound on \(\overline{\mathrm{CDF}}_{t}(\rho)\) for each grid point \(\rho\) using the fixed-value confidence sequence oracle \(\Xi_{t}\). Then, for a given value \(v\), at each level \(d\) we make use of the fixed-value confidence sequence for smallest grid point \(\rho_{d}\geq v\), and we search for the level \(d\) which yields the minimal upper confidence bound. A union bound over the (countably infinite) possible choices for \(\rho_{d}\) controls the coverage of the overall procedure. Because the error probability \(\delta_{d}\) decreases with \(d\) (and the fixed-value confidence radius \(\Xi_{t}\) increases as \(\delta\) decreases), the procedure can terminate whenever no observations remain between the desired value \(v\) and the current upper bound \(\rho_{d}\), as all subsequent bounds are dominated.

The lower bound is derived analogously in Algorithm 2 (which we have left to Appendix B.2 for the sake of brevity) and leverages a lower confidence sequence \(\Lambda_{t}\left(\rho;\delta,\Psi_{t}\right)\) (instead of an upper confidence sequence) evaluated at an increasingly refined lower bound on the value \(\rho\leftarrow\epsilon(d)^{-1}|\epsilon(d)v|\).

**Theorem 3.1**.: _If \(\epsilon(d)\uparrow\infty\) as \(d\uparrow\infty\), then Algorithms 1 and 2 terminate with probability one. Furthermore, if for all \(\rho\), \(\delta\), and \(d\) the algorithms \(\Lambda_{t}(\rho;\delta,\Psi_{t})\) and \(\Xi_{t}(\rho;\delta,\Psi_{t})\) satisfy_

\[P(\forall t:\overline{\mathrm{CDF}}_{t}(\rho)\geq\Lambda_{t}( \rho;\delta,\Psi_{t}))\geq 1-\delta, \tag{4}\] \[P(\forall t:\overline{\mathrm{CDF}}_{t}(\rho)\leq\Xi_{t}(\rho; \delta,\Psi_{t}))\geq 1-\delta, \tag{5}\]

_then guarantee (2) holds with \(U_{t},L_{t}\) given by the outputs of Algorithms 1 and 2, respectively._

Proof.: See Appendix B.3.

Theorem 3.1 ensures Algorithms 1 and 2 yield the desired time- and value-uniform coverage, essentially due to the union bound and the coverage guarantees of the oracles \(\Xi_{t},\Lambda_{t}\). However, coverage is also guaranteed by the trivial bounds \(0\leq\overline{\mathrm{CDF}}_{t}(v)\leq 1\). The critical question is: what is the bound width?

Smoothed Regret GuaranteeEven assuming \(X\) is entirely supported on the unit interval, on what distributions will Algorithm 1 provide a non-trivial bound? Because each \([\Lambda_{t}(\rho;\delta,\Psi_{t}),\Xi_{t}(\rho;\delta,\Psi_{t})]\) is a confidence sequence for the mean of the bounded random variable \(1_{X_{s}\leq\rho}\), we enjoy width guarantees at each of the (countably infinite) \(\rho\) which are covered by the union bound, but the guarantee degrades as the depth \(d\) increases. If the data generating process focuses on an increasingly small part of the unit interval over time, the width guarantees on our discretization will be insufficient to determine the distribution. Indeed, explicit constructions demonstrating the lack of sequential uniform convergence of linear threshold functions increasingly focus in this manner (Block et al., 2022).

Conversely, if \(\forall t:\overline{\mathrm{CDF}}_{t}(v)\) was Lipschitz continuous in \(v\), then our increasingly granular discretization would eventually overwhelm any fixed Lipschitz constant and guarantee uniform convergence. Theorem 3.3 expresses this intuition, but using the concept of smoothness rather than Lipschitz, as smoothness will allow us to generalize further (Rakhlin et al., 2011; Haghtalab et al., 2020, 2022b; Bacon et al., 2022).

**Definition 3.2**.: A distribution \(D\) is \(\xi\)-smooth wrt reference measure \(M\) if \(D\ll M\) and \(\operatorname{ess\,sup}_{M}\left({}^{\text{dD}}\!/_{\text{dM}}\right)\leqslant \xi^{-1}\).

When the reference measure is the uniform distribution on the unit interval, \(\xi\)-smoothness implies an \(\xi^{-1}\)-Lipschitz CDF. However, when the reference measure has its own curvature, or charges points, the concepts diverge. When reading Theorem 3.3, note \(\xi\leqslant 1\) (since the reference measure is a probability distribution) and as \(\xi\to 0\) the smoothness constraint is increasingly relaxed. Thus Theorem 3.3 states "for less smooth distributions, convergence is slowed."

**Theorem 3.3**.: _Let \(U_{t}(v)\) and \(L_{t}(v)\) be the upper and lower bounds returned by Algorithm 1 and Algorithm 2 respectively, when evaluated with \(\epsilon(d)=2^{d}\) and the confidence sequences \(\Lambda_{t}\) and \(\Xi_{t}\) of Equation (15). If \(\forall t:\mathbb{P}_{t}\) is \(\xi_{t}\)-smooth wrt the uniform distribution on the unit interval then_

\[\forall t,\forall v:U_{t}(v)-L_{t}(v)\leqslant \tag{6}\] \[\sqrt{\frac{V_{t}}{t}}+\tilde{O}\left(\sqrt{\frac{V_{t}}{t}}\log \left(\xi_{t}^{-2}\alpha^{-1}t^{3/2}\right)\right),\]

_where \(q_{t}\doteq\overline{\operatorname{CDF}}_{t}(v)\); \(V_{t}\doteq\nicefrac{{1}}{{t}}+\nicefrac{{(q_{t}-1/2)}}{{\log(q_{t}/1-q_{t})}}\); and \(\tilde{O}()\) elides polylog \(V_{t}\) factors._

Proof.: See Appendix C.

Theorem 3.3 matches our empirical results in two important aspects: (i) logarithmic dependence upon smoothness (e.g., Figure 4); (ii) tighter intervals for more extreme quantiles (e.g., Figure 2). Note the choice \(\epsilon(d)=2^{d}\) ensures the loop in Algorithm 1 terminates after at most \(\log_{2}(\Delta)\) iterations, where \(\Delta\) is the minimum difference between two distinct realized values.

Worked ExampleTo build intuition, in Appendix B.1 we explicitly calculate Algorithm 1 for a synthetic data set.

### Extensions

Arbitrary SupportIn Appendix D.1 we describe a variant of Algorithm 1 which uses a countable dense subset of the entire real line. It enjoys a similar guarantee to Theorem 3.3, but with an additional width which is logarithmic in the probe value \(v\): \(\tilde{O}\left(\sqrt{\frac{V_{t}}{t}}\log\left(\left(2+\xi_{t}|v|t^{-1/2} \right)^{2}\xi_{t}^{-2}\alpha^{-1}t^{3/2}\right)\right)\). Note in this case \(\xi_{t}\) is defined relative to (unnormalized) Lebesgue measure and can therefore exceed 1.

Discrete JumpsIf \(\mathbb{P}_{t}\) is smooth wrt a reference measure which charges a countably infinite number of known discrete points, we can explicitly union bound over these additional points proportional to their density in the reference measure. In this case we preserve the above value-uniform guarantees. See Appendix D.2 for more details.

For distributions which charge unknown discrete points, we note the proof of Theorem 3.3 only exploits smoothness local to \(v\). Therefore if the set of discrete points is nowhere dense, we eventually recover the guarantee of Equation (6) after a "burn-in" time \(t\) which is logarithmic in the minimum distance from \(v\) to a charged discrete point.

Figure 3: Nonstationary Polya simulation for two seeds approaching different average conditional CDFs. Bounds successfully track the true CDFs in both cases. See Section 4.2.

Figure 2: CDF bounds approaching the true CDF when sampling i.i.d. from a Beta(6,3) distribution. Note these bounds are simultaneously valid for all times and values.

### Importance-Weighted Variant

An important use case is estimating a distribution based upon observations produced from another distribution with a known shift, e.g., arising in transfer learning (Pan and Yang, 2010) or off-policy evaluation (Waudby-Smith et al., 2022). In this case the observations are tuples \((W_{t},X_{t})\), where the importance weight \(W_{t}\) is a Radon-Nikodym derivative, implying \(\forall t:\mathbb{E}_{t}\left[W_{t}\right]=1\) and a.s. \(W_{t}\geq 0\); and the goal is to estimate \(\overline{\mathrm{CDF}}_{t}(v)=t^{-1}\sum_{s\in t}\mathbb{E}_{s-1}\left[W_{s}1 \chi_{s\leq v}\right]\). The basic approach in Algorithm 1 and Algorithm 2 is still applicable in this setting, but different \(\Lambda_{t}\) and \(\Xi_{t}\) are required. In Appendix E we present details on two possible choices for \(\Lambda_{t}\) and \(\Xi_{t}\): the first is based upon the empirical Bernstein construction of Howard et al. (2021), and the second based upon the DDRM construction of Mineiro (2022). Both constructions leverage the \(L^{*}\) Adagrad bound of Orabona (2019) to enable lazy evaluation. The empirical Bernstein version is amenable to analysis and computationally lightweight, but requires finite importance weight variance to converge (the variance bound need not be known, as the construction adapts to the unknown variance). The DDRM version requires more computation but produces tighter intervals. See Section 4.1 for a comparison.

Inspired by the empirical Bernstein variant, the following analog of Theorem 3.3 holds. Note \(\mathbb{P}_{t}\) is the target (importance-weighted) distribution, not the observation (non-importance-weighted) distribution.

**Theorem 3.4**.: _Let \(U_{t}(v)\) and \(L_{t}(v)\) be the upper and lower bounds returned by Algorithm 1 and Algorithm 2 respectively with \(\epsilon(d)=2^{d}\) and the confidence sequences \(\Lambda_{t}\) and \(\Xi_{t}\) of Equation (18). If \(\forall t:\mathbb{P}_{t}\) is \(\xi_{t}\)-smooth wrt the uniform distribution on the unit interval then_

\[\begin{split}\forall t,&\forall v:U_{t}(v)-L_{t}(v) \leq\\ & B_{t}+\sqrt{\frac{(\tau+V_{t})/t}{t}}\\ &+\tilde{O}\left(\sqrt{\frac{(\tau+V_{t})/t}{t}}\log\left(\xi_{t} ^{-2}\alpha^{-1}\right)\right)\\ &+\tilde{O}(t^{-1}\log\left(\xi_{t}^{-2}\alpha^{-1}\right)),\end{split} \tag{7}\]

_where \(q_{t}\doteq\overline{\mathrm{CDF}}_{t}(v)\), \(K(q_{t})\doteq{(q_{t}-1/2)/\log(q_{t}/_{1-q_{t}})}\); \(V_{t}=O\left(K(q_{t})\sum_{s\leq t}W_{s}^{2}\right)\), \(B_{t}\doteq t^{-1}\sum_{s\leq t}(W_{s}-1)\), and \(\tilde{O}()\) elides polylog \(V_{t}\) factors._

Proof.: See Appendix E.2.

Theorem 3.4 exhibits the following key properties: (i) logarithmic dependence upon smoothness; (ii) tighter intervals for extreme quantiles and importance weights with smaller quadratic variation; (iii) no explicit dependence upon importance weight range; (iv) asymptotic zero width for importance weights with sub-linear quadratic variation.

Additional RemarksFirst, the importance-weighted average CDF is a well-defined mathematical quantity, but the interpretation as a counterfactual distribution of outcomes given different actions in the controlled experimentation setting involves subtleties: we refer the interested reader to Waudby-Smith et al. (2022) for a complete discussion. Second, the need for nonstationarity techniques for

Figure 4: As smoothness’decreases, we require more time to reach the same maximum confidence width. For low smoothness, DKW dominates our method. The logarithmic dependence matches our theory. See Section 4.1.

Figure 5: CDF bounds’approaching the true counterfactual CDF when sampling i.i.d. from a Beta(6,3) with infinite-variance importance weights, using DDRM for the oracle confidence sequence.

estimating the importance-weighted CDF is driven by the outcomes \((X_{t})\) and not the importance-weights \((W_{t})\). For example with off-policy contextual bandits, a changing historical policy does not induce nonstationarity, but a changing conditional reward distribution does.

## 4 Simulations

These simulations explore the empirical behaviour of Algorithm 1 and Algorithm 2 when instantiated with \(\epsilon(d)=2^{d}\) and curved boundary oracles \(\Lambda\) and \(\Xi\). To save space, precise details on the experiments as well additional figures are elided to Appendix F. Reference implementations which reproduce the figures are available at [https://github.com/microsoft/cserbust](https://github.com/microsoft/cserbust).

### The i.i.d. setting

These simulations exhibit our techniques on i.i.d. data. Although the i.i.d. setting does not fully exercise the technique, it is convenient for visualizing convergence to the unique true CDF. In this setting the DKW inequality applies, so to build intuition about our statistical efficiency, we compare our bounds with a naive time-uniform version of DKW resulting from a \((\nicefrac{{6}}{{\pi^{2}}}\epsilon^{2})\) union bound over time.

Beta distributionIn this case the data is smooth wrt the uniform distribution on \([0,1]\) so we can directly apply Algorithm 1 and Algorithm 2. Figure 2 shows the bounds converging to the true CDF as \(t\) increases for an i.i.d. Beta\((6,3)\) realization. Figure 8 compares the bound width to time-uniform DKW at \(t=10000\) for Beta distributions that are increasingly less smooth with respect to the uniform distribution. The DKW bound is identical for all, but our bound width increases as the smoothness decreases.

The additional figures in Appendix F clearly indicate tighter bounds at extreme quantiles, in correspondence with Theorem 3.3.

Beyond the unit intervalIn Figure 7 (main text) and Appendix F.1 we present further simulations of i.i.d. lognormal and Gaussian random variables, ranging over \(\mathbb{R}^{+}\) and \(\mathbb{R}\) respectively, and using Algorithm 3. The logarithmic dependence of the bound width upon the probe value is evident.

An Exhibition of FailureFigure 4 shows the (empirical) relative convergence when the data is simulated i.i.d. uniform over \([0,\epsilon]\) for decreasing \(\epsilon\) (hence decreasing smoothness). The reference width is the maximum bound width obtained with Algorithm 1 and Algorithm 2 at \(t_{\text{ref}}=10000\) and \(\epsilon=1/16\), and shown is the multiplicative factor of time required for the maximum bound width to match the reference width as smoothness varies. The trend is consistent with arbitrarily poor convergence with arbitrarily small \(\epsilon\). Because this is i.i.d. data, DKW applies and a uniform bound (independent of \(\epsilon\)) is available. Thus while our instance-dependent guarantees are valuable in practice, they can be dominated by stronger guarantees leveraging additional assumptions. On a positive note, a logarithmic dependence on smoothness is evident over many orders of magnitude, confirming the analysis of Theorem 3.3.

Figure 6: A nonstationary, importance-weighted simulation in which the factual distribution (red) diverges dramatically from the counterfactual distribution (green). The bound correctly covers the counterfactual CDF.

Importance-WeightedIn these simulations, in addition to being i.i.d., \(X_{t}\) and \(W_{t}\) are drawn independently of each other, so the importance weights merely increase the difficulty of ultimately estimating the same quantity.

In the importance-weighted case, an additional aspect is whether the importance-weights have finite or infinite variance. Figures 5 and 13 demonstrate convergence in both conditions when using DDRM for pointwise bounds. Figures 14 and 15 show the results using empirical Bernstein pointwise bounds. In theory, with enough samples and infinite precision, the infinite variance Pareto simulation would eventually cause the empirical Bernstein variant to reset to trivial bounds, but in practice this is not observed. Instead, DDRM is consistently tighter but also consistently more expensive to compute, as exemplified in Table 2. Thus either choice is potentially preferable.

(i.e. not time-uniform). In other words, given a sample of i.i.d. random variables \(X_{1},\ldots,X_{n}\sim F\), these fixed time bounds \([\dot{L}_{n}(x),\dot{U}_{n}(x)]_{x\in\mathbb{R}}\) satisfy a guarantee of the form:

\[\mathbb{P}(\forall x\in\mathbb{R},\;\dot{L}_{n}(x)\leq F(x)\leq\dot{U}_{n}(x)) \geq 1-\alpha, \tag{8}\]

for any desired error level \(\alpha\in(0,1)\). Howard and Ramdas (2022) developed confidence bands \([\overline{L}_{t}(x),\,\overline{U}_{t}(x)]_{x\in\mathbb{R},t\in\mathbb{N}}\) that are both quantile- _and_ time-uniform, meaning that they satisfy the stronger guarantee:

\[\mathbb{P}(\forall x\in\mathbb{R},t\in\mathbb{N},\;\overline{L}_{t}(x)\leq F(x )\leq\bar{U}_{t}(x))\geq 1-\alpha. \tag{9}\]

However, the bounds presented in Howard and Ramdas (2022) ultimately focused on the classical i.i.d. _on-policy_ setup, meaning the CDF for which confidence bands are derived is the same CDF as those of the observations \((X_{t})_{t=1}^{\infty}\). This is in contrast to off-policy evaluation problems such as in randomized controlled trials, adaptive A/B tests, or contextual bandits, where the goal is to estimate a distribution different from that which was collected (e.g. collecting data based on a Bernoulli experiment with the goal of estimating the counterfactual distribution under treatment or control). Chandak et al. (2021) and Huang et al. (2021) both introduced fixed-time (i.e. non-time-uniform) confidence bands for the off-policy CDF in contextual bandit problems, though their procedures are quite different, rely on different proof techniques, and have different properties from one another. Waudby-Smith et al. (2022, Section 4) later developed _time-uniform_ confidence bands in the off-policy setting, using a technique akin to Howard and Ramdas (2022, Theorem 5) and has several desirable properties in comparison to Chandak et al. (2021) and Huang et al. (2021) as outlined in Waudby-Smith et al. (2022, Table 2).

Nevertheless, regardless of time-uniformity or on/off-policy estimation, all of the aforementioned prior works assume that the distribution to be estimated is _fixed and unchanging over time_. The present paper takes a significant departure from the existing literature by deriving confidence bands that allow the distribution to change over time in a data-dependent manner, all while remaining time-uniform and applicable to off-policy problems in contextual bandits. Moreover, we achieve this by way of a novel stitching technique which is closely related to those of Howard and Ramdas (2022) and Waudby-Smith et al. (2022).

## 6 Discussion

This work constructs bounds by tracking specific values, in contrast with i.i.d. techniques which track specific quantiles. The value-based approach is amenable to proving correctness qua Theorem 3.1, but has the disadvantage of sensitivity to monotonic transformations. We speculate it is possible to be covariant to a fixed (wrt time) but unknown monotonic transformation without violating known impossibility results. A technique with this property would have increased practical utility.

## Acknowledgments and Disclosure of Funding

The authors thank Ian Waudby-Smith for insightful discussion and review.

## References

* Block et al. (2022) Adam Block, Yuval Dagan, Noah Golowich, and Alexander Rakhlin. Smoothed online learning is as easy as statistical learning. _arXiv preprint arXiv:2202.04690_, 2022.
* Cantelli (1933) Francesco Paolo Cantelli. Sulla determinazione empirica delle leggi di probabilita. _Giorn. Ist. Ital. Attuari_, 4(421-424), 1933.
* Chandak et al. (2021) Yash Chandak, Scott Niekum, Bruno da Silva, Erik Learned-Miller, Emma Brunskill, and Philip S Thomas. Universal off-policy evaluation. _Advances in Neural Information Processing Systems_, 34:27475-27490, 2021.
* Chatzigeorgiou (2013) Ioannis Chatzigeorgiou. Bounds on the lambert function and their application to the outage analysis of user cooperation. _IEEE Communications Letters_, 17(8):1505-1508, 2013.
* Chatzigeorgiou et al. (2021)Aryeh Dvoretzky, Jack Kiefer, and Jacob Wolfowitz. Asymptotic minimax character of the sample distribution function and of the classical multinomial estimator. _The Annals of Mathematical Statistics_, pages 642-669, 1956.
* Fan et al. (2015) Xiequan Fan, Ion Grama, and Quansheng Liu. Exponential inequalities for martingales with applications. _Electronic Journal of Probability_, 20:1-22, 2015.
* Feller (1958) William Feller. _An introduction to probability theory and its applications, 3rd edition_. Wiley series in probability and mathematical statistics, 1958.
* Glivenko (1933) Valery Glivenko. Sulla determinazione empirica delle leggi di probabilita. _Gion. Ist. Ital. Attauri._, 4:92-99, 1933.
* Haghtalab et al. (2020) Nika Haghtalab, Tim Roughgarden, and Abhishek Shetty. Smoothed Analysis of Online and Differentially Private Learning. In _Advances in Neural Information Processing Systems_, volume 33, pages 9203-9215, 2020.
* Haghtalab et al. (2022a) Nika Haghtalab, Yanjun Han, Abhishek Shetty, and Kunhe Yang. Oracle-Efficient Online Learning for Beyond Worst-Case Adversaries, November 2022a. arXiv:2202.08549 [cs, stat].
* Haghtalab et al. (2022b) Nika Haghtalab, Tim Roughgarden, and Abhishek Shetty. Smoothed Analysis with Adaptive Adversaries. In _2021 IEEE 62nd Annual Symposium on Foundations of Computer Science (FOCS)_, pages 942-953, 2022b.
* Howard and Ramdas (2022) Steven R Howard and Aaditya Ramdas. Sequential estimation of quantiles with applications to A/B testing and best-arm identification. _Bernoulli_, 28(3):1704-1728, 2022.
* Howard et al. (2021) Steven R Howard, Aaditya Ramdas, Jon McAuliffe, and Jasjeet Sekhon. Time-uniform, nonparametric, nonasymptotic confidence sequences. _The Annals of Statistics_, 49(2):1055-1080, 2021.
* Huang et al. (2021) Audrey Huang, Liu Leqi, Zachary Lipton, and Kamyar Azizzadenesheli. Off-policy risk assessment in contextual bandits. _Advances in Neural Information Processing Systems_, 34:23714-23726, 2021.
* Kearns and Saul (1998) Michael Kearns and Lawrence Saul. Large deviation methods for approximate probabilistic inference. In _Proceedings of the Fourteenth conference on Uncertainty in artificial intelligence_, pages 311-319, 1998.
* Lindon et al. (2022) Michael Lindon, Chris Sanden, and Vache Shirikian. Rapid regression detection in software deployments through sequential testing. In _Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining_, pages 3336-3346, 2022.
* Liu (2021) Wentao Liu. _Risk-Aware Financial Portfolio Management with Distributional Deep Deterministic Policy Gradient_. PhD thesis, University of Toronto (Canada), 2021.
* Massart (1990) Pascal Massart. The tight constant in the Dvoretzky-Kiefer-Wolfowitz inequality. _The annals of Probability_, pages 1269-1283, 1990.
* Mineiro (2022) Paul Mineiro. A lower confidence sequence for the changing mean of non-negative right heavy-tailed observations with bounded mean. _arXiv preprint arXiv:2210.11133_, 2022.
* Olver et al. (2010) Frank WJ Olver, Daniel W Lozier, Ronald F Boisvert, and Charles W Clark. _NIST handbook of mathematical functions hardback and CD-ROM_. Cambridge university press, 2010.
* Orabona (2019) Francesco Orabona. A modern introduction to online learning. _arXiv preprint arXiv:1912.13213_, 2019.
* Pan and Yang (2010) Sinno Jialin Pan and Qiang Yang. A survey on transfer learning. _IEEE Transactions on knowledge and data engineering_, 22(10):1345-1359, 2010.
* Pinelis (2020) Iosif Pinelis. Exact lower and upper bounds on the incomplete gamma function. _arXiv preprint arXiv:2005.06384_, 2020.
* Pan and Yang (2010)Alexander Rakhlin, Karthik Sridharan, and Ambuj Tewari. Online Learning: Stochastic, Constrained, and Smoothed Adversaries. In _Advances in Neural Information Processing Systems_, volume 24, 2011.
* Rakhlin et al. (2015) Alexander Rakhlin, Karthik Sridharan, and Ambuj Tewari. Sequential complexities and uniform martingale laws of large numbers. _Probability theory and related fields_, 161(1):111-153, 2015.
* Shafer et al. (2011) Glenn Shafer, Alexander Shen, Nikolai Vereshchagin, and Vladimir Vovk. Test martingales, Bayes factors and p-values. _Statistical Science_, 26(1):84-101, 2011.
* Shen et al. (2022) Yi Shen, Jessilyn Dunn, and Michael M Zavlanos. Risk-averse multi-armed bandits with unobserved confounders: A case study in emotion regulation in mobile health. In _2022 IEEE 61st Conference on Decision and Control (CDC)_, pages 144-149. IEEE, 2022.
* Wang and Chapman (2022) Yuheng Wang and Margaret P Chapman. Risk-averse autonomous systems: A brief history and recent developments from the perspective of optimal control. _Artificial Intelligence_, page 103743, 2022.
* Waudby-Smith et al. (2022) Ian Waudby-Smith, Lili Wu, Aaditya Ramdas, Nikos Karampatziakis, and Paul Mineiro. Anytime-valid off-policy inference for contextual bandits. _arXiv preprint arXiv:2210.10768_, 2022.

Conflidence Sequences for Fixed \(v\)

Since our algorithm operates via reduction to pointwise confidence sequences, we provide a brief self-contained review here. We refer the interested reader to Howard et al. (2021) for a more thorough treatment.

A confidence sequence for a random process \(X_{t}\) is a time-indexed collection of confidence sets \(\text{CI}_{t}\) with a time-uniform coverage property \(\mathbb{P}\left(\forall t\in\mathbb{N}:X_{t}\in\text{CI}_{t}\right)\geqslant 1-\alpha\). For real random variables, the concept of a lower confidence sequence can be defined via \(\mathbb{P}\left(\forall t\in\mathbb{N}:X_{t}\geqslant L_{t}\right)\geqslant 1-\alpha\), and analogously for upper confidence sequences; and a lower and upper confidence sequence can be combined to form a confidence sequence \(\text{CI}_{t}\doteq\{x|L_{t}\leqslant x\leqslant U_{t}\}\) with coverage \((1-2\alpha)\) via a union bound.

One method for constructing a lower confidence sequence for a real valued parameter \(z\) is to exhibit a real-valued random process \(E_{t}(z)\) which, when evaluated at the true value \(z\)* of the parameter of interest, is a non-negative supermartingale with initial value of 1, in which case Ville's inequality ensures \(\mathbb{P}\left(\forall t\in\mathbb{N}:E_{t}(z^{\text{*}})\leqslant\alpha^{-1 }\right)\geqslant 1-\alpha\). If the process \(E_{t}(z)\) is monotonically increasing in \(z\), then the supremum of the lower contour set \(L_{t}\doteq\sup_{z}\left\{z|E_{t}(z)\leqslant\alpha^{-1}\right\}\) is suitable as a lower confidence sequence; an upper confidence sequence can be analogously defined.

We use the above strategy as follows. We bound these deviations using the following nonnegative martingale,

\[E_{t}(\lambda)\doteq\exp\left(\lambda S_{t}-\sum_{s\leqslant t} \log\left(h(\lambda,\theta_{s})\right)\right), \tag{10}\]

where \(\lambda\in\mathbb{R}\) is fixed and \(h(\lambda,z)\doteq(1-z)e^{-\lambda z}+ze^{\lambda(1-z)}\), the moment-generating function of a centered Bernoulli\((z)\) random variable. Equation (10) is a test martingale qua Shafer et al. (2011), i.e., it can be used to construct time-uniform bounds on \(\hat{q}_{t}-q_{t}\) via Ville's inequality.

Next we lower bound Equation (10),

\[E_{t}(\lambda)\doteq\exp\left(\lambda S_{t}-\sum_{s\leqslant t} \log\left(h(\lambda,\theta_{s})\right)\right), \tag{10}\]

and eliminate the explicit dependence upon \(\theta_{s}\), by noting \(\log h(\lambda,\cdot)\) is concave and therefore

\[E_{t}(\lambda)\geqslant\exp\left(\lambda t\left(q_{t}-\hat{q}_{ t}\right)-t\ \log h\left(\lambda,q_{t}\right)\right), \tag{11}\]

because \(\left(tf(q)=\max_{\theta\big{|}1^{\top}\theta=tq}\sum_{s\leqslant t}f(\theta_ {s})\right)\) for any concave \(f\). Equation (11) is monotonically increasing in \(q_{t}\) and therefore defines a lower confidence sequence. For an upper confidence sequence we use \(q_{t}=1-(1-q_{t})\) and a lower confidence sequence on \((1-q_{t})\).

Regarding the choice of \(\lambda\), in practice many \(\lambda\) are (implicitly) used via stitching (i.e., using different \(\lambda\) in different time epochs and majorizing the resulting bound in closed form) or mixing (i.e., using a particular fixed mixture of Equation (11) via a discrete sum or continuous integral over \(\lambda\)); our choices will depend upon whether we are designing for tight asymptotic rates or low computational footprint. We provide specific details associated with each theorem or experiment.

Note Equation (11) is invariant to permutations of \(X_{1:t}\) and hence the empirical CDF at time \(t\) is a sufficient statistic for calculating Equation (11) at any \(v\).

### Challenge with quantile space

In this section assume all CDFs are invertible for ease of exposition.

In the i.i.d. setting, Equation (10) can be evaluated at the (unknown) fixed \(v(q)\) which corresponds to quantile \(q\). Without knowledge of the values, one can assert the existence of such values for a countably infinite collection of quantiles and a careful union bound of Ville's inequality on a particular discretization can yield an LIL rate: this is the approach of Howard and Ramdas (2022). A key advantage of this approach is covariance to monotonic transformations.

Beyond the i.i.d. setting, one might hope to analogously evaluate Equation (10) at an unknown fixed value \(v_{t}(q)\) which for each \(t\) corresponds to quantile \(q\). Unfortunately, \(v_{t}(q)\) is not just unknown,but also unpredictable with respect to the initial filtration, and the derivation that Equation (10) is a martingale depends upon \(v\) being predictable. In the case that \(X_{t}\) is independent but not identically distributed, \(v_{t}(q)\) is initially predictable and therefore this approach could work, but would only be valid under this assumption.

The above argument does not completely foreclose the possibility of a quantile space approach, but merely serves to explain why the authors pursued a value space approach in this work. We encourage the interested reader to innovate.

## Appendix B Unit Interval Bounds

### Worked Example

Our (synthetic) data set consists of five values, each of which has occurred 1000 times:

\[D=\left\{(1000,0),(1000,\nicefrac{{1}}{{7}}),(1000,\nicefrac{{2}}{{7}}),(1000, \nicefrac{{3}}{{7}}),(1000,\nicefrac{{6}}{{7}})\right\}.\]

We use resolution \(\epsilon(d)=2^{d}\) and coverage error \(\alpha=\nicefrac{{1}}{{20}}\).

Upper bound for \(v=\nicefrac{{4}}{{7}}\)The upper bound algorithm starts with the trivial upper bound of 1. The first evaluated point \(\rho_{1}=2^{-1}[2^{1}v]=1\) again yields the trivial bound of 1. There are still empirical counts between the probe value (\(v=\nicefrac{{4}}{{7}}\)) and the bound value (\(\rho_{1}=1\)) so the algorithm continues. The second evaluated point \(\rho_{2}=2^{-2}[2^{2}v]=\nicefrac{{3}}{{4}}\), for which there are 4000 empirical counts below \(\rho_{2}\) out of 5000 total. The pointwise confidence sequence is evaluated with counts \((4000,5000)\) and coverage error \(\delta_{2}=\nicefrac{{\alpha}}{{24}}\), resulting in improved bound \(\approx 0.825\). Now, there are no empirical counts between the probe value (\(v=\nicefrac{{4}}{{7}}\)) and the bound value (\(\rho_{2}=\nicefrac{{3}}{{4}}\)) so the algorithm terminates. To see all subsequent bounds are dominated, note that a tighter upper bound \(\rho_{d>2}\) will result in the same empirical counts (4000 out of 5000) but a looser coverage error and hence worse bound.

Upper bound for \(v=\nicefrac{{13}}{{28}}\)The upper bound algorithm starts with the trivial upper bound of 1. The first evaluated point \(\rho_{1}=2^{-1}[2^{1}v]=\nicefrac{{1}}{{2}}\), for which there are 4000 empirical counts below \(\rho_{2}\) out of 5000 total. The pointwise confidence sequence is evaluated with counts \((4000,5000)\) and coverage error \(\delta_{1}=\nicefrac{{\alpha}}{{4}}\), resulting in improved bound \(\approx 0.822\). There are no empirical counts between the probe value \((v=\nicefrac{{13}}{{28}})\) and the bound value (\(\rho_{1}=\nicefrac{{1}}{{2}}\)) so the algorithm terminates. Relative to the previous example, the bound is slightly tighter as the discretization worked better for this \(v\).

Upper bound for \(v=\nicefrac{{5}}{{14}}\)The upper bound algorithm starts with the trivial upper bound of 1. The first evaluated point \(\rho_{1}=2^{-1}[2^{1}v]=\nicefrac{{1}}{{2}}\), for which there are 4000 empirical counts below \(\rho_{2}\) out of 5000 total. The pointwise confidence sequence is evaluated with counts \((4000,5000)\) and coverage error \(\delta_{1}=\nicefrac{{\alpha}}{{4}}\), resulting in improved bound \(\approx 0.822\). There are empirical counts between the probe value \((v=\nicefrac{{5}}{{14}})\) and the bound value (\(\rho_{1}=\nicefrac{{1}}{{2}}\)) so the algorithm continues. The second evaluated point \(\rho_{2}=2^{-2}[2^{2}v]=\nicefrac{{1}}{{2}}\), which has the same empirical counts but worse coverage error at this level, and hence does not improve the bound. There are empirical counts between the probe value \((v=\nicefrac{{5}}{{14}})\) and the bound value (\(\rho_{2}=\nicefrac{{1}}{{2}}\)) so the algorithm continues. The third evaluated point \(\rho_{3}=2^{-3}[2^{3}v]=\nicefrac{{3}}{{8}}\), for which there are 3000 empirical counts below \(\rho_{3}\) out of 5000 total. The pointwise confidence sequence is evaluated with counts \((3000,5000)\) and coverage error \(\delta_{3}=\nicefrac{{\alpha}}{{96}}\), resulting in improved bound \(\approx 0.633\). There are no empirical counts between the probe value \((v=\nicefrac{{5}}{{14}})\) and the bound value (\(\rho_{3}=\nicefrac{{3}}{{8}}\)) so the algorithm terminates.

### Lower Bound

Algorithm 2 is extremely similar to Algorithm 1: the differences are indicated in comments. Careful inspection reveals the output of Algorithm 1, \(U_{t}(v)\), can be obtained from the output of Algorithm 2, \(L_{t}(v)\), via \(U_{t}(v)=1-L_{t}(1-v)\); but only if the sufficient statistics are adjusted such that \(\Xi_{t}(\rho_{d};\delta,\Psi_{t})=1-\Lambda_{t}(1-\rho_{d};\delta,\Psi_{t}^{ \prime})\). The reference implementation uses this strategy.

### Proof of Theorem 3.1

We prove the results for the upper bound Algorithm 1; the argument for the lower bound Algorithm 2 is similar.

The algorithm terminates when we find a \(d\) such that \(0=\sum_{s\leqslant t}1_{X_{s}\in(v,\rho_{d}]}\). Since \(\epsilon(d)\uparrow\infty\) as \(d\uparrow\infty\), we have \(\rho_{d}=\epsilon(d)[\epsilon(d)^{-1}v]\downarrow v\), so that \(\sum_{s\leqslant t}1_{X_{s}\in(v,\rho_{d}]}\downarrow 0\). So the algorithm must terminate.

At level \(d\), we have \(\epsilon(d)\) confidence sequences. The \(i^{\text{th}}\) confidence sequence at level \(d\) satisfies

\[P(\exists t:\overline{\text{CDF}}_{t}(i/\epsilon(d))>\Xi_{t}(i/\epsilon(d); \delta_{d},d,\Psi_{t}))\leqslant\frac{\alpha}{2^{d}\epsilon(d)}. \tag{12}\]

Taking a union bound over all confidence sequences at all levels, we have

\[P\left(\exists d\in\mathbb{N},i\in\{1,\ldots,d\},t\in\mathbb{N}:\overline{ \text{CDF}}_{t}(i/\epsilon(d))>\Xi_{t}(i/\epsilon(d);\delta,d,\Psi_{t})\right) \leqslant\alpha. \tag{13}\]

Thus we are assured that, for any \(v\in\mathbb{R}\),

\[P(\forall t,d:\overline{\text{CDF}}_{t}(v)\leqslant\overline{\text{CDF}}_{t}( \rho_{d})\leqslant\Xi_{t}(\rho_{d};\delta_{d},d,\Psi_{t}))\geqslant 1-\alpha. \tag{14}\]

Algorithm 1 will return \(\Xi_{t}(\rho_{d};\delta_{d},d,\Psi_{t})\) for some \(d\) unless all such values are larger than one, in which case it returns the trivial upper bound of one. This proves the upper-bound half of guarantee (2). A similar argument proves the lower-bound half, and union bound over the upper and lower bounds finishes the argument.

```
Input: value \(v\); confidence \(\alpha\); sufficient statistic \(\Psi_{t}\). // comments below indicate differences from upper bound //\(\Psi_{t}\doteq X_{1:t}\) or \(\Psi_{t}\doteq(W_{1:t},X_{1:t})\) Output:\(L_{t}(v)\) satisfying Equation (2). if\(v<0\)thenreturn\(0\)endif// check for underflow of range rather than overflow \(l\gets 0\)// initialize with \(0\) instead of \(1\) \(v\leftarrow\min\left(1,v\right)\)// project onto \([0,1]\) using \(\min\) instead of \(\max\) for\(d=1\)to\(\infty\)do \(\rho_{d}\leftarrow\epsilon(d)^{-1}[\epsilon(d)v]\)// use floor instead of ceiling \(\delta_{d}\leftarrow\nicefrac{{\alpha}}{{2^{d}\epsilon(d)}}\) \(l\leftarrow\max\left(l,\Lambda_{t}\left(\rho_{d};\delta,\Psi_{t}\right)\right)\)// use lower bound instead of upper bound if\(0=\sum_{s\leqslant t}1_{X_{s}\in[\rho_{d},v)}\)thenreturn\(l\) endif endfor
```

**Algorithm 2** Unit Interval Lower Bound. \(\epsilon(d)\) is an increasing function specifying the resolution of discretization at level \(d\). \(\Lambda_{t}\left(\rho;\delta,d,\Psi_{t}\right)\) is a lower confidence sequence for fixed value \(\rho\) with coverage at least \((1-\delta)\).

## Appendix C Proof of Theorem 3.3

**Theorem 3.3**.: _Let \(U_{t}(v)\) and \(L_{t}(v)\) be the upper and lower bounds returned by Algorithm 1 and Algorithm 2 respectively, when evaluated with \(\epsilon(d)=2^{d}\) and the confidence sequences \(\Lambda_{t}\) and \(\Xi_{t}\) of Equation (15). If \(\forall t:\mathbb{P}_{t}\) is \(\xi_{t}\)-smooth wrt the uniform distribution on the unit interval then_

\[\forall t,\forall v:U_{t}(v)-L_{t}(v)\leqslant \tag{6}\] \[\sqrt{\frac{V_{t}}{t}}+\tilde{O}\left(\sqrt{\frac{V_{t}}{t}\log \left(\xi_{t}^{-2}\alpha^{-1}t^{3/2}\right)}\right),\]

_where \(q_{t}\doteq\overline{\text{CDF}}_{t}(v)\); \(V_{t}\doteq\nicefrac{{1}}{{t}}+\nicefrac{{(q_{t}-1/2)}}{{\log(\nicefrac{{q_{t }}}{{1-q_{t}}})}}\); and \(\tilde{O}()\) elides polylog \(V_{t}\) factors._

Note \(v\) is fixed for the entire argument below, and \(\xi_{t}\) denotes the unknown smoothness parameter at time \(t\).

We will argue that the upper confidence radius \(U_{t}(v)-t^{-1}\sum_{s\leqslant t}1_{X_{s}\leqslant v}\) has the desired rate. An analogous argument applies to the lower confidence radius \(t^{-1}\sum_{s\leqslant t}1_{X_{s}\leqslant v}-L_{t}(v)\), and the confidence width \(U_{t}(v)-L_{t}(v)\) is the sum of these two.

For the proof we introduce an integer parameter \(\eta\geqslant 2\) which controls both the grid spacing (\(\epsilon(d)=\eta^{d}\)) and the allocation of error probabilities to levels (\(\delta_{d}=\alpha/(\eta^{d}\epsilon(d))\)). In the main paper we set \(\eta=2\).

At level \(d\) we construct \(\eta^{d}\) confidence sequences on an evenly-spaced grid of values \(1/\eta^{d},2/\eta^{d},\ldots,1\). We divide total error probability \(\alpha/\eta^{d}\) at level \(d\) among these \(\eta^{d}\) confidence sequences, so that each individual confidence sequence has error probability \(\alpha/\eta^{2d}\).

For a fixed bet \(\lambda\) and value \(\rho\), \(S_{t}\) defined in Section 3.2 is sub-Bernoulli qua Howard et al. (2021, Definition 1) and therefore sub-Gaussian with variance process \(V_{t}\doteq tK(q_{t})\), where \(K(p)\doteq\nicefrac{{(2p-1)}}{{2}\log(\nicefrac{{\eta}}{{1-\rho}})}\) is from Kearns and Saul (1998); from Howard et al. (2021, Proposition 5) it follows that there exists an explicit mixture distribution over \(\lambda\) such that

\[M(t;q_{t},\tau)\doteq\sqrt{2\left(tK(q_{t})+\tau\right)\log\left(\frac{\eta^{2 d}}{2\alpha}\sqrt{\frac{tK(q_{t})+\tau}{\tau}}+1\right)} \tag{15}\]

is a (curved) uniform crossing boundary, i.e., satisfies

\[\frac{\alpha}{\eta^{2d}}\geqslant\mathbb{P}\left(\exists t\geqslant 1:S_{t} \geqslant\frac{M(t;q_{t},\tau)}{t}\right),\]

where \(S_{t}\doteq\overline{\mathrm{CDF}}_{t}(\rho)-t^{-1}\sum_{s\leqslant t}1_{X_{s} \leqslant\rho}\) is from Equation (10), and \(\tau\) is a hyperparameter to be determined further below.

Because the values at level \(d\) are \(1/\eta^{d}\) apart, the worst-case discretization error in the estimated average CDF value is

\[\overline{\mathrm{CDF}}_{t}(\epsilon(d)[\epsilon(d)^{-1}v])-\overline{\mathrm{ CDF}}_{t}(v)\leqslant 1/(\xi_{t}\eta^{d}),\]

and the total worst-case confidence radius including discretization error is

\[r_{d}(t)=\frac{1}{\xi_{t}\eta^{d}}+\sqrt{\frac{2\left(K(q_{t})+\tau/t\right)}{ t}\log\left(\frac{\eta^{2d}}{2\alpha}\sqrt{\frac{tK(q_{t})+\tau}{\tau}}+1 \right)}.\]

Now evaluate at \(d\) such that \(\sqrt{\psi_{t}}<\xi_{t}\eta^{d}\leqslant\eta\sqrt{\psi_{t}}\) where \(\psi_{t}\doteq t\left(K(q_{t})+\tau/t\right)^{-1}\),

\[r_{d}(t)\leqslant\sqrt{\frac{K(q_{t})+\tau/t}{t}}+\sqrt{\frac{2\left(K(q_{t}) +\tau/t\right)}{t}\log\left(\frac{\xi_{t}^{-2}\eta^{2}}{2\alpha}\left(\frac{t }{K(q_{t})+\tau/t}\right)\sqrt{\frac{tK(q_{t})+\tau}{\tau}}+1\right)}.\]

The final result is not very sensitive to the choice of \(\tau\), and we use \(\tau=1\) in practice.

## Appendix D Extensions

### Arbitrary Support

Algorithm 3 is a variation on Algorithm 1 which does not assume a bounded range, and instead uses a countably discrete dense subset of the entire real line. Using the same argument of Theorem 3.3 with the modified probability from the modified union bound, we have

\[|k_{d}|-1<\eta^{-d}|v|\leqslant|k_{d}|,\] \[\xi_{t}/\sqrt{\psi_{t}}>\eta^{-d}\geqslant\eta^{-1}\xi_{t}/\sqrt {\psi_{t}}\] \[\implies 1+|k_{d}|<2+\xi_{t}|v|/\sqrt{\psi_{t}}\] \[\implies r_{d}(t)\leqslant\tilde{O}\left(\sqrt{\frac{V_{t}}{t}}\log \left(\left(2+\xi_{t}|v|t^{-1/2}\right)^{2}\xi_{t}^{-2}\alpha^{-1}t^{3/2} \right)\right),\]

demonstrating a logarithmic penalty in the probe value \(v\) (e.g., Figure 7).

**Algorithm 3** Entire Real Line Upper Bound. \(\epsilon(d)\) is an increasing function specifying the resolution of discretization at level \(d\). \(\Xi_{t}\left(\rho;\delta,d,\Psi_{t}\right)\) is an upper confidence sequence for fixed value \(\rho\) with coverage at least \((1-\delta)\).

**Input:** value \(v\); confidence \(\alpha\); sufficient statistic \(\Psi_{t}\).

//e.g. \(\Psi_{t}\doteq X_{1:t}\) or \(\Psi_{t}\doteq(W_{1:t},X_{1:t})\)

**Output:**\(U_{t}(v)\) satisfying Equation (2).

\(u\gets 1\)

**for**\(d=1\)**to**\(\infty\)**do**

\(k_{d}\leftarrow[\epsilon(d)^{-1}v]\) // Sub-optimal: see text for details

\(\rho_{d}\leftarrow\epsilon(d)k_{d}\)

\(\delta_{d}\leftarrow(\nicefrac{{\alpha}}{{2^{d}}})\left(\nicefrac{{3}}{{( \pi^{2}-3)(1+|k_{d}|)^{2}}}\right)\) // Union bound over \(d\in\mathbb{N}\) and \(k_{d}\in\mathbb{Z}\)

\(u\leftarrow\min\left(u,\Xi_{t}\left(\rho_{d};\delta_{d},d,\Psi_{t}\right)\right)\)

**if**\(0=\sum_{s\leqslant t}1_{X_{s}\epsilon(v,\rho_{d})}\)**then**

**return**\(u\)

**end for**

**Algorithm 4** Entire Real Line Upper Bound. \(\epsilon(d)\) is an increasing function specifying the resolution of discretization at level \(d\). \(\Xi_{t}\left(\rho;\delta,d,\Psi_{t}\right)\) is an upper confidence sequence for fixed value \(\rho\) with coverage at least \((1-\delta)\).

**Sub-optimality of \(k_{d}\)**The choice of \(k_{d}\) in Algorithm 3 is amenable to analysis, but unlike in Algorithm 1, it is not optimal. In Algorithm 1 the probability is allocated uniformly at each depth, and therefore the closest grid point provides the tightest estimate. However in Algorithm 3, the probability budget decreases with \(|k_{d}|\) and because \(k_{d}\) can be negative, it is possible that a different \(k_{d}\) can produce a tighter upper bound. Since every \(k_{d}\) is covered by the union bound, in principle we could optimize over all \(k_{d}\) but it is unclear how to do this efficiently. In our implementation we do not search over all \(k_{d}\), but we do adjust \(k_{d}\) to be closest to the origin with the same empirical counts.

### Discrete Jumps

Known Countably InfiniteSuppose \(D\) is smooth wrt a reference measure \(M\), where \(M\) is of the form

\[M=\tilde{M}+\sum_{i\in I}\zeta_{i}1_{v_{i}},\]

with \(I\) a countable index set, \(1\geqslant\sum_{i\in I}\zeta_{i}\) and \(\tilde{M}\) a sub-probability measure normalizing to \((1-\sum_{i\in I}\zeta_{i})\). Then we can allocate \((1-\sum_{i\in I}\zeta_{i})\) of our overall coverage probability to bounding \(\tilde{M}\) using Algorithm 1 and Algorithm 2. For the remaining \(\{v_{i}\}_{i\in I}\) we can run explicit pointwise bounds each with coverage probability fraction \(\zeta_{i}\).

Computationally, early termination of the infinite search over the discrete bounds is possible. Suppose (wlog) \(I\) indexes \(\zeta\) in non-increasing order, i.e., \(i\leqslant j\implies\zeta_{i}\leqslant\zeta_{j}\): then as soon as there are no remaining empirical counts between the desired value \(v\) and the most recent discrete value \(v_{i}\), the search over discrete bounds can terminate.

## Appendix E Importance-Weighted Variant

### Modified Bounds

Algorithm 1 and Algorithm 2 are unmodified, with the caveat that the oracles \(\Lambda_{t}\) and \(\Xi_{t}\) must now operate on an importance-weighted realization \((W_{1:t},X_{1:t})\), rather then directly on the realization \(X_{1:t}\).

#### e.1.1 DDRM Variant

For simplicity we describe the lower bound \(\Lambda_{t}\) only. The upper bound is derived analogously via the equality \(Y_{s}=W_{s}-(W_{s}-Y_{s})\) and a lower bound on \((W_{s}-Y_{s})\): see Waudby-Smith et al. (2022, Remark 3) for more details.

This is the Heavy NSM from Mineiro (2022) combined with the \(L\)* bound of Orabona (2019, SS4.2.3). The Heavy NSM allow us to handle importance weights with unbounded variance, while the Adagrad \(L\)* bound facilitates lazy evaluation.

For fixed \(v\), let \(Y_{t}=W_{t}1_{X_{s}\succeq v}\) be a non-negative real-valued discrete-time random process, let \(\hat{Y}_{t}\in[0,1]\) be a predictable sequence, and let \(\lambda\in[0,1)\) be a fixed scalar bet. Then

\[E_{t}(\lambda)\doteq\exp\left(\lambda\left(\sum_{s\leq t}\hat{Y}_{s}-\mathbb{E} _{s-1}\left[Y_{s}\right]\right)+\sum_{s\leq t}\log\left(1+\lambda\left(Y_{s}- \hat{Y}_{s}\right)\right)\right)\]

is a test supermartingale (Mineiro, 2022, SS3). Manipulating,

\[E_{t}(\lambda) =\exp\left(\lambda\left(\sum_{s\leq t}Y_{s}-\mathbb{E}_{s-1}\left[ Y_{s}\right]\right)-\sum_{s\leq t}\underbrace{\left(\lambda\left(Y_{s}-\hat{Y}_{s} \right)-\log\left(1+\lambda\left(Y_{s}-\hat{Y}_{s}\right)\right)\right)}_{ \triangleq h\left(\lambda\left(Y_{s}-\hat{Y}_{s}\right)\right)}\right)\] \[=\exp\left(\lambda\left(\sum_{s\leq t}Y_{s}-\mathbb{E}_{s-1}\left[ Y_{s}\right]\right)-\sum_{s\leq t}h\left(\lambda\left(Y_{s}-\hat{Y}_{s}\right) \right)\right)\] \[\geqslant\exp\left(\lambda\left(\sum_{s\leq t}Y_{s}-\mathbb{E}_{s -1}\left[Y_{s}\right]\right)-\left(\sum_{s\leq t}h\left(\lambda\left(Y_{s}- \hat{Y}_{t}^{*}\right)\right)\right)-\text{Reg}(t)\right)\] ( \[\dagger\] ) \[=\exp\left(\lambda\left(t\hat{Y}_{t}^{*}-\sum_{s\leq t}\mathbb{E} _{s-1}\left[Y_{s}\right]\right)+\sum_{s\leq t}\log\left(1+\lambda\left(Y_{s}- \hat{Y}_{t}^{*}\right)\right)-\text{Reg}(t)\right),\]

where for \((\dagger)\) we use a no-regret learner on \(h()\) with regret \(\text{Reg}(t)\) to any constant prediction \(\hat{Y}_{t}^{*}\in[0,1]\). The function \(h()\) is \(M\)-smooth with \(M=\frac{\lambda^{2}}{(1-\lambda)^{2}}\) so we can get an \(L^{*}\) bound (Orabona, 2019, SS4.2.3) of

\[\text{Reg}(t) =4\frac{\lambda^{2}}{(1-\lambda)^{2}}+4\frac{\lambda}{1-\lambda} \sqrt{\sum_{s\leq t}h\left(\lambda\left(Y_{s}-\hat{Y}_{t}^{*}\right)\right)}\] \[=4\frac{\lambda^{2}}{(1-\lambda)^{2}}+4\frac{\lambda}{1-\lambda} \sqrt{\left(-t\hat{Y}_{t}^{*}+\sum_{s\leq t}Y_{s}\right)-\sum_{s\leq t}\log \left(1+\lambda\left(Y_{s}-\hat{Y}_{t}^{*}\right)\right)},\]

thus essentially our variance process is inflated by a square-root. In exchange we do not have to actually run the no-regret algorithm, which eases the computational burden. We can compete with any in-hindsight prediction: if we choose to compete with the clipped running mean \(\overline{Y_{t}}\) then we end up with

\[E_{t}(\lambda)\geqslant\exp\left(\lambda\left(\min\left(t,\sum_{s\leq t}Y_{s} \right)-\mathbb{E}_{s-1}\left[Y_{s}\right]\right)+\sum_{s\leq t}\log\left(1+ \lambda\left(Y_{s}-\overline{Y_{t}}\right)\right)-\text{Reg}(t)\right), \tag{16}\]

which is implemented in the reference implementation as LogApprox:getLowerBoundWithRegret(lam). The \(\lambda\)-s are mixed using DDRM from Mineiro (2022, Thm. 4), implemented via the DDRM class and the getDDRMCSLowerBound method in the reference implementation. getDDRMCSLowerBound provably correctly early terminates the infinite sum by leveraging

\[\sum_{s\leq t}\log\left(1+\lambda\left(Y_{s}-\overline{Y_{t}}\right)\right) \leqslant\lambda\left(\sum_{s\leq t}Y_{s}-t\overline{Y_{t}}\right)\]

as seen in the termination criterion of the inner method logwealth(mu).

To minimize computational overhead, we can lower bound \(\log(a+b)\) for \(b\geqslant 0\) using strong concavity qua Mineiro (2022, Thm. 3), resulting in the following geometrically spaced collection of sufficient statistics:

\[(1+k)^{n_{t}}=z_{l}\leqslant z<z_{u}=(1+k)z_{l}=(1+k)^{n_{l}+1},\]

along with distinct statistics for \(z=0\). \(k\) is a hyperparameter controlling the granularity of the discretization (tighter lower bound vs. more space overhead): we use \(k=1/4\) exclusively in our experiments. Note the coverage guarantee is preserved for any choice of \(k\) since we are lower bounding the wealth.

Given these statistics, the wealth can be lower bounded given any bet \(\lambda\) and any in-hindsight prediction \(\hat{Y}_{t}^{*}\) via

\[f(z) \doteq\log\left(1+\lambda\left(z-\hat{Y}_{t}^{*}\right)\right),\] \[f(z) \geqslant\alpha f(z_{l})+(1-\alpha)f(z_{u})+\frac{1}{2}\alpha(1- \alpha)m(z_{l}),\] \[\alpha \doteq\frac{z_{u}-z}{z_{u}-z_{l}},\] \[m(z_{l}) \doteq\left(\frac{kz_{l}\lambda}{kz_{l}\lambda+1-\lambda\hat{Y}_{t }^{*}}\right)^{2}.\]

Thus when accumulating the statistics, for each \(Y_{s}=W_{s}1_{X_{s}\geqslant v}\), a value of \(\alpha\) must be accumulated at key \(f(z_{l})\), a value of \((1-\alpha)\) accumulated at key \(f(z_{u})\), and a value of \(\alpha(1-\alpha)\) accumulated at key \(m(z_{l})\). The LogApprox::update method from the reference implementation implements this.

Because these sufficient statistics are data linear, a further computational trick is to accumulate the sufficient statistics with equality only, i.e., for \(Y_{s}=W_{s}1_{X_{s}=v}\); and when the CDF curve is desired, combine these point statistics into cumulative statistics. In this manner only \(O(1)\) incremental work is done per datapoint; while an additional \(O(t\log(t))\) work is done to accumulate all the sufficient statistics only when the bounds need be computed. The method StreamingDDRMECDF::Frozen::__init__from the reference implementation contains this logic.

#### e.1.2 Empirical Bernstein Variant

For simplicity we describe the lower bound \(\Lambda_{t}\) only. The upper bound is derived analogously via the equality \(Y_{s}=W_{s}-(W_{s}-Y_{s})\) and a lower bound on \((W_{s}-Y_{s})\): see Waudby-Smith et al. (2022, Remark 3) for more details.

This is the empirical Bernstein NSM from Howard et al. (2021) combined with the \(L\)* bound of Orabona (2019, SS4.2.3). Relative to DDRM it is faster to compute, has a more concise sufficient statistic, and is easier to analyze; but it is wider empirically, and theoretically requires finite importance weight variance to converge.

For fixed \(v\), let \(Y_{t}=W_{t}1_{X_{t}\geqslant v}\) be a non-negative real-valued discrete-time random process, let \(\hat{Y}_{t}\in[0,1]\) be a predictable sequence, and let \(\lambda\in[0,1)\) be a fixed scalar bet. Then

\[E_{t}(\lambda)\doteq\exp\left(\lambda\left(\sum_{s\in t}\hat{Y}_{s}-\mathbb{E} _{s-1}\left[Y_{s}\right]\right)+\sum_{s\in t}\log\left(1+\lambda\left(Y_{s}- \hat{Y}_{s}\right)\right)\right)\]

is a test supermartingale (Mineiro, 2022, SS3). Manipulating,

\[E_{t}(\lambda) \doteq\exp\left(\lambda\left(\sum_{s\in t}Y_{s}-\mathbb{E}_{s-1} \left[Y_{s}\right]\right)-\sum_{s\in t}\underbrace{\left(\lambda\left(Y_{s}- \hat{Y}_{s}\right)-\log\left(1+\lambda\left(Y_{s}-\hat{Y}_{s}\right)\right) \right)}_{\triangleq h\left(\lambda\left(Y_{s}-\hat{Y}_{s}\right)\right)}\right)\] \[\geqslant\exp\left(\lambda\left(\sum_{s\in t}Y_{s}-\mathbb{E}_{s- 1}\left[Y_{s}\right]\right)-h(-\lambda)\sum_{s\in t}\left(Y_{s}-\hat{Y}_{s} \right)^{2}\right)\] [Fan, Lemma 4.1] \[\geqslant\exp\left(\lambda\left(\sum_{s\in t}Y_{s}-\mathbb{E}_{s- 1}\left[Y_{s}\right]\right)-h(-\lambda)\left(\text{Reg}(t)+\sum_{s\in t}\left( Y_{s}-Y_{t}^{*}\right)^{2}\right)\right)\] ( \[\dagger\] ) \[\doteq\exp\left(\lambda S_{t}-h(-\lambda)V_{t}\right),\]

where \(S_{t}=\sum_{s\in t}Y_{s}-\mathbb{E}_{s-1}\left[Y_{s}\right]\) and for (\(\dagger\)) we use a no-regret learner on squared loss on feasible set \([0,1]\) with regret \(\text{Reg}(t)\) to any constant in-hindsight prediction \(\hat{Y}_{t}^{*}\in[0,1]\). Since \(Y_{s}\) is unbounded above, the loss is not Lipschitz and we can't get fast rates for squared loss, but we can run Adagrad and get an \(L^{*}\) bound,

\[\text{Reg}(t) =2\sqrt{2}\sqrt{\sum_{s\leq t}g_{s}^{2}}\] \[=4\sqrt{2}\sqrt{\sum_{s\leq t}(Y_{s}-\hat{Y}_{s})^{2}}\] \[\leq 4\sqrt{2}\sqrt{\text{Reg}(t)+\sum_{s\leq t}(Y_{s}-\hat{Y}_{t }^{*})^{2}},\] \[\implies\text{Reg}(t) \leq 16+4\sqrt{2}\sqrt{8+\sum_{s\leq t}(Y_{s}-\hat{Y}_{t}^{*})^{2}}.\]

Thus basically our variance process is inflated by an additive square root.

We will compete with \(Y_{t}^{*}=\min\left(1,\frac{1}{t}\sum_{s}Y_{s}\right)\).

A key advantage of the empirical Bernstein over DDRM is the availability of both a conjugate (closed-form) mixture over \(\lambda\) and a closed-form majorized stitched boundary. This yields both computational speedup and analytical tractability.

For a conjugate mixture, we use the truncated gamma prior from Waudby-Smith et al. (2022, Theorem 2) which yields mixture wealth

\[M_{t}^{\text{EB}}\doteq\left(\frac{\tau^{\tau}e^{-\tau}}{\Gamma(\tau)-\Gamma( \tau,\tau)}\right)\left(\frac{1}{\tau+V_{t}}\right){}_{1}F_{1}\left(1,V_{t}+ \tau+1,S_{t}+V_{t}+\tau\right), \tag{17}\]

where \({}_{1}F_{1}(\ldots)\) is Kummer's confluent hypergeometric function and \(\Gamma(\cdot,\cdot)\) is the upper incomplete gamma function. For the hyperparameter, we use \(\tau=1\).

### Proof of Theorem 3.4

**Theorem 3.4**.: _Let \(U_{t}(v)\) and \(L_{t}(v)\) be the upper and lower bounds returned by Algorithm 1 and Algorithm 2 respectively with \(\epsilon(d)=2^{d}\) and the confidence sequences \(\Lambda_{t}\) and \(\Xi_{t}\) of Equation (18). If \(\forall t:\mathbb{P}_{t}\) is \(\xi_{t}\)-smooth wrt the uniform distribution on the unit interval then_

\[\begin{split}\forall t,\forall v&:U_{t}(v)-L_{t}(v) \leqslant\\ B_{t}&+\sqrt{\frac{(\tau+V_{t})/t}{t}}\\ &+\tilde{O}\left(\sqrt{\frac{(\tau+V_{t})/t}{t}}\log\left(\xi_{t} ^{-2}\alpha^{-1}\right)\right)\\ &+\tilde{O}(t^{-1}\log\left(\xi_{t}^{-2}\alpha^{-1}\right)),\end{split} \tag{7}\]

_where \(q_{t}\doteq\overline{\text{CDF}}_{t}(v)\), \(K(q_{t})\doteq(q_{t}-1/2)/_{\log(q_{t}/1-q_{t})}\); \(V_{t}=O\left(K(q_{t})\sum_{s\leq t}W_{s}^{2}\right)\), \(B_{t}\doteq t^{-1}\sum_{s\leq t}(W_{s}-1)\), and \(\tilde{O}()\) elides polylog \(V_{t}\) factors._

Note \(v\) is fixed for the entire argument below, and \(\xi_{t}\) denotes the unknown smoothness parameter at time \(t\).

We will argue that the upper confidence radius \(U_{t}(v)-t^{-1}\sum_{s\leq t}W_{s}1_{X_{s}\leq v}\) has the desired rate. An analogous argument applies to the lower confidence radius. One difference from the non-importance-weighted case is that, to be sub-exponential, the lower bound is constructed from an upper bound on \(U_{t}^{\prime}(v)=W_{s}(1-1_{X_{s}\leq v})\) via \(L_{t}(v)-1-U_{t}^{\prime}(v)\), which introduces an additional \(B_{t}=t^{-1}\sum_{s\leq t}(W_{s}-1)\) term to the width. (Note, because \(\forall t:\mathbb{E}_{t}[W_{t}-1]=0\), this term will concentrate, but we will simply use the realized value here.)

For the proof we introduce an integer parameter \(\eta\geqslant 2\) which controls both the grid spacing (\(\epsilon(d)=\eta^{d}\)) and the allocation of error probabilities to levels (\(\delta_{d}=\alpha/(\eta^{d}\epsilon(d))\)). In the main paper we set \(\eta=2\).

At level \(d\) we construct \(\eta^{d}\) confidence sequences on an evenly-spaced grid of values \(1/\eta^{d},2/\eta^{d},\ldots,1\). We divide total error probability \(\alpha/\eta^{d}\) at level \(d\) among these \(\eta^{d}\) confidence sequences, so that each individual confidence sequence has error probability \(\alpha/\eta^{2d}\).

For a fixed bet \(\lambda\) and value \(\rho\), \(S_{t}\) defined in Appendix E.1.2 is sub-exponential qua Howard et al. (2021, Definition 1) and therefore from Lemma E.1 there exists an explicit mixture distribution over \(\lambda\) inducing (curved) boundary

\[\frac{\alpha}{\eta^{2d}} \geq\mathbb{P}\left(\exists t\geq 1:\frac{S_{t}}{t}\geq\max \left(\frac{C(\tau)}{t},u\left(V_{t};\tau,\frac{\alpha}{\eta^{2d}}\right) \right)\right),\] \[u\left(V_{t};\tau,\frac{\alpha}{\eta^{2d}}\right) =\sqrt{2\left(\frac{(\tau+V_{t})/t}{t}\right)\log\left(\sqrt{\frac {\tau+V_{t}}{2\pi}}e^{-\frac{1}{12(\tau+V_{t})+1}}\left(\frac{1+\eta^{2d} \alpha^{-1}}{C(\tau)}\right)\right)}\] \[\qquad+\frac{1}{t}\log\left(\sqrt{\frac{\tau+V_{t}}{2\pi}}e^{- \frac{1}{12(\tau+V_{t})+1}}\left(\frac{1+\eta^{2d}\alpha^{-1}}{C(\tau)}\right) \right), \tag{18}\]

where \(S_{t}\doteq\overline{\mathrm{CDF}}_{t}(\rho)-t^{-1}\sum_{s\leq t}W_{s}1_{X_{s} \leq\rho}\), and \(\tau\) is a hyperparameter to be determined further below.

Because the values at level \(d\) are \(1/\eta^{d}\) apart, the worst-case discretization error in the estimated average CDF value is

\[\overline{\mathrm{CDF}}_{t}(\epsilon(d)[\epsilon(d)^{-1}v])-\overline{\mathrm{ CDF}}_{t}(v)\leq 1/(\xi_{t}\eta^{d}),\]

and the total worst-case confidence radius including discretization error is

\[r_{d}(t)=\frac{1}{\xi_{t}\eta^{d}}+\max\left(\frac{C(\tau)}{t},u\left(V_{t}; \tau,\frac{\alpha}{\eta^{2d}}\right)\right).\]

Now evaluate at \(d\) such that \(\sqrt{\psi_{t}}<\xi_{t}\eta^{d}\leq\eta\sqrt{\psi_{t}}\) where \(\psi_{t}\doteq t\left((\tau+V_{t})/t\right)^{-1}\),

\[r_{d}(t) \leq\frac{1}{\sqrt{\psi_{t}}}+\max\left(\frac{C(\tau)}{t},u\left( V_{t};\tau,\frac{\alpha}{\eta^{2}\xi_{t}^{-2}\psi_{t}}\right)\right)\] \[=\sqrt{\frac{(\tau+V_{t})/t}{t}}+\tilde{O}\left(\sqrt{\frac{( \tau+V_{t})/t}{t}\log\left(\xi_{t}^{-2}\alpha^{-1}\right)}\right)+\tilde{O}(t^ {-1}\log\left(\xi_{t}^{-2}\alpha^{-1}\right)),\]

where \(\tilde{O}()\) elides polylog \(V_{t}\) factors. The final result is not very sensitive to the choice of \(\tau\), and we use \(\tau=1\) in practice.

**Lemma E.1**.: _Suppose_

\[\exp\left(\lambda S_{t}-\psi_{e}(\lambda)V_{t}\right),\] \[\psi_{e}(\lambda)\doteq-\lambda-\log(1-\lambda),\]

_is sub-\(\psi_{e}\) qua Howard et al. (2021, Definition 1); then there exists an explicit mixture distribution over \(\lambda\) with hyperparameter \(\tau>0\) such that_

\[\alpha \geq\mathbb{P}\left(\exists t\geq 1:\frac{S_{t}}{t}\geq\max \left(\frac{C(\tau)}{t},u\left(V_{t};\tau,\alpha\right)\right)\right),\] \[u\left(V_{t};\tau,\alpha\right) =\sqrt{2\left(\frac{(\tau+V_{t})/t}{t}\right)\log\left(\sqrt{ \frac{\tau+V_{t}}{2\pi}}e^{-\frac{1}{12(\tau+V_{t})+1}}\left(\frac{1+\alpha^ {-1}}{C(\tau)}\right)\right)}\] \[\qquad+\frac{1}{t}\log\left(\sqrt{\frac{\tau+V_{t}}{2\pi}}e^{- \frac{1}{12(\tau+V_{t})+1}}\left(\frac{1+\alpha^{-1}}{C(\tau)}\right)\right),\] \[C(\tau) \doteq\frac{\tau^{\tau}e^{-\tau}}{\Gamma(\tau)-\Gamma(\tau,\tau)},\]

_is a (curved) uniform crossing boundary._Proof.: We can form the conjugate mixture using a truncated gamma prior from Howard et al. (2021, Proposition 9), in the form from Waudby-Smith et al. (2022, Theorem 2), which is our Equation (17).

\[M_{t}^{\text{EB}}\doteq\left(\frac{\tau^{\tau}e^{-\tau}}{\Gamma(\tau)-\Gamma( \tau,\tau)}\right)\left(\frac{1}{\tau+V_{t}}\right){}_{1}F_{1}\left(1,V_{t}+ \tau+1,S_{t}+V_{t}+\tau\right),\]

where \({}_{1}F_{1}(\ldots)\) is Kummer's confluent hypergeometric function. Using Olver et al. (2010, identity 13.6.5),

\[{}_{1}F_{1}(1,a+1,x)=e^{x}ax^{-a}\left(\Gamma(a)-\Gamma(a,x)\right)\]

where \(\Gamma(a,x)\) is the (unregularized) upper incomplete gamma function. From Pinelis (2020, Theorem 1.2) we have

\[\Gamma(a,x) <\frac{x^{a}e^{-x}}{x-a}\] \[\implies{}_{1}F_{1}(1,a+1,x) \geqslant e^{x}ax^{-a}\Gamma(a)-\frac{a}{x-a}.\]

Applying this to the mixture yields

\[M_{t}^{\text{EB}} \geqslant\frac{C(\tau)e^{\tau+V_{t}+S_{t}}}{(\tau+V_{t}+S_{t})^{ \tau+V_{t}}}\Gamma\left(\tau+V_{t}\right)-\frac{C(\tau)}{S_{t}}\] \[\geqslant\frac{C(\tau)e^{\tau+V_{t}+S_{t}}}{(\tau+V_{t}+S_{t})^{ \tau+V_{t}}}\Gamma\left(\tau+V_{t}\right)-1, (\dagger)\]

where \((\dagger)\) follows from the self-imposed constraint \(S_{t}\geqslant C(\tau)\). This yields crossing boundary

\[\alpha^{-1} =\frac{C(\tau)e^{\tau+V_{t}+S_{t}}}{(\tau+V_{t}+S_{t})^{\tau+V_{t} }}\Gamma\left(\tau+V_{t}\right)-1,\] \[\frac{e^{\tau+V_{t}+S_{t}}}{\left(1+\frac{S_{t}}{\tau+V_{t}} \right)^{\tau+V_{t}}} =\left(\frac{(\tau+V_{t})^{\tau+V_{t}}}{\Gamma\left(\tau+V_{t} \right)}\right)\left(\frac{1+\alpha^{-1}}{C(\tau)}\right)\doteq\left(\frac{( \tau+V_{t})^{\tau+V_{t}}}{\Gamma\left(\tau+V_{t}\right)}\right)\phi_{t}(\tau, \alpha),\] \[\frac{e^{1+\frac{S_{t}}{\tau+V_{t}}}}{\left(1+\frac{S_{t}}{\tau+ V_{t}}\right)} =\left(\frac{(\tau+V_{t})^{\tau+V_{t}}}{\Gamma\left(\tau+V_{t} \right)}\right)^{\frac{1}{\tau+V_{t}}}\phi_{t}(\tau,\alpha)^{\frac{1}{\tau+V_ {t}}}\doteq z_{t},\] \[S_{t} =(\tau+V_{t})\left(-1-W_{-1}\left(-z_{t}^{-1}\right)\right).\]

Chatzgeorgiou (2013, Theorem 1) states

\[W_{-1}(-e^{-u-1}) \in-1-\sqrt{2u}+\left[-u,-\frac{2}{3}u\right]\] \[\implies-1-W_{-1}(-e^{-u-1}) \in\sqrt{2u}+\left[\frac{2}{3}u,u\right].\]

Substituting yields

\[(\tau+V_{t})\left(-1-W_{-1}\left(-z_{t}^{-1}\right)\right)\leqslant(\tau+V_{t })\left(\sqrt{2\log\left(\frac{z_{t}}{e^{1}}\right)}+\log\left(\frac{z_{t}}{ e^{1}}\right)\right). \tag{19}\]

From Feller (1958, Equation (9.8)) we have

\[\Gamma(1+n) \in\sqrt{2\pi n}\left(\frac{n}{e^{1}}\right)^{n}\left[e^{\frac{1} {2n+1}},e^{\frac{1}{2n}}\right]\] \[\implies\left(\frac{(\tau+V_{t})^{\tau+V_{t}}}{\Gamma\left(\tau+ V_{t}\right)}\right)^{\frac{1}{\tau+V_{t}}} \in\left(\frac{\tau+V_{t}}{2\pi}\right)^{\frac{1}{2(\tau+V_{t})}}e^{1}\left[e^ {-\frac{1}{12(\tau+V_{t})^{2}}},e^{-\frac{1}{12(\tau+V_{t})^{2}+(\tau+V_{t})} }\right].\]

Therefore

\[(\tau+V_{t})\sqrt{2\log\left(\frac{z_{t}}{e^{1}}\right)} \leqslant(\tau+V_{t})\sqrt{2\log\left(\left(\frac{\tau+V_{t}}{2 \pi}\right)^{\frac{1}{2(\tau+V_{t})}}e^{-\frac{1}{12(\tau+V_{t})^{2}+(\tau+V_{ t})}}\phi_{t}(\tau,\alpha)^{\frac{1}{\tau+V_{t}}}\right)}\] \[=\sqrt{2\left(\tau+V_{t}\right)\log\left(\sqrt{\frac{\tau+V_{t}} {2\pi}}e^{-\frac{1}{12(\tau+V_{t})+\tau}}\phi_{t}(\tau,\alpha)\right)}, \tag{20}\]\[(\tau+V_{t})\log\left(\frac{z_{t}}{e^{1}}\right) \leqslant(\tau+V_{t})\log\left(\left(\frac{\tau+V_{t}}{2\pi}\right) ^{\frac{1}{2(\tau+V_{t})}}e^{-\frac{1}{12(\tau+V_{t})^{2}+(\tau+V_{t})}}\phi_{ t}(\tau,\alpha)^{\frac{1}{\tau+V_{t}}}\right)\] \[=\log\left(\sqrt{\frac{\tau+V_{t}}{2\pi}}e^{-\frac{1}{12(\tau+V_{t })+1}}\phi_{t}(\tau,\alpha)\right). \tag{21}\]

Combining Equations (19) to (21) yields the crossing boundary

\[\frac{S_{t}}{t} =\sqrt{2\left(\frac{(\tau+V_{t})/t}{t}\right)\log\left(\sqrt{ \frac{\tau+V_{t}}{2\pi}}e^{-\frac{1}{12(\tau+V_{t})+1}}\left(\frac{1+\alpha^{ -1}}{C(\tau)}\right)\right)}\] \[\qquad+\frac{1}{t}\log\left(\sqrt{\frac{\tau+V_{t}}{2\pi}}e^{- \frac{1}{12(\tau+V_{t})+1}}\left(\frac{1+\alpha^{-1}}{C(\tau)}\right)\right).\]

## Appendix F Simulations

### i.i.d. setting

For non-importance-weighted simulations, we use the Beta-Binomial boundary of Howard et al. (2021) for \(\Lambda_{t}\) and \(\Xi_{t}\). The curved boundary is induced by the test NSM

\[W_{t}(b;\hat{q}_{t},q_{t}) =\frac{\int_{q_{t}}^{1}d\text{Beta}\left(p;bq_{t},b(1-q_{t})\right) \,\left(\frac{p}{q_{t}}\right)^{t\hat{q}_{t}}\left(\frac{1-p}{1-q_{t}}\right)^ {t(1-\hat{q}_{t})}}{\int_{q_{t}}^{1}d\text{Beta}\left(p;bq_{t},b(1-q_{t})\right)}\] \[=\frac{1}{(1-q_{t})^{t(1-\hat{q}_{t})}q_{t}^{t\hat{q}_{t}}}\left( \frac{\text{Beta}(q_{t},1,bq_{t}+t\hat{q}_{t},b(1-q_{t})+t(1-\hat{q}_{t}))}{ \text{Beta}(q_{t},1,bq_{t},b(1-q_{t}))}\right)\]

with prior parameter \(b=1\). Further documentation and details are in the reference implementation csnsquantile.ipynb.

The importance-weighted simulations use the constructions from Appendix E: the reference implementation is in csnsopquantile.ipynb for the DDRM variant and csnsopquantile-ebern.ipynb for the empirical Bernstein variant.

Figure 11: Demonstration of the variant described in Section 3.3 and Appendix D.1 for distributions with arbitrary support, based on i.i.d. sampling from a variety of Gaussian distributions. Logarithmic range dependence is evident.

Figure 12: Maximum bound width, scaled by \(\sqrt{t/\log(t)}\) to remove the primary trend, as a function of \(t\), for nonstationary Polya simulations with different \(\gamma_{t}\) schedules. See Section 4.2

Figure 8: Comparison to naive time-uniform DKW (which is only valid in the i.i.d. setting) for Beta distributions of varying smoothness. Decreasing smoothness degrades our bound.

Figure 7: Demonstration of the variant described in Section 3.3 and Appendix D.1 for distributions with arbitrary support, based on i.i.d. sampling from a variety of lognormal distributions. Logarithmic range dependence is evident.

Figure 9: CDF bounds approaching the true CDF when sampling i.i.d. from a lognormal(0, 1) distribution. Recall these bounds are simultaneously valid for all times and values.

Figure 8: Comparison to naive time-uniform DKW (which is only valid in the i.i.d. setting) for Beta distributions of varying smoothness. Decreasing smoothness degrades our bound.

Figure 10: CDF bounds approaching the true CDF when sampling i.i.d. from a Gaussian(0, 1) distribution. Recall these bounds are simultaneously valid for all times and values.

Figure 7: Demonstration of the variant described in Section 3.3 and Appendix D.1 for distributions with arbitrary support, based on i.i.d. sampling from a variety of lognormal distributions. Logarithmic range dependence is evident.

Figure 14: CDF bounds\({}^{\prime}\) approaching the true counterfactual CDF when sampling i.i.d. from a Beta(6,3) with finite-variance importance weights, using Empirical Bernstein for the oracle confidence sequence.

Figure 13: CDF bounds’ approaching the true counterfactual CDF when sampling i.i.d. from a Beta(6,3) with finite-variance importance weights, using DDRM for the oracle confidence sequence.

Figure 15: CDF bounds’ approaching the true counterfactual CDF when sampling i.i.d. from a Beta(6,3) with infinite-variance importance weights, using Empirical Bernstein for the oracle confidence sequence. Despite apparent convergence, eventually this simulation would reset the Empirical Bernstein oracle confidence sequence to trivial bounds.

Response:
## Review

### Summary

This paper proposes a method for constructing time-uniform and value-uniform bounds on the cumulative distribution function (CDF) of the averaged historical distribution of a real-valued random process, particularly under nonstationarity and dependence among variables. Building on the work of Howard and Ramdas (2022), the authors derive uniform bounds that do not require the assumption of independent and identically distributed (i.i.d.) samples. The paper includes theoretical guarantees for the proposed methods and simulations demonstrating their effectiveness in various scenarios. However, it requires clearer explanations and definitions to enhance accessibility for readers not deeply familiar with the material.

### Strengths

- Reliable bounds on the CDF for dependent data settings, which is valuable as iid assumptions do not always hold.
- The considered problem of constructing confidence bands for CDFs under nonstationarity is of great practical importance.
- The proposed technique appears new and interesting, with theoretical guarantees provided in Theorem 3.3.
- The authors simulate their algorithm on several synthetic examples, showcasing its adaptive performance.

### Weaknesses

- The paper is hard to read and follow, assuming prior knowledge from the reader, thus requiring more detailed explanations.
- Key concepts, such as confidence sequences and their role in algorithms, are insufficiently explained.
- The introduction lacks clarity regarding the problem being solved and could benefit from including more background information.
- Comparisons with existing results need to be clearer, particularly regarding technical difficulties and contributions.
- Some figures are poorly labeled or too small, impacting readability.

### Questions

- What is the instance-adaptivity of Theorem 3.3, and how does it maintain time-uniformity without dependence on previous parameters?
- Is the i.i.d. assumption made explicit throughout Section 3?
- Can the authors clarify the roles of certain inputs in Algorithm 1 and provide concrete examples to aid comprehension?
- What are the major technical difficulties addressed in deriving the main results, and could the authors summarize these more explicitly?

### Soundness

**Score:** 2

**Description:** Fair: The methodology is generally sound, but there are concerns regarding the clarity of certain theoretical aspects and proofs.

### Presentation

**Score:** 2

**Description:** Fair: The writing is often difficult to follow, with numerous instances of insufficient explanations and definitions, though some parts are well-structured.

### Contribution

**Score:** 3

**Description:** Good: The paper introduces noteworthy contributions to the field, especially concerning confidence bands for dependent variables.

### Rating

**Score:** 5

**Description:** Borderline accept: The paper is technically solid with significant contributions but requires improvements in clarity and presentation.

### Paper Decision

**Decision:** Accept (poster)

**Reasons:** The paper presents valuable contributions to the understanding and methodology of constructing confidence bands for nonstationary processes. Despite the concerns regarding presentation and clarity, the technical merits and originality of the work warrant acceptance, provided that the authors address the feedback related to exposition and detail in future revisions.