$$\rho\_{\mathbf{m}}(X;Y) := \sup\_{f,g} \mathbb{E}[f(X)g(Y)],\tag{117}$$

*where the supremum is taken over all real-valued functions f and g such that*

$$\mathbb{E}[f(X)] = \mathbb{E}[\mathbf{g}(Y)] = 0, \quad \mathbb{E}[f^2(X)] \le 1,\\ \mathbb{E}[\mathbf{g}^2(Y)] \le 1. \tag{118}$$

It is well-known [60] that, if *X* ∼ *QX* and *Y* ∼ *QY* = *QXWY*|*X*, then the contraction coefficient for the chi-squared divergence *µχ*2 (*QX*, *WY*|*X*) is equal to the square of the maximal correlation between the random variables *X* and *Y*, i.e.,

$$
\rho\_{\mathbf{m}}(\mathbf{X};\mathbf{Y}) = \sqrt{\mu\_{\chi^2}(Q\_{\mathbf{X}\prime}W\_{\mathbf{Y}|\mathbf{X}})}.\tag{119}
$$

A simple application of Corollary 1 and (119) gives the following result.

**Proposition 4.** *In the setting of Definition 7, for s* ∈ [0, 1]*, let Xs* ∼ (1 − *s*)*PX* + *sQX and Ys* ∼ (1 − *s*)*PY* + *sQY with PX* 6= *QX and PX QX. Then, the following inequality holds:*

$$\sup\_{s \in [0,1]} \rho\_{\mathrm{m}}(X\_{\mathrm{s}}; Y\_{\mathrm{s}}) \geq \max \left\{ \sqrt{\frac{D(P\_{\mathrm{Y}} \| Q\_{\mathrm{Y}})}{D(P\_{\mathrm{X}} \| Q\_{\mathrm{X}})}}, \sqrt{\frac{D(Q\_{\mathrm{Y}} \| P\_{\mathrm{Y}})}{D(Q\_{\mathrm{X}} \| P\_{\mathrm{X}})}} \right\}.\tag{120}$$

**Proof.** See Section 5.11.

#### **5. Proofs**

This section provides proofs of the results in Sections 3 and 4.

#### *5.1. Proof of Theorem 1*

*Proof of* (22): We rely on an integral representation of the logarithm function (on base e):

$$\log\_e x = \int\_0^1 \frac{x - 1}{x + (1 - x)v} \,\mathrm{d}v, \quad \forall x > 0. \tag{121}$$

Let *µ* be a dominating measure of *P* and *Q* (i.e., *P*, *Q µ*), and let *p* := d*P* d*µ* , *q* := d*Q* d*µ* , and

$$r\_{\lambda} := \frac{\mathbf{d}R\_{\lambda}}{\mathbf{d}\mu} = (1 - \lambda)p + \lambda q\_{\prime} \quad \forall \lambda \in [0, 1], \tag{122}$$

where the last equality is due to (21). For all *λ* ∈ [0, 1],

$$\frac{1}{\log \text{e}} D(P \| R\_{\lambda}) = \int p \log\_{\text{e}} \left( \frac{p}{r\_{\lambda}} \right) \text{d}\mu \tag{123}$$

$$=\int\_{0}^{1}\int \frac{p(p-r\_{\lambda})}{p+v(r\_{\lambda}-p)}\,\mathrm{d}\mu\,\mathrm{d}v,\tag{124}$$

where (124) holds due to (121) with *x* := *p rλ* , and by swapping the order of integration. The inner integral on the right side of (124) satisfies, for all *v* ∈ (0, 1],

$$\int \frac{p(p - r\_{\lambda})}{p + v(r\_{\lambda} - p)} \, \mathrm{d}\mu = \int (p - r\_{\lambda}) \left( 1 + \frac{v(p - r\_{\lambda})}{p + v(r\_{\lambda} - p)} \right) \, \mathrm{d}\mu \tag{125}$$

$$=\int (p - r\_{\lambda}) \, \mathrm{d}\mu + v \int \frac{(p - r\_{\lambda})^2}{p + v(r\_{\lambda} - p)} \, \mathrm{d}\mu \tag{126}$$

$$= v \int \frac{(p - r\_{\lambda})^2}{(1 - v)p + vr\_{\lambda}} \, \text{d}\mu \tag{127}$$

$$=\frac{1}{v}\int \frac{\left(p-\left[(1-v)p+vr\_{\lambda}\right]\right)^{2}}{(1-v)p+vr\_{\lambda}}\,\mathrm{d}\mu\,\tag{128}$$

$$=\frac{1}{v}\chi^2\{P\,\|\,(1-v)P+v\mathcal{R}\_\lambda\}\_\prime\tag{129}$$

where (127) holds since R *p* d*µ* = 1, and R *rλ* d*µ* = 1. From (21), for all (*λ*, *v*) ∈ [0, 1] × [0, 1],

$$(1 - v)P + vR\_{\lambda} = (1 - \lambda v)P + \lambda v\\\bigotimes = R\_{\lambda v}.\tag{130}$$

The substitution of (130) into the right side of (129) gives that, for all (*λ*, *v*) ∈ [0, 1] × (0, 1],

$$\int \frac{p(p - r\_{\lambda})}{p + v(r\_{\lambda} - p)} \, \mathrm{d}\mu = \frac{1}{v} \, \chi^{2}(P || R\_{\lambda \nu}).\tag{131}$$

Finally, substituting (131) into the right side of (124) gives that, for all *λ* ∈ (0, 1],

$$\frac{1}{\text{loge}} D(P \| \| R\_{\lambda}) = \int\_{0}^{1} \frac{1}{v} \chi^{2}(P \| \| R\_{\lambda v}) \, \text{d}v \tag{132}$$

$$=\int\_0^\lambda \frac{1}{s} \chi^2(P \| \mathbf{R}\_s) \, \mathrm{d}s,\tag{133}$$

where (133) holds by the transformation *s* := *λv*. Equality (133) also holds for *λ* = 0 since we have *D*(*P*k*R*0) = *D*(*P*k*P*) = 0.

*Proof of* (23): For all *s* ∈ (0, 1],

$$\begin{split} \chi^2(P \| \mathcal{Q}) &= \int \frac{(p-q)^2}{q} \, \mathrm{d}\mu \\ &= \frac{1}{s^2} \int \frac{\left[ \left( sp + (1-s)q \right) - q \right]^2}{q} \, \mathrm{d}\mu \end{split} \tag{134}$$

$$=\frac{1}{s^2} \int \frac{\left(r\_{1-s} - q\right)^2}{q} \,\mathrm{d}\mu\,\tag{135}$$

$$=\frac{1}{s^2}\chi^2(\mathbb{R}\_{1-s}\parallel\mathbb{Q}),\tag{136}$$

where (135) holds due to (122). From (136), it follows that for all *λ* ∈ [0, 1],

$$\int\_0^\lambda \frac{1}{s} \chi^2(\mathbb{R}\_{1-s} \| \, \mathbb{Q}) \, \mathrm{d}s = \int\_0^\lambda s \, \mathrm{d}s \, \chi^2(P \| \, \mathbb{Q}) = \frac{1}{2} \lambda^2 \chi^2(P \| \, \mathbb{Q}).\tag{137}$$

*5.2. Proof of Proposition 1*

(a) *Simple Proof of Pinsker's Inequality*: By [61] or [62] [(58)],

$$\chi^2(P\|\|Q) \ge \begin{cases} |P - Q|^2, & \text{if } |P - Q| \in [0, 1], \\ \frac{|P - Q|}{2 - |P - Q|}, & \text{if } |P - Q| \in (1, 2]. \end{cases} \tag{138}$$

We need the weaker inequality *χ* 2 (*P*k*Q*) ≥ |*P* − *Q*| 2 , proved by the Cauchy–Schwarz inequality:

$$\chi^2(P\|\mathbb{Q}) = \int \frac{(p-q)^2}{q} \, \text{d}\mu \int q \, \text{d}\mu \tag{139}$$

$$\geq \left( \int \frac{|p-q|}{\sqrt{q}} \cdot \sqrt{q} \, \mathrm{d}\mu \right)^2 \tag{140}$$

$$= |P - Q|^2. \tag{141}$$

By combining (24) and (139)–(141), it follows that

$$\frac{1}{\text{loge}}D(P\|\|Q) = \int\_0^1 \chi^2(P\|\|(1-s)P + sQ) \, \frac{\text{ds}}{\text{s}} \tag{142}$$

$$\geq \int\_{0}^{1} \left| P - \left( (1 - s)P + sQ \right) \right|^{2} \frac{ds}{s} \tag{143}$$

$$=\int\_0^1 \mathbf{s} \left| P - \mathbb{Q} \right|^2 \mathbf{d} \mathbf{s} \tag{144}$$

$$= \frac{1}{2} \|P - Q\|^2. \tag{145}$$

(b) *Proof of* (30) *and its local tightness*:

$$\frac{1}{2\log e}D(P\|\|\mathcal{Q}) = \int\_0^1 \chi^2(P\|\left(1-s\right)P + s\mathcal{Q})\,\frac{\mathrm{d}s}{s} \tag{146}$$

$$\mathcal{I} = \int\_0^1 \left( \int \frac{\left[ p - ((1-s)p + sq) \right]^2}{(1-s)p + sq} \, \mathrm{d}\mu \right) \frac{\mathrm{d}s}{\mathrm{s}} \tag{147}$$

$$\rho = \int\_0^1 \int \frac{s(p-q)^2}{(1-s)p+sq} \,\mathrm{d}\mu\,\mathrm{d}s \tag{148}$$

$$1 \le \int\_0^1 \int s(p-q)^2 \left(\frac{1-s}{p} + \frac{s}{q}\right) \mathrm{d}\mu \text{ ds} \tag{149}$$

$$=\int\_0^1 s^2 \, \mathrm{d}s \, \int \frac{(p-q)^2}{q} \, \mathrm{d}\mu + \int\_0^1 s(1-s) \, \mathrm{d}s \, \int \frac{(p-q)^2}{p} \, \mathrm{d}\mu \tag{150}$$

$$=\frac{1}{5}\chi^2(P\|Q) + \frac{1}{6}\chi^2(Q\|P). \tag{151}$$

where (146) is (24), and (149) holds due to Jensen's inequality and the convexity of the hyperbola. We next show the local tightness of inequality (30) by proving that (31) yields (32). Let {*Pn*} be a sequence of probability measures, defined on a measurable space (X , F ), and assume that {*Pn*}

converges to a probability measure *P* in the sense that (31) holds. In view of [16] [Theorem 7] (see also [15] [Section 4.F] and [63]), it follows that

$$\lim\_{n \to \infty} D(P\_n || P) = \lim\_{n \to \infty} \chi^2(P\_n || P) = 0,\tag{152}$$

and

$$\lim\_{n \to \infty} \frac{D(P\_n || P)}{\chi^2(P\_n || P)} = \frac{1}{2} \text{ log\,e},\tag{153}$$

$$\lim\_{n \to \infty} \frac{\chi^2(P\_n || P)}{\chi^2(P || P\_n)} = 1,\tag{154}$$

which therefore yields (32).

(c) *Proof of* (33) *and* (34): The proof of (33) relies on (28) and the following lemma.

**Lemma 1.** *For all s*, *θ* ∈ (0, 1)*,*

$$\frac{D\_{\Phi^\*}(P||Q)}{D\_{\Phi^\*}(P||Q)} \ge \min\left\{\frac{1-\theta}{1-s}, \frac{\theta}{s}\right\}.\tag{155}$$

**Proof.**

$$D\_{\Phi\_{\mathbb{S}}}(P\|Q) = \int \frac{(p-q)^2}{(1-s)p + sq} \, \mathrm{d}\mu \tag{156}$$

$$=\int \frac{(p-q)^2}{(1-\theta)p+\theta q} \frac{(1-\theta)p+\theta q}{(1-s)p+sq} \, \mathrm{d}\mu \tag{157}$$

$$\geq \min\left\{ \frac{1-\theta}{1-s}, \frac{\theta}{s} \right\} \int \frac{(p-q)^2}{(1-\theta)p+\theta q} \cdot \mathbf{d}\mu \tag{158}$$

$$=\min\left\{\frac{1-\theta}{1-s}, \frac{\theta}{s}\right\} D\_{\Phi\flat}(P||Q). \tag{159}$$

From (28) and (155), for all *θ* ∈ (0, 1),

$$\frac{1}{2\log e}D(P\|\|Q) = \int\_0^\theta s D\_{\Phi\_\theta}(P\|\|Q) \, \mathrm{d}s + \int\_\theta^1 s D\_{\Phi\_\theta}(P\|\|Q) \, \mathrm{d}s \tag{160}$$

$$\geq \int\_{0}^{\theta} \frac{s \left(1 - \theta\right)}{1 - s} \cdot D\_{\theta\theta} \left(P \|\| Q \right) \, \mathrm{d}s + \int\_{\theta}^{1} \theta \, D\_{\theta\theta} \left(P \|\| Q \right) \, \mathrm{d}s \tag{161}$$

$$=\left[-\theta + \log\_e\left(\frac{1}{1-\theta}\right)\right] \left(1-\theta\right) D\rho\_\theta(P\|\mathcal{Q}) + \theta(1-\theta)D\rho\_\theta(P\|\mathcal{Q})\tag{162}$$

$$=(1-\theta)\log\_{\theta}\left(\frac{1}{1-\theta}\right)D\_{\theta\theta}(P\|\!|Q\!).\tag{163}$$

This proves (33). Furthermore, under the assumption in (31), for all *θ* ∈ [0, 1],

$$\lim\_{n \to \infty} \frac{D(P \| P\_n)}{D\_{\Phi\sharp}(P \| P\_n)} = \lim\_{n \to \infty} \frac{D(P \| P\_n)}{\chi^2(P \| P\_n)} \lim\_{n \to \infty} \frac{\chi^2(P \| P\_n)}{D\_{\Phi\sharp}(P \| P\_n)}\tag{164}$$

$$=\frac{1}{2}\log\mathbf{e}\cdot\frac{2}{\phi\_{\theta}^{\prime\prime}(1)}\tag{165}$$

$$= \frac{1}{2} \log \mathbf{e}\_{\prime} \tag{166}$$

where (165) holds due to (153) and the local behavior of *f*-divergences [63], and (166) holds due to (17) which implies that *φ* 00 *θ* (1) = 2 for all *θ* ∈ [0, 1]. This proves (34).

(d) *Proof of* (35): From (24), we get

$$\frac{1}{\text{loge}}D(P\|\|Q) = \int\_0^1 \chi^2(P\|\|\mathbf{1}(1-\mathbf{s})\mathbf{P} + \mathbf{s}\mathbf{Q}) \, \frac{\mathbf{ds}}{\mathbf{s}} \tag{167}$$

$$I = \int\_0^1 \left[ \chi^2(P \parallel (1 - s)P + sQ) - s^2 \chi^2(P \parallel Q) \right] \frac{ds}{s} + \int\_0^1 s \, \text{ds} \, \chi^2(P \parallel Q) \tag{168}$$

$$=\int\_0^1 \left[\chi^2(P\|\left(1-s\right)P+s\mathbb{Q})-s^2\,\chi^2(P\|\mathbb{Q})\right] \frac{ds}{s} + \frac{1}{2}\chi^2(P\|\mathbb{Q}).\tag{169}$$

Referring to the integrand of the first term on the right side of (169), for all *s* ∈ (0, 1],

$$\frac{1}{s} \left[ \chi^2(P \parallel (1-s)P + sQ) - s^2 \chi^2(P \parallel Q) \right]$$

$$\xi = s \int (p-q)^2 \left[ \frac{1}{(1-s)p + sq} - \frac{1}{q} \right] d\mu \tag{170}$$

$$s = s(1 - s) \int \frac{(q - p)^3}{q \left[ (1 - s)p + sq \right]} \, \text{d}\mu \tag{171}$$

$$\rho = s(1-s) \int |q-p| \cdot \underbrace{\frac{|q-p|}{q} \cdot \frac{q-p}{p+s(q-p)}}\_{\leq \frac{1}{s} \, 1 \{q \geq p\}} \, \text{d}\mu \tag{172}$$

$$\leq (1-s)\int (q-p)\,\mathbf{1}\{q \geq p\} \,\mathrm{d}\mu \tag{173}$$

*s*

$$= \frac{1}{2}(1 - s)\left|P - Q\right|\tag{174}$$

where the last equality holds since the equality R (*q* − *p*) d*µ* = 0 implies that

$$\int (q-p)\,\mathbf{1}\{q \ge p\} \,\mathrm{d}\mu = \int (p-q)\,\mathbf{1}\{p \ge q\} \,\mathrm{d}\mu \tag{175}$$

$$=\frac{1}{2}\int |p-q|\,\mathrm{d}\mu = \frac{1}{2}\,|P-Q|.\tag{176}$$

From (170)–(174), an upper bound on the right side of (169) results. This gives

$$\frac{1}{2\log\mathrm{e}}D(P||Q) \le \frac{1}{2}\int\_0^1 (1-s)\,\mathrm{ds}\,\left|P-Q\right| + \frac{1}{2}\chi^2(P||Q) \tag{177}$$

$$=\frac{1}{4}\left|P-Q\right|+\frac{1}{2}\chi^{2}(P\|\|Q).\tag{178}$$

It should be noted that [15] [Theorem 2(a)] shows that inequality (35) is tight. To that end, let *ε* ∈ (0, 1), and define probability measures *Pε* and *Qε* on the set A = {0, 1} with *Pε*(1) = *ε* 2 and *Qε*(1) = *ε*. Then,

$$\lim\_{\varepsilon \downarrow 0} \frac{\frac{1}{\log e} D(P\_{\varepsilon} || Q\_{\varepsilon})}{\frac{1}{4} |P\_{\varepsilon} - Q\_{\varepsilon}| + \frac{1}{2} \chi^{2} (P\_{\varepsilon} || Q\_{\varepsilon})} = 1. \tag{179}$$

#### *5.3. Proof of Theorem 2*

We first prove Item (a) in Theorem 2. In view of the Hammersley–Chapman–Robbins lower bound on the *χ* 2 divergence, for all *λ* ∈ [0, 1]

$$\chi^2(P\|(1-\lambda)P+\lambda Q) \ge \frac{\left(\mathbb{E}[X] - \mathbb{E}[Z\_{\lambda}]\right)^2}{\text{Var}(Z\_{\lambda})},\tag{180}$$

where *X* ∼ *P*, *Y* ∼ *Q* and *Zλ* ∼ *Rλ* := (1 − *λ*)*P* + *λQ* is defined by

$$Z\_{\lambda} := \begin{cases} X, & \text{with probability } 1 - \lambda, \\ Y, & \text{with probability } \lambda. \end{cases} \tag{181}$$

For *λ* ∈ [0, 1],

$$\mathbb{E}[Z\_{\lambda}] = (1 - \lambda)m\_P + \lambda m\_{Q'} \tag{182}$$

and it can be verified that

$$\text{Var}(Z\_{\lambda}) = (1 - \lambda)\sigma\_P^2 + \lambda\sigma\_Q^2 + \lambda(1 - \lambda)(m\_P - m\_Q)^2. \tag{183}$$

We now rely on (24)

$$\frac{1}{\text{loge}}D(P||Q) = \int\_0^1 \chi^2(P||(1-\lambda)P + \lambda Q) \, \frac{d\lambda}{\lambda} \tag{184}$$

to get a lower bound on the relative entropy. Combining (180), (183) and (184) yields

$$\frac{1}{\text{logse}}D(P\|Q) \ge (m\_P - m\_Q)^2 \int\_0^1 \frac{\lambda}{(1-\lambda)\sigma\_P^2 + \lambda\sigma\_Q^2 + \lambda(1-\lambda)(m\_P - m\_Q)^2}d\lambda. \tag{185}$$

From (43) and (44), we get

$$\int\_0^1 \frac{\lambda}{(1-\lambda)\sigma\_P^2 + \lambda\sigma\_Q^2 + \lambda(1-\lambda)(m\_P - m\_Q)^2} \,\mathrm{d}\lambda = \int\_0^1 \frac{\lambda}{(a-a\lambda)(\beta+a\lambda)} \,\mathrm{d}\lambda,\tag{186}$$

where

$$\mathfrak{a} := \sqrt{\sigma\_P^2 + \frac{b^2}{4a^2}} + \frac{b}{2a'} \tag{187}$$

$$\mathcal{B} := \sqrt{\sigma\_P^2 + \frac{b^2}{4a^2}} - \frac{b}{2a}. \tag{188}$$

By using the partial fraction decomposition of the integrand on the right side of (186), we get (after multiplying both sides of (185) by log e)

$$D(P||Q) \ge \frac{(m\_P - m\_Q)^2}{a^2} \left[ \frac{a}{a+\beta} \log\left(\frac{a}{a-a}\right) + \frac{\beta}{a+\beta} \log\left(\frac{\beta}{\beta+a}\right) \right] \tag{189}$$

$$\alpha = \frac{\alpha}{\alpha + \beta} \log \left( \frac{\alpha}{\alpha - a} \right) + \frac{\beta}{\alpha + \beta} \log \left( \frac{\beta}{\beta + a} \right) \tag{190}$$

$$
\zeta = d \left( \frac{a}{a+\beta} \parallel \frac{a-a}{a+\beta} \right),
\tag{191}
$$

where (189) holds by integration since *α* − *aλ* and *β* + *aλ* are both non-negative for all *λ* ∈ [0, 1]. To verify the latter claim, it should be noted that (43) and the assumption that *mP* 6= *mQ* imply that *a* 6= 0. Since *α*, *β* > 0, it follows that, for all *λ* ∈ [0, 1], either *α* − *aλ* > 0 or *β* + *aλ* > 0 (if *a* < 0, then the former is positive, and, if *a* > 0, then the latter is positive). By comparing the denominators of both integrands on the left and right sides of (186), it follows that (*α* − *aλ*)(*β* + *aλ*) ≥ 0 for all *λ* ∈ [0, 1]. Since the product of *α* − *aλ* and *β* + *aλ* is non-negative and at least one of these terms is positive, it follows that *α* − *aλ* and *β* + *aλ* are both non-negative for all *λ* ∈ [0, 1]. Finally, (190) follows from (43).

If *mP* − *mQ* → 0 and *σP* 6= *σQ*, then it follows from (43) and (44) that *a* → 0 and *b* → *σ* 2 *P* − *σ* 2 *Q* 6= 0. Hence, from (187) and (188), *α* ≥ *b a* → ∞ and *β* → 0, which implies that the lower bound on *D*(*P*k*Q*) in (191) tends to zero.

Letting *r* := *α α*+*β* and *s* := *α*−*a α*+*β* , we obtain that the lower bound on *D*(*P*k*Q*) in (40) holds. This bound is consistent with the expressions of *r* and *s* in (41) and (42) since, from (45), (187) and (188),

$$r = \frac{a}{a+\beta} = \frac{v + \frac{b}{\Delta a}}{\Delta v} = \frac{1}{2} + \frac{b}{4av'} \tag{192}$$

$$s = \frac{\alpha - a}{\alpha + \beta} = r - \frac{a}{\alpha + \beta} = r - \frac{a}{2v}.\tag{193}$$

It should be noted that *r*,*s* ∈ [0, 1]. First, from (187) and (188), *α* and *β* are positive if *σP* 6= 0, which yields *r* = *α α*+*β* ∈ (0, 1). We next show that *s* ∈ [0, 1]. Recall that *α* − *aλ* and *β* + *aλ* are both non-negative for all *λ* ∈ [0, 1]. Setting *λ* = 1 yields *α* ≥ *a*, which (from (193)) implies that *s* ≥ 0. Furthermore, from (193) and the positivity of *α* + *β*, it follows that *s* ≤ 1 if and only if *β* ≥ −*a*. The latter holds since *β* + *aλ* ≥ 0 for all *λ* ∈ [0, 1] (in particular, for *λ* = 1). If *σP* = 0, then it follows from (41)–(45) that *v* = *b* 2|*a*| , *b* = *a* 2 + *σ* 2 *Q* , and (recall that *a* 6= 0)

$$\begin{aligned} \text{(i)} \qquad \text{If } a > 0 \text{, then } v = \frac{b}{2a} \text{ implies that } r = \frac{1}{2} + \frac{b}{4av} = 1 \text{, and } s = r - \frac{a}{2v} = 1 - \frac{a^2}{b} = \frac{\sigma\_Q^2}{\sigma\_Q^2 + a^2} \in [0, 1];\\ \text{(iii)} \qquad \text{v} \qquad \text{and} \qquad \text{h} \qquad \text{h} \qquad \text{v} \qquad \text{h} \qquad \text{h} \qquad \text{h} \qquad \text{s} \qquad \text{h} \qquad \text{s}^2 \qquad \text{s}^2 \qquad \text{h} \qquad \text{h} \qquad \text{s} \qquad \text{h} \end{aligned}$$

$$\text{(ii)}\qquad\text{if }a<0,\text{ then }v=-\frac{b}{2a}\text{ implies that }r=0,\text{and }s=r-\frac{a}{2v}=\frac{a^2}{b}=\frac{a^2}{a^2+v\_Q^2}\in[0,1].$$

We next prove Item (b) in Theorem 2 (i.e., the achievability of the lower bound in (40)). To that end, we provide a technical lemma, which can be verified by the reader.

**Lemma 2.** *Let r*,*s be given in* (41)*–*(45)*, and let u*1,2 *be given in* (47)*. Then,*

$$(s-r)(\mu\_1 - \mu\_2) = m\_Q - m\_{P\_I} \tag{194}$$

$$
\mu\_1 + \mu\_2 = m\_P + m\_Q + \frac{\sigma\_Q^2 - \sigma\_P^2}{m\_Q - m\_P}.\tag{195}
$$

Let *X* ∼ *P* and *Y* ∼ *Q* be defined on a set U = {*u*1, *u*2} (for the moment, the values of *u*1 and *u*2 are not yet specified) with *P*[*X* = *u*1] = *r*, *P*[*X* = *u*2] = 1 − *r*, *Q*[*Y* = *u*1] = *s*, and *Q*[*Y* = *u*2] = 1 − *s*. We now calculate *u*1 and *u*2 such that E[*X*] = *mP* and Var(*X*) = *σ* 2 *P* . This is equivalent to

$$
\mathfrak{r}u\_1 + (1-r)u\_2 = \mathfrak{m}\_{P\_1} \tag{196}
$$

$$
\sigma u\_1^2 + (1 - r)u\_2^2 = m\_P^2 + \sigma\_P^2. \tag{197}
$$

Substituting (196) into the right side of (197) gives

$$r u\_1^2 + (1 - r) u\_2^2 = \left[ r u\_1 + (1 - r) u\_2 \right]^2 + \sigma\_{\rm P'}^2 \tag{198}$$

*Entropy* **2020**, *22*, 563

which, by rearranging terms, also gives

$$
\mu\_1 - \mu\_2 = \pm \sqrt{\frac{\sigma\_P^2}{r(1-r)}}.\tag{199}
$$

Solving simultaneously (196) and (199) gives

$$
\mu\_1 = m\_P \pm \sqrt{\frac{(1-r)\sigma\_P^2}{r}},\tag{200}
$$

$$
\mu\_2 = m\_P \mp \sqrt{\frac{r\sigma\_P^2}{1-r}}.\tag{201}
$$

We next verify that, by setting *u*1,2 as in (47), one also gets (as desired) that E[*Y*] = *mQ* and Var(*Y*) = *σ* 2 *Q* . From Lemma 2, and, from (196) and (197), we have

$$\mathbb{E}[Y] = su\_1 + (1 - s)u\_2 \tag{202}$$

$$\mathbf{u} = \left(r\boldsymbol{u}\_1 + (1 - r)\boldsymbol{u}\_2\right) + (\mathbf{s} - r)(\boldsymbol{u}\_1 - \boldsymbol{u}\_2) \tag{203}$$

$$
\dot{\mathbf{u}}\_1 = m\_P + (s - r)(\mathbf{u}\_1 - \mathbf{u}\_2) = m\_{Q'} \tag{204}
$$

$$\mathbb{E}[Y^2] = su\_1^2 + (1-s)u\_2^2\tag{205}$$

$$=ru\_1^2 + (1-r)u\_2^2 + (s-r)(u\_1^2 - u\_2^2) \tag{206}$$

$$\mathbb{E} = \mathbb{E}[X^2] + (s - r)(\mu\_1 - \mu\_2)(\mu\_1 + \mu\_2) \tag{207}$$

$$\sigma = m\_P^2 + \sigma\_P^2 + (m\_Q - m\_P) \left( m\_P + m\_Q + \frac{\sigma\_Q^2 - \sigma\_P^2}{m\_Q - m\_P} \right) \tag{208}$$

$$
\sigma = m\_Q^2 + \sigma\_Q^2. \tag{209}
$$

By combining (204) and (209), we obtain Var(*Y*) = *σ* 2 *Q* . Hence, the probability mass functions *P* and *Q* defined on U = {*u*1, *u*2} (with *u*1 and *u*2 in (47)) such that

$$P(\mu\_1) = 1 - P(\mu\_2) = r, \quad Q(\mu\_1) = 1 - Q(\mu\_2) = s \tag{210}$$

satisfy the equality constraints in (39), while also achieving the lower bound on *D*(*P*k*Q*) that is equal to *d*(*r*k*s*). It can be also verified that the second option where

$$u\_1 = m\_P - \sqrt{\frac{(1-r)\sigma\_P^2}{r}}, \quad u\_2 = m\_P + \sqrt{\frac{r\sigma\_P^2}{1-r}}\tag{211}$$

does *not* yield the satisfiability of the conditions E[*Y*] = *mQ* and Var(*Y*) = *σ* 2 *Q* , so there is only a unique pair of probability measures *P* and *Q*, defined on a two-element set that achieves the lower bound in (40) under the equality constraints in (39).

We finally prove Item (c) in Theorem 2. Let *m* ∈ R, *σ* 2 *P* , and *σ* 2 *Q* be selected arbitrarily such that *σ* 2 *Q* ≥ *σ* 2 *P* . We construct probability measures *Pε* and *Qε* , depending on a free parameter *ε*, with means *mP* = *mQ* := *m* and variances *σ* 2 *P* and *σ* 2 *Q* , respectively (means and variances are independent of *ε*), and which are defined on a three-element set U := {*u*1, *u*2, *u*3} as follows:

$$P\_{\varepsilon}(\mathfrak{u}\_{1}) = r, \quad P\_{\varepsilon}(\mathfrak{u}\_{2}) = 1 - r, \quad \qquad P\_{\varepsilon}(\mathfrak{u}\_{3}) = \mathbf{0}, \tag{212}$$

$$Q\_{\varepsilon}(\mathfrak{u}\_{1}) = \mathfrak{s}, \quad Q\_{\varepsilon}(\mathfrak{u}\_{2}) = 1 - \mathfrak{s} - \varepsilon, \quad Q\_{\varepsilon}(\mathfrak{u}\_{3}) = \varepsilon,\tag{213}$$

with *ε* > 0. We aim to set the parameters *r*,*s*, *u*1, *u*2 and *u*3 (as a function of *m*, *σP*, *σQ* and *ε*) such that

$$\lim\_{\varepsilon \to 0^{+}} D(P\_{\varepsilon} \| Q\_{\varepsilon}) = 0. \tag{214}$$

Proving (214) yields (48), while it also follows that the infimum on the left side of (48) can be restricted to probability measures which are defined on a three-element set.

In view of the constraints on the means and variances in (39), with equal means *m*, we get the following set of equations from (212) and (213):

$$\begin{cases} ru\_1 + (1 - r)u\_2 = m\_\prime \\ su\_1 + (1 - s - \varepsilon)u\_2 + \varepsilon u\_3 = m\_\prime \\ ru\_1^2 + (1 - r)u\_2^2 = m^2 + \sigma\_{P'}^2 \\ su\_1^2 + (1 - s - \varepsilon)u\_2^2 + \varepsilon u\_3^2 = m^2 + \sigma\_Q^2. \end{cases} \tag{215}$$

The first and second equations in (215) refer to the equal means under *P* and *Q*, and the third and fourth equations in (215) refer to the second moments in (39). Furthermore, in view of (212) and (213), the relative entropy is given by

$$D(P\_{\varepsilon} \| Q\_{\varepsilon}) = r \log \frac{r}{s} + (1 - r) \log \frac{1 - r}{1 - s - \varepsilon}. \tag{216}$$

Subtracting the square of the first equation in (215) from its third equation gives the equivalent set of equations

$$\begin{cases} ru\_1 + (1 - r)u\_2 = m\_\star \\ su\_1 + (1 - s - \varepsilon)u\_2 + \varepsilon u\_3 = m\_\star \\\ r(1 - r)(u\_1 - u\_2)^2 = \sigma\_{P\prime}^2 \\ su\_1^2 + (1 - s - \varepsilon)u\_2^2 + \varepsilon u\_3^2 = m^2 + \sigma\_Q^2. \end{cases} \tag{217}$$

We next select *u*1 and *u*2 such that *u*1 − *u*2 := 2*σP*. Then, the third equation in (217) gives *r*(1 − *r*) = 1 4 , so *r* = 1 2 . Furthermore, the first equation in (217) gives

$$
u\_1 = m + \sigma\_{P'} \tag{218}$$

$$
\mu\_2 = m - \sigma\_P.\tag{219}
$$

Since *r*, *u*1, and *u*2 are independent of *ε*, so is the probability measure *Pε* := *P*. Combining the second equation in (217) with (218) and (219) gives

$$
\mu\_3 = m - \left(1 + \frac{2s - 1}{\varepsilon}\right) \sigma\_P.\tag{220}
$$

Substituting (218)–(220) into the fourth equation of (217) gives a quadratic equation for *s*, whose selected solution (such that *s* and *r* = 1 2 be close for small *e* > 0) is equal to

$$s = \frac{1}{2} \left[ 1 - \varepsilon + \sqrt{\left( \frac{\sigma\_Q^2}{\sigma\_P^2} - 1 + \varepsilon \right)} \varepsilon \right]. \tag{221}$$

Hence, *s* = 1 2 + *O*( √ *ε*), which implies that *s* ∈ (0, 1 − *ε*) for sufficiently small *ε* > 0 (as it is required in (213)). In view of (216), it also follows that *D*(*P*k*Qε*) vanishes as we let *ε* tend to zero.

We finally outline an alternative proof, which refers to the case of equal means with arbitrarily selected *σ* 2 *P* and *σ* 2 *Q* . Let (*σ* 2 *P* , *σ* 2 *Q* ) ∈ (0, ∞) 2 . We next construct a sequence of pairs of probability measures {(*Pn*, *Qn*)} with zero mean and respective variances (*σ* 2 *P* , *σ* 2 *Q* ) for which *D*(*Pn*k*Qn*) → 0 as *n* → ∞ (without any loss of generality, one can assume that the equal means are equal to zero). We start by assuming (*σ* 2 *P* , *σ* 2 *Q* ) ∈ (1, ∞) 2 . Let

$$
\mu\_n := \sqrt{1 + n(\sigma\_Q^2 - 1)},
\tag{222}
$$

and define a sequence of quaternary real-valued random variables with probability mass functions

$$Q\_n(a) := \begin{cases} \frac{1}{2} - \frac{1}{2n} & a = \pm 1, \\ \frac{1}{2n} & a = \pm \mu\_n. \end{cases} \tag{223}$$

It can be verified that, for all *n* ∈ N, *Qn* has zero mean and variance *σ* 2 *Q* . Furthermore, let

$$P\_n(a) := \begin{cases} \frac{1}{2} - \frac{\frac{\pi}{2}}{2n} & a = \pm 1, \\\frac{\frac{\pi}{2}}{2n} & a = \pm \mu\_{n\prime} \end{cases} \tag{224}$$

with

$$
\zeta := \frac{\sigma\_P^2 - 1}{\sigma\_Q^2 - 1}.
\tag{225}
$$

If *ξ* > 1, for *n* = 1, . . . , d*ξ*e, we choose *Pn* arbitrarily with mean 0 and variance *σ* 2 *P* . Then,

$$\text{Var}(P\_{\text{fl}}) = 1 - \frac{\frac{\pi}{2}}{n} + \frac{\frac{\pi}{n}}{n} \mu\_{\text{n}}^2 = \sigma\_{\text{P}}^2 \tag{226}$$

$$D(P\_n \| Q\_n) = d\left(\frac{\xi}{n} \middle| \frac{1}{n}\right) \to 0. \tag{227}$$

Next, suppose min{*σ* 2 *P* , *σ* 2 *Q* } := *σ* 2 < 1, then construct *P* 0 *n* and *Q*0 *n* as before with variances 2*σ* 2 *P σ* 2 > 1 and 2*σ* 2 *Q σ* 2 > 1, respectively. If *Pn* and *Qn* denote the random variables *P* 0 *n* and *Q*0 *n* scaled by a factor of √*σ* 2 , then their variances are *σ* 2 *P* , *σ* 2 *Q* , respectively, and *D*(*Pn*k*Qn*) = *D*(*P* 0 *n*k*Q*0 *n* ) → 0 as we let *n* → ∞.

To conclude, it should be noted that the sequences of probability measures in the latter proof are defined on a four-element set. Recall that, in the earlier proof, specialized to the case of (equal means with) *σ* 2 *P* ≤ *σ* 2 *Q* , the introduced probability measures are defined on a three-element set, and the reference probability measure *P* is fixed while referring to an equiprobable binary random variable.

#### *5.4. Proof of Theorem 3*

We first prove (52). Differentiating both sides of (22) gives that, for all *λ* ∈ (0, 1],

$$F'(\lambda) = \frac{1}{\lambda} \chi^2(P \| R\_{\lambda}) \text{ log e} \tag{228}$$

$$\frac{1}{\lambda} \left[ \exp \left( D(P \| R\_{\lambda}) \right) - 1 \right] \log \mathbf{e} \tag{229}$$

$$\hat{\lambda} = \frac{1}{\lambda} \left[ \exp \left( F(\lambda) \right) - 1 \right] \text{ log\,e},\tag{230}$$

where (228) holds due to (21), (22) and (50); (229) holds by (16) and (230) is due to (21) and (50). This gives (52).

We next prove (53), and the conclusion which appears after it. In view of [16] [Theorem 8], applied to *f*(*t*) := − log *t* for all *t* > 0, we get (it should be noted that, by the definition of *F* in (50), the result in [16] [(195)–(196)] is used here by swapping *P* and *Q*)

$$\lim\_{\lambda \to 0^+} \frac{F(\lambda)}{\lambda^2} = \frac{1}{2} \chi^2(Q \| P) \text{ log.e.} \tag{231}$$

Since lim *λ*→0+ *F*(*λ*) = 0, it follows by L'Hôpital's rule that

$$\lim\_{\lambda \to 0^+} \frac{F'(\lambda)}{\lambda} = 2 \lim\_{\lambda \to 0^+} \frac{F(\lambda)}{\lambda^2} = \chi^2(Q \| P) \log \mathbf{e}\_\prime \tag{232}$$

which gives (53). A comparison of the limit in (53) with a lower bound which follows from (52) gives

$$\lim\_{\lambda \to 0^+} \frac{F'(\lambda)}{\lambda} \ge \lim\_{\lambda \to 0^+} \frac{1}{\lambda^2} \left[ \exp \left( F(\lambda) \right) - 1 \right] \log \mathbf{e} \tag{233}$$

$$=\lim\_{\lambda \to 0^{+}} \frac{F(\lambda)}{\lambda^{2}} \lim\_{\lambda \to 0^{+}} \frac{\exp\left(F(\lambda)\right) - 1}{F(\lambda)} \cdot \log \mathbf{e} \tag{234}$$

$$=\lim\_{\lambda \to 0^{+}} \frac{F(\lambda)}{\lambda^2} \lim\_{\mu \to 0} \frac{\mathbf{e}^{\mu} - 1}{\mu} \tag{235}$$

$$=\frac{1}{2}\chi^2(\mathbb{Q}||P)\text{ log\,e}\,\tag{236}$$

where (236) relies on (231). Hence, the limit in (53) is twice as large as its lower bound on the right side of (236). This proves the conclusion which comes right after (53).

We finally prove the known result in (51), by showing an alternative proof which is based on (52). The function *F* is non-negative on [0, 1], and it is strictly positive on (0, 1] if *P* 6= *Q*. Let *P* 6= *Q* (otherwise, (51) is trivial). Rearranging terms in (52) and integrating both sides over the interval [*λ*, 1], for *λ* ∈ (0, 1], gives that

$$\int\_{\lambda}^{1} \frac{F'(t)}{\exp\{F(t)\} - 1} \, dt \ge \int\_{\lambda}^{1} \frac{\mathrm{d}t}{t} \, \log \mathrm{e} \tag{237}$$

$$=\log\frac{1}{\lambda'} \quad \forall \,\lambda \in (0,1]. \tag{238}$$

The left side of (237) satisfies

$$\int\_{\lambda}^{1} \frac{F'(t)}{\exp(F(t)) - 1} \, \mathrm{d}t = \int\_{\lambda}^{1} \frac{F'(t) \exp(-F(t))}{1 - \exp(-F(t))} \, \mathrm{d}t \tag{239}$$

$$=\int\_{\lambda}^{1} \frac{\mathbf{d}}{\mathbf{d}t} \left\{ \log \left( 1 - \exp \left( -F(t) \right) \right) \right\} \, \mathrm{d}t \tag{240}$$

$$=\log\left(\frac{1-\exp\left(-D(P||Q)\right)}{1-\exp\left(-F(\lambda)\right)}\right),\tag{241}$$

where (241) holds since *F*(1) = *D*(*P*k*Q*) (see (50)). Combining (237)–(241) gives

$$\frac{1 - \exp\left(-D(P\|Q)\right)}{1 - \exp\left(-F(\lambda)\right)} \ge \frac{1}{\lambda}, \quad \forall \lambda \in (0, 1], \tag{242}$$

which, due to the non-negativity of *F*, gives the right side inequality in (51) after rearrangement of terms in (242).

#### *5.5. Proof of Theorem 4*

**Lemma 3.** *Let f*0 : (0, ∞) → R *be a convex function with f*0(1) = 0*, and let* { *fk* (·)} ∞ *k*=0 *be defined as in* (58)*. Then,* { *fk* (·)} ∞ *k*=0 *is a sequence of convex functions on* (0, ∞)*, and*

$$f\_k(\mathbf{x}) \ge f\_{k+1}(\mathbf{x}), \quad \forall \mathbf{x} > \mathbf{0}, \ k \in \{0, 1, \ldots\}. \tag{243}$$

**Proof.** We prove the convexity of { *fk* (·)} on (0, ∞) by induction. Suppose that *fk* (·) is a convex function with *fk* (1) = 0 for a fixed integer *k* ≥ 0. The recursion in (58) yields *fk*+1 (1) = 0 and, by the change of integration variable *s* := (1 − *x*)*s* 0 ,

$$f\_{k+1}(\mathbf{x}) = \int\_0^1 f\_k(s'\mathbf{x} - s' + 1) \frac{\mathbf{ds}'}{\mathbf{s'}}, \quad \mathbf{x} > 0. \tag{244}$$

Consequently, for *t* ∈ (0, 1) and *x* 6= *y* with *x*, *y* > 0, applying (244) gives

$$f\_{k+1}((1-t)\mathbf{x} + ty) = \int\_0^1 f\_k(s'[(1-t)\mathbf{x} + ty] - s' + 1) \frac{\mathbf{ds}'}{\mathbf{s}'} \tag{245}$$

$$=\int\_{0}^{1} f\_{k}\left((1-t)(s'x-s'+1) + t(s'y-s'+1)\right) \frac{\mathbf{ds}'}{\mathbf{s}'}\tag{246}$$

$$\leq (1-t)\int\_{0}^{1} f\_{k}(s'x - s' + 1) \frac{\mathrm{d}s'}{\mathrm{s'}} + t \int\_{0}^{1} f\_{k}(s'y - s' + 1) \frac{\mathrm{d}s'}{\mathrm{s'}} \tag{247}$$

$$=(1-t)f\_{k+1}(\mathbf{x}) + tf\_{k+1}(y),\tag{248}$$

where (247) holds since *fk* (·) is convex on (0, ∞) (by assumption). Hence, from (245)–(248), *fk*+1 (·) is also convex on (0, ∞) with *fk*+1 (1) = 0. By mathematical induction and our assumptions on *f*0, it follows that { *fk* (·)} ∞ *k*=0 is a sequence of convex functions on (0, ∞) which vanish at 1.

We next prove (243). For all *x*, *y* > 0 and *k* ∈ {0, 1, . . .},

$$f\_{k+1}(y) \ge f\_{k+1}(\mathbf{x}) + f\_{k+1}'(\mathbf{x}) \left(y - \mathbf{x}\right) \tag{249}$$

$$f = f\_{k+1}(\mathbf{x}) + \frac{f\_k(\mathbf{x})}{\mathbf{x} - 1} \ (y - \mathbf{x}),\tag{250}$$

where (249) holds since *fk* (·) is convex on (0, ∞), and (250) relies on the recursive equation in (58). Substituting *y* = 1 into (249)–(250), and using the equality *fk*+1 (1) = 0, gives (243).

We next prove Theorem 4. From Lemma 3, it follows that *Dfk* (*P*k*Q*) is an *f*-divergence for all integers *k* ≥ 0, and the non-negative sequence *Dfk* (*P*k*Q*)} ∞ *k*=0 is monotonically non-increasing. From (21) and (58), it also follows that, for all *λ* ∈ [0, 1] and integer *k* ∈ {0, 1, . . .},

$$D\_{f\_{k+1}}(\mathcal{R}\_{\lambda} || P) = \int p \, f\_{k+1} \left( \frac{r\_{\lambda}}{p} \right) \, \mathrm{d}\mu \tag{251}$$

$$I = \int p \int\_0^{(p-q)\lambda/p} f\_k(1-s) \left. \frac{\mathbf{d}s}{s} \right| \mathbf{d}\mu \tag{252}$$

$$=\int p\int\_{0}^{\lambda} f\_k \left(1 + \frac{(q-p)s'}{p}\right) \frac{\mathbf{d}s'}{\mathbf{s'}} \,\mathrm{d}\mu\tag{253}$$

$$I = \int\_0^\lambda \int pf\_k \left(\frac{r\_{s'}}{p}\right) \,\mathrm{d}\mu \,\,\frac{\mathrm{d}s'}{\mathrm{s'}}\,\tag{254}$$

= Z *λ* 0 *Dfk* (*Rs* 0k*P*) d*s* 0 *s* 0 , (255)

where the substitution *s* := (*p*−*q*)*s* 0 *p* is invoked in (253), and then (254) holds since *r s* 0 *p* = 1 + (*q*−*p*)*s* 0 *p* for *s* 0 ∈ [0, 1] (this follows from (21)) and by interchanging the order of the integrations.

#### *5.6. Proof of Corollary 5*

Combining (60) and (61) yields (58); furthermore, *f*0 : (0, ∞) → R, given by *f*0(*x*) = 1 *x* − 1 for all *x* > 0, is convex on (0, ∞) with *f*0(1) = 0. Hence, Theorem 4 holds for the selected functions { *fk* (·)} ∞ *k*=0 in (61), which therefore are all convex on (0, ∞) and vanish at 1. This proves that (59) holds for all *λ* ∈ [0, 1] and *k* ∈ {0, 1, . . .}. Since *f*0(*x*) = 1 *x* − 1 and *f*1(*x*) = − loge (*x*) for all *x* > 0 (see (60) and (61)), then, for every pair of probability measures *P* and *Q*:

$$D\_{f\_0}(P\|\|Q) = \chi^2(Q\|P), \quad D\_{f\_1}(P\|\|Q) = \frac{1}{\log e} \, D(Q\|P). \tag{256}$$

Finally, combining (59), for *k* = 0, together with (256), gives (22) as a special case.

#### *5.7. Proof of Theorem 5 and Corollary 6*

For an arbitrary measurable set E ⊆ X , we have from (62)

$$
\mu\_{\mathcal{C}}(\mathcal{E}) = \int\_{\mathcal{E}} \frac{1\_{\mathcal{C}}(\mathbf{x})}{\mu(\mathcal{C})} \, \mathbf{d}\mu(\mathbf{x}) \,. \tag{257}
$$

where 1C : X → {0, 1} is the indicator function of C ⊆ X , i.e., 1C (*x*) := 1{*x* ∈ C} for *x* ∈ X . Hence,

$$\frac{\mathrm{d}\mu\_{\mathcal{C}}}{\mathrm{d}\mu}(\mathbf{x}) = \frac{\mathbf{1}\_{\mathcal{C}}(\mathbf{x})}{\mu(\mathcal{C})}, \quad \forall \mathbf{x} \in \mathcal{X}, \tag{258}$$

and

$$D(\mu\_{\mathcal{C}} \| \mu) = \int\_{\mathcal{X}} f\left(\frac{\mathbf{d}\mu\_{\mathcal{C}}}{\mathbf{d}\mu}\right) \,\mathrm{d}\mu\tag{259}$$

$$=\int\_{\mathcal{C}} f\left(\frac{1}{\mu(\mathcal{C})}\right) \,\mathrm{d}\mu(\mathbf{x}) + \int\_{\mathcal{K}\backslash\mathcal{C}} f(0) \,\mathrm{d}\mu(\mathbf{x})\tag{260}$$

$$=\mu(\mathcal{C})\,f\left(\frac{1}{\mu(\mathcal{C})}\right) + \mu(\mathcal{X}\,\backslash\,\mathcal{C})\,f(0) \tag{261}$$

$$=\tilde{f}\left(\mu(\mathcal{C})\right) + \left(1 - \mu(\mathcal{C})\right)f(0),\tag{262}$$

where the last equality holds by the definition of *f*ein (63). This proves Theorem 5. Corollary 6 is next proved by first proving (67) for the Rényi divergence. For all *α* ∈ (0, 1) ∪ (1, ∞),

$$D\_{\mathfrak{a}}\left(\mu\_{\mathcal{C}}\|\mu\right) = \frac{1}{\mathfrak{a}-1} \log \int\_{\mathcal{X}} \left(\frac{\mathbf{d}\mu\_{\mathcal{C}}}{\mathbf{d}\mu}\right)^{\mathfrak{a}} \mathrm{d}\mu \tag{263}$$

$$\epsilon = \frac{1}{\alpha - 1} \log \int\_{\mathcal{C}} \left( \frac{1}{\mu(\mathcal{C})} \right)^{\alpha} \mathrm{d}\mu \tag{264}$$

$$=\frac{1}{\alpha - 1} \quad \log\left(\left(\frac{1}{\mu(\mathcal{C})}\right)^{\alpha}\mu(\mathcal{C})\right) \tag{265}$$

$$=\log\frac{1}{\mu(\mathcal{C})}.\tag{266}$$

The justification of (67) for *α* = 1 is due to the continuous extension of the order-*α* Rényi divergence at *α* = 1, which gives the relative entropy (see (13)). Equality (65) is obtained from (67) at *α* = 1. Finally, (66) is obtained by combining (15) and (67) with *α* = 2.

#### *5.8. Proof of Theorem 6*

(100) is an equivalent form of (27). From (91) and (100), for all *α* ∈ [0, 1],

$$\frac{1}{\log \text{e}} \, \mathcal{S}\_{\mathfrak{a}}(P \| \! \| Q) = \mathfrak{a} \, \frac{1}{\log \text{e}} \, \mathcal{K}\_{\mathfrak{a}}(P \| \! \| Q) + (1 - \mathfrak{a}) \, \frac{1}{\log \text{e}} \, \mathcal{K}\_{1-\mathfrak{a}}(Q \| \! \| P) \tag{267}$$

$$\mathfrak{a} = \mathfrak{a} \int\_{0}^{\mathfrak{a}} s D\_{\Phi\_{\mathfrak{s}}}(P \| Q) \, \mathrm{d}s + (1 - \mathfrak{a}) \int\_{0}^{1 - \mathfrak{a}} s D\_{\Phi\_{\mathfrak{s}}}(Q \| P) \, \mathrm{d}s \tag{268}$$

$$=\mathfrak{a}\int\_{0}^{\mathfrak{a}} s D\_{\Phi\_{\mathfrak{s}}}(P \| Q) \, \mathrm{d}s + (1 - \mathfrak{a}) \int\_{\mathfrak{a}}^{1} (1 - s) D\_{\Phi\_{\mathfrak{l} - \mathfrak{s}}}(Q \| P) \, \mathrm{d}s.\tag{269}$$

Regarding the integrand of the second term in (269), in view of (18), for all *s* ∈ (0, 1)

$$D\_{\Phi\_{1-s}}(Q \| P) = \frac{1}{(1-s)^2} \cdot \chi^2(Q \| (1-s)P + sQ) \tag{270}$$

$$\hat{\lambda} = \frac{1}{s^2} \cdot \chi^2 \left( P \parallel (1 - s)P + sQ \right) \tag{271}$$

$$=D\_{\Phi\_s}(P\|\|Q)\_\prime\tag{272}$$

where (271) readily follows from (9). Since we also have *Dφ*1 (*P*k*Q*) = *χ* 2 (*P*k*Q*) = *Dφ*0 (*Q*k*P*) (see (18)), it follows that

$$D\_{\Phi\_{1-s}}(Q \| P) = D\_{\Phi\_s}(P \| Q), \quad s \in [0, 1]. \tag{273}$$

By using this identity, we get from (269) that, for all *α* ∈ [0, 1]

$$\frac{1}{\log \text{e}} \mathcal{S}\_{\mathfrak{a}}(P||\mathcal{Q}) = \mathfrak{a} \int\_{0}^{\mathfrak{a}} s D\_{\mathfrak{G}\_{\mathfrak{s}}}(P||\mathcal{Q}) \, \text{ds} + (1 - \mathfrak{a}) \int\_{\mathfrak{a}}^{1} (1 - s) D\_{\mathfrak{G}\_{\mathfrak{s}}}(P||\mathcal{Q}) \, \text{ds} \tag{274}$$

$$=\int\_0^1 g\_a(s) \, D\_{\Phi}(P \| Q) \, \mathrm{d}s,\tag{275}$$

where the function *gα* : [0, 1] → R is defined in (102). This proves the integral identity (101).

The lower bounds in (103) and (104) hold since, if *f* : (0, ∞) → R is convex, continuously twice differentiable and strictly convex at 1, then

$$
\mu\_{\chi^2}(Q\_{X'}W\_{Y|X}) \le \mu\_f(Q\_{X'}W\_{Y|X})\_\prime \tag{276}
$$

(see, e.g., [46] [Proposition II.6.5] and [50] [Theorem 2]). Hence, this holds in particular for the *f*-divergences in (95) and (96) (since the required properties are satisfied by the parametric functions in (97) and (98), respectively). We next prove the upper bound on the contraction coefficients in (103) and (104) by relying on (100) and (101), respectively. In the setting of Definition 7, if *PX* 6= *QX*, then it follows from (100) that for *α* ∈ (0, 1],

$$\frac{K\_{\mathfrak{A}}(P\_{Y} \| Q\_{Y})}{K\_{\mathfrak{A}}(P\_{\mathfrak{X}} \| Q\_{\mathfrak{X}})} = \frac{\int\_{0}^{\mathfrak{A}} sD\_{\mathfrak{A}\_{s}}(P\_{Y} \| Q\_{Y}) \, \mathrm{d}s}{\int\_{0}^{\mathfrak{A}} sD\_{\mathfrak{A}\_{s}}(P\_{\mathfrak{X}} \| Q\_{\mathfrak{X}}) \, \mathrm{d}s} \tag{277}$$

$$\leq \frac{\int\_{0}^{\mathfrak{a}} s \, \mu\_{\phi\_{\mathfrak{s}}}(Q\_{\mathbf{X}\prime} \, W\_{\mathbf{Y}|\mathbf{X}}) \, D\_{\phi\_{\mathfrak{s}}}(P\_{\mathbf{X}} \| Q\_{\mathbf{X}}) \, \mathrm{ds}}{\int\_{0}^{\mathfrak{a}} s \, D\_{\phi\_{\mathfrak{s}}}(P\_{\mathbf{X}} \| Q\_{\mathbf{X}}) \, \mathrm{ds}} \tag{278}$$

$$\leq \sup\_{s \in (0, \mathfrak{a}]} \mu\_{\phi\_s}(Q\_{\mathbf{X}\prime} W\_{\mathbf{Y}|\mathbf{X}}).\tag{279}$$

Finally, taking the supremum of the left-hand side of (277) over all probability measures *PX* such that 0 < *Kα*(*PX*k*QX*) < ∞ gives the upper bound on *µkα* (*QX*, *WY*|*X*) in (103). The proof of the upper bound on *µsα* (*QX*, *WY*|*X*), for all *α* ∈ [0, 1], follows similarly from (101), since the function *gα*(·) as defined in (102) is positive over the interval (0, 1).

*Entropy* **2020**, *22*, 563

#### *5.9. Proof of Corollary 7*

The upper bounds in (106) and (107) rely on those in (103) and (104), respectively, by showing that

$$\sup\_{s \in (0,1]} \mu\_{\phi\_{\mathbb{S}}}(Q\_{\mathbf{X}\mathcal{I}} \mathcal{W}\_{\mathbf{Y}|\mathbf{X}}) \le \mu\_{\chi^2}(\mathcal{W}\_{\mathbf{Y}|\mathbf{X}}).\tag{280}$$

Inequality (280) is obtained as follows, similarly to the concept of the proof of [51] [Remark 3.8]. For all *s* ∈ (0, 1] and *PX* 6= *QX*,

$$\begin{split} \frac{D\_{\phi\_{\mathbb{S}}}(P\_{\mathcal{X}}\mathcal{W}\_{\mathcal{Y}|\mathcal{X}} \parallel Q\_{\mathcal{X}}\mathcal{W}\_{\mathcal{Y}|\mathcal{X}})}{D\_{\phi\_{\mathbb{S}}}(P\_{\mathcal{X}} \parallel Q\_{\mathcal{X}})} \\ = \frac{\chi^{2}(P\_{\mathcal{X}}\mathcal{W}\_{\mathcal{Y}|\mathcal{X}} \parallel (1-s)P\_{\mathcal{X}}\mathcal{W}\_{\mathcal{Y}|\mathcal{X}} + sQ\_{\mathcal{X}}\mathcal{W}\_{\mathcal{Y}|\mathcal{X}})}{\chi^{2}(P\_{\mathcal{X}} \parallel (1-s)P\_{\mathcal{X}} + sQ\_{\mathcal{X}})} \end{split} \tag{281}$$

$$0 \le \mu\_{\chi^2}((1-s)P\_X + sQ\_{X'}\,\mathcal{W}\_{Y|X})\tag{282}$$

$$1 \le \mu\_{\chi^2}(\mathcal{W}\_{Y|X})\_\prime \tag{283}$$

where (281) holds due to (18), and (283) is due to the definition in (105).

#### *5.10. Proof of Proposition 3*

The lower bound on the contraction coefficients in (108) and (109) is due to (276). The derivation of the upper bounds relies on [49] [Theorem 2.2], which states the following. Let *f* : [0, ∞) → R be a three–times differentiable, convex function with *f*(1) = 0, *f* 00(1) > 0, and let the function *z* : (0, ∞) → R defined as *z*(*t*) := *f*(*t*)−*f*(0) *t* , for all *t* > 0, be concave. Then,

$$
\mu\_f(\mathbf{Q}\_{X'}\mathcal{W}\_{\mathbf{Y}|X}) \le \frac{f'(1) + f(0)}{f''(1)} \cdot \mu\_{\chi^2}(\mathbf{Q}\_{X'}\mathcal{W}\_{\mathbf{Y}|X}).\tag{284}
$$

For *α* ∈ (0, 1], let *zα*,1 : (0, ∞) → R and *zα*,2 : (0, ∞) → R be given by

$$z\_{\mathfrak{a},1}(t) := \frac{k\_{\mathfrak{a}}(t) - k\_{\mathfrak{a}}(0)}{t}, \quad t > 0,\tag{285}$$

$$z\_{a,2}(t) := \frac{s\_a(t) - s\_a(0)}{t}, \quad t > 0,\tag{286}$$

with *kα* and *sα* in (97) and (98). Straightforward calculus shows that, for *α* ∈ (0, 1] and *t* > 0,

$$z\_{\frac{1}{\log e}} z\_{a,1}^{\prime\prime}(t) = -\frac{a^2 + 2a(1-a)t}{t^2 \left[a + (1-a)t\right]^2} < 0,\tag{287}$$

$$\begin{split} \frac{1}{t^{\frac{1}{\log \mathfrak{e}}}} z\_{n,2}^{\prime\prime}(t) &= -\frac{a^2 \left[ a + 2(1-a)t \right]}{t^2 \left[ a + (1-a)t \right]^2} \\ &- \frac{2(1-a)}{t^3} \left[ \log\_{\mathfrak{e}} \left( 1 + \frac{(1-a)t}{a} \right) - \frac{(1-a)t}{a + (1-a)t} - \frac{(1-a)^2 t^2}{2\left[ a + (1-a)t \right]^2} \right]. \end{split} \tag{288}$$

The first term on the right side of (288) is negative. For showing that the second term is also negative, we rely on the power series expansion loge (1 + *u*) = *u* − 1 2 *u* 2 + 1 3 *u* 3 − . . . for *u* ∈ (−1, 1]. Setting *u* := − *x* 1+*x* , for *x* > 0, and using Leibnitz theorem for alternating series yields

$$\log\_{\mathbf{e}}(1+\mathbf{x}) = -\log\_{\mathbf{e}}\left(1 - \frac{\mathbf{x}}{1+\mathbf{x}}\right) > \frac{\mathbf{x}}{1+\mathbf{x}} + \frac{\mathbf{x}^2}{2(1+\mathbf{x})^2}, \qquad \mathbf{x} > 0. \tag{289}$$

*Entropy* **2020**, *22*, 563

Consequently, setting *x* := (1−*α*)*t α* ∈ [0, ∞) in (289), for *t* > 0 and *α* ∈ (0, 1], proves that the second term on the right side of (288) is negative. Hence, *z* 00 *α*,1(*t*), *z* 00 *α*,2(*t*) < 0, so both *zα*,1, *zα*,2 : (0, ∞) → R are concave functions.

In view of the satisfiability of the conditions of [49] [Theorem 2.2] for the *f*-divergences with *f* = *kα* or *f* = *sα*, the upper bounds in (108) and (109) follow from (284), and also since

$$k\_{\mathfrak{a}}(\mathbf{0}) = \mathbf{0}, \qquad \qquad \qquad k\_{\mathfrak{a}}'(\mathbf{1}) = \mathfrak{a} \ \log \mathbf{e}, \qquad \qquad \qquad k\_{\mathfrak{a}}''(\mathbf{1}) = \mathfrak{a}^2 \ \log \mathbf{e}, \tag{290}$$

$$s\_{\mathfrak{a}}(0) = -(1 - \mathfrak{a}) \log \mathfrak{a}, \quad s\_{\mathfrak{a}}'(1) = (2\mathfrak{a} - 1) \log \mathfrak{e}, \quad s\_{\mathfrak{a}}''(1) = (1 - 3\mathfrak{a} + 3\mathfrak{a}^2) \log \mathfrak{e}.\tag{291}$$

#### *5.11. Proof of Proposition 4*

In view of (24), we get

$$\frac{D(P\_Y \| Q\_Y)}{D(P\_X \| Q\_X)} = \frac{\int\_0^1 \chi^2(P\_Y \| \, (1-s)P\_Y + sQ\_Y) \, \frac{ds}{s}}{\int\_0^1 \chi^2(P\_X \| \, (1-s)P\_X + sQ\_X) \, \frac{ds}{s}} \tag{292}$$

$$\leq \frac{\int\_{0}^{1} \mu\_{\chi^{2}}((1-s)P\_{X} + sQ\_{X\prime}, \mathcal{W}\_{Y|X}) \, \chi^{2}(P\_{X} \parallel (1-s)P\_{X} + sQ\_{X}) \, \frac{\mathrm{d}s}{\mathrm{s}}}{\int\_{0}^{1} \chi^{2}(P\_{X} \parallel (1-s)P\_{X} + sQ\_{X}) \, \frac{\mathrm{d}s}{\mathrm{s}}} \tag{293}$$

$$\leq \sup\_{s \in [0,1]} \mu\_{\chi^2}((1-s)P\_X + sQ\_{X\prime}, W\_{Y|X}).\tag{294}$$

In view of (119), the distributions of *Xs* and *Ys* , and since (1 − *s*)*PX* + *sQX WY*|*X* = (1 − *s*)*PY* + *sQY* holds for all *s* ∈ [0, 1], it follows that

$$\rho\_{\rm mf}(\mathbf{X}\_{\rm s};\mathbf{Y}\_{\rm s}) = \sqrt{\mu\_{\chi^2}((1-\mathbf{s})\mathbf{P}\_{\rm X} + \mathbf{s}\mathbf{Q}\_{\rm X\prime}\,\mathcal{W}\_{\rm Y\vert X})}, \quad \mathbf{s} \in [0,1], \tag{295}$$

which, from (292)–(295), implies that

$$\sup\_{s \in [0,1]} \rho\_{\mathbf{m}}(X\_{\mathbf{s}}; Y\_{\mathbf{s}}) \ge \sqrt{\frac{D(P\_Y \| Q\_Y)}{D(P\_{\mathbf{X}} \| Q\_{\mathbf{X}})}}.\tag{296}$$

Switching *PX* and *QX* in (292)–(294) and using the mapping *s* 7→ 1 − *s* in (294) gives (due to the symmetry of the maximal correlation)

$$\sup\_{s \in [0,1]} \rho\_{\mathbf{m}}(\mathbf{X}\_{\mathbf{s}}; \mathbf{Y}\_{\mathbf{s}}) \ge \sqrt{\frac{D(Q\_{\mathbf{Y}} \| P\_{\mathbf{Y}})}{D(Q\_{\mathbf{X}} \| P\_{\mathbf{X}})}},\tag{297}$$

and, finally, taking the maximal lower bound among those in (296) and (297) gives (120).

**Author Contributions:** Both coauthors contributed to this research work, and to the writing and proofreading of this article. The starting point of this work was in independent derivations of preliminary versions of Theorems 1 and 2 in two separate un-published works [24,25]. All authors have read and agreed to the published version of the manuscript.

**Funding:** This research received no external funding.

**Acknowledgments:** Sergio Verdú is gratefully acknowledged for a careful reading, and well-appreciated feedback on the submitted version of this paper.

**Conflicts of Interest:** The authors declare no conflict of interest.

#### **References**

1. Kullback, S.; Leibler, R.A. On information and sufficiency. *Ann. Math. Stat.* **1951**, *22*, 79–86. [CrossRef]

© 2020 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).

## *Article* **A Two-Moment Inequality with Applications to Rényi Entropy and Mutual Information**

#### **Galen Reeves 1,2**

1 Department of Electrical and Computer Engineering, Duke University, Durham, NC 27708, USA<EMAIL_ADDRESS>
2 Department of Statistical Science, Duke University, Durham, NC 27708, USA

Received: 14 September 2020; Accepted: 6 October 2020; Published: 1 November 2020

**Abstract:** This paper explores some applications of a two-moment inequality for the integral of the *r*th power of a function, where 0 < *r* < 1. The first contribution is an upper bound on the Rényi entropy of a random vector in terms of the two different moments. When one of the moments is the zeroth moment, these bounds recover previous results based on maximum entropy distributions under a single moment constraint. More generally, evaluation of the bound with two carefully chosen nonzero moments can lead to significant improvements with a modest increase in complexity. The second contribution is a method for upper bounding mutual information in terms of certain integrals with respect to the variance of the conditional density. The bounds have a number of useful properties arising from the connection with variance decompositions.

**Keywords:** information inequalities; mutual information; Rényi entropy; Carlson–Levin inequality

#### **1. Introduction**

The interplay between inequalities and information theory has a rich history, with notable examples including the relationship between the Brunn–Minkowski inequality and the entropy power inequality as well as the matrix determinant inequalities obtained from differential entropy [1]. In this paper, the focus is on a "two-moment" inequality that provides an upper bound on the integral of the *r*th power of a function. Specifically, if *f* is a nonnegative function defined on R*n* and *p*, *q*,*r* are real numbers satisfying 0 < *r* < 1 and *p* < 1/*r* − 1 < *q*, then

$$\left(\int f(\mathbf{x})^r \, d\mathbf{x}\right)^{\frac{1}{r}} \le \mathbb{C}\_{n, p, q, r} \left(\int \|\mathbf{x}\|^{np} f(\mathbf{x}) \, d\mathbf{x}\right)^{\frac{qr+r-1}{(q-p)r}} \left(\int \|\mathbf{x}\|^{nq} f(\mathbf{x}) \, d\mathbf{x}\right)^{\frac{1-r}{(q-p)r}},\tag{1}$$

where the best possible constant *Cn*,*p*,*q*,*r* is given exactly; see Propositions 2 and 3 ahead. The one-dimensional version of this inequality is a special case of the classical Carlson–Levin inequality [2–4], and the multidimensional version is a special case of a result presented by Barza et al. [5]. The particular formulation of the inequality used in this paper was derived independently in [6], where the proof follows from a direct application of Hölder's inequality and Jensen's inequality.

In the context of information theory and statistics, a useful property of the two-moment inequality is that it provides a bound on a nonlinear functional, namely the *r*-quasi-norm k · k*r* , in terms of integrals that are linear in *f* . Consequently, this inequality is well suited to settings where *f* is a mixture of simple functions whose moments can be evaluated. We note that this reliance on moments to bound a nonlinear functional is closely related to bounds obtained from variational characterizations such as the Donsker–Varadhan representation of Kullback divergence [7] and its generalizations to Rényi divergence [8,9].

The first application considered in this paper concerns the relationship between the entropy of a probability measure and its moments. This relationship is fundamental to the principle of maximum entropy, which originated in statistical physics and has since been applied to statistical inference problems [10]. It also plays a prominent role in information theory and estimation theory where the fact that the Gaussian distribution maximizes differential entropy under second moment constraints ([11], [Theorem 8.6.5]) plays a prominent role. Moment–entropy inequalities for Rényi entropy were studied in a series of works by Lutwak et al. [12–14], as well as related works by Costa et al. [15,16] and Johonson and Vignat [17], in which it is shown that, under a single moment constraint, Rényi entropy is maximized by a family of generalized Gaussian distributions. The connection between these moment–entropy inequalities and the Carlson–Levin inequality was noted recently by Nguyen [18].

In this direction, one of the contributions of this paper is a new family of moment–entropy inequalities. This family of inequalities follows from applying Inequality (1) in the setting where *f* is a probability density function, and thus there is a one-to-one correspondence between the integral of the *r*th power and the Rényi entropy of order *r*. In the special case where one of the moments is the zeroth moment, this approach recovers the moment–entropy inequalities given in previous work. More generally, the additional flexibility provided by considering two different moments can lead to stronger results. For example, in Proposition 6, it is shown that if *f* is the standard Gaussian density function defined on R*n* , then the difference between the Rényi entropy and the upper bound given by the two-moment inequality (equivalently, the ratio between the left- and right-hand sides of (1)) is bounded uniformly with respect to *n* under the following specification of the moments:

$$p\_n = \frac{1-r}{r} - \frac{1}{r} \sqrt{\frac{2(1-r)}{n+1}}, \qquad q\_n = \frac{1-r}{r} + \frac{1}{r} \sqrt{\frac{2(1-r)}{n+1}}.\tag{2}$$

Conversely, if one of the moments is restricted to be equal to zero, as is the case in the usual moment–entropy inequalities, then the difference between the Rényi entropy and the upper bound diverges with *n*.

The second application considered in this paper is the problem of bounding mutual information. In conjunction with Fano's inequality and its extensions, bounds on mutual information play a prominent role in establishing minimax rates of statistical estimation [19] as well as the information-theoretic limits of detection in high-dimensional settings [20]. In many cases, one of the technical challenges is to provide conditions under which the dependence between the observations and an underlying signal or model parameters converges to zero in the limit of high dimension.

This paper introduces a new method for bounding mutual information, which can be described as follows. Let *PX*,*Y* be a probability measure on X × Y such that *PY*|*X*=*x* and *PY* have densities *f*(*y* | *x*) and *f*(*y*) with respect to the Lebesgue measure on R*n* . We begin by showing that the mutual information between *X* and *Y* satisfies the upper bound

$$I(X;Y) \le \int \sqrt{\mathsf{Var}(f(y \mid X))} \,\mathrm{d}y,\tag{3}$$

where Var(*p*(*y* | *X*)) = R (*f*(*y* | *x*) − *f*(*y*)) 2 d*PX*(*x*) is the variance of *f*(*y* | *X*); see Proposition 8 ahead. In view of (3), an application of the two-moment Inequality (1) with *r* = 1/2 leads to an upper bound with respect to the moments of the variance of the density:

$$\int \|y\|\|^{\text{ns}} \mathsf{Var}(f(y \mid X)) \,\mathrm{d}y \tag{4}$$

where this expression is evaluated at *s* ∈ {*p*, *q*} with *p* < 1 < *q*. A useful property of this bound is that the integrated variance is quadratic in *PX*, and thus Expression (4) can be evaluated by swapping the integration over *y* and with the expectation of over two independent copies of *X*. For example, when *PX*,*Y* is a Gaussian scale mixture, this approach provides closed-form upper bounds in terms of

the moments of the Gaussian density. An early version of this technique is used to prove Gaussian approximations for random projections [21] arising in the analysis of a random linear estimation problem appearing in wireless communications and compressed sensing [22,23].

#### **2. Moment Inequalities**

Let *L p* (*S*) be the space of Lebesgue measurable functions from *S* to R whose *p*th power is absolutely integrable, and for *p* 6= 0, define

$$\|f\|\_{p} := \left(\int\_{S} |f(\mathfrak{x})|^{p} \,\mathrm{d}x\right)^{1/p}.$$

Recall that k · k*p* is a norm for *p* ≥ 1 but only a quasi-norm for 0 < *p* < 1 because it does not satisfy the triangle inequality. The *s*th moment of *f* is defined as

$$\mathcal{M}\_s(f) := \int\_S \|\mathbf{x}\|^s \, |f(\mathbf{x})| \, \mathbf{d}x\_s$$

where k · k denotes the standard Euclidean norm on vectors.

The two-moment Inequality (1) can be derived straightforwardly using the following argument. For *r* ∈ (0, 1), the mapping *f* 7→ k *f* k*r* is concave on the subset of nonnegative functions and admits the variational representation

$$\|f\|\_{r} = \inf \left\{ \frac{\|fg\|\_{1}}{\|g\|\_{r^\*}} : g \in L^{r^\*} \right\},\tag{5}$$

where *r* ∗ = *r*/(*r* − 1) ∈ (−∞, 0) is the Hölder conjugate of *r*. Consequently, each *g* ∈ *L r* ∗ leads to an upper bound on k *f* k*r* . For example, if *f* has bounded support *S*, choosing *g* to be the indicator function of *S* leads to the basic inequality k *f* k*r* ≤ (Vol(*S*))(1−*r*)/*r*k *f* k1. The upper bound on k *f* k*r* given in Inequality (1) can be obtained by restricting the minimum in Expression (5) to the parametric class of functions of the form *g*(*x*) = *ν*1k*x*k *np* + *ν*2 k*x*k *nq* with *ν*1, *ν*2 > 0 and then optimizing over the parameters (*ν*1, *ν*2). Here, the constraints on *p*, *q* are necessary and sufficient to ensure that *g* ∈ *L r* ∗ (R*n* ).

In the following sections, we provide a more detailed derivation, starting with the problem of maximizing k *f* k*r* under multiple moment constraints and then specializing to the case of two moments. For a detailed account of the history of the Carlson type inequalities as well as some further extensions, see [4].

#### *2.1. Multiple Moments*

Consider the following optimization problem:

$$\begin{aligned} \text{maximize} \quad & \|f\|\_{r} \\ \text{subject to} \quad & f(\mathbf{x}) \ge 0 \qquad \text{for all } \mathbf{x} \in \mathcal{S} \\ & \mathcal{M}\_{\mathbf{s}\_i}(f) \le m\_i \quad \text{for } 1 \le i \le k. \end{aligned}$$

For *r* ∈ (0, 1), this is a convex optimization problem because k · k*r r* is concave and the moment constraints are linear. By standard theory in convex optimization (e.g., [24]), it can be shown that if the problem is feasible and the maximum is finite, then the maximizer has the form

$$f^\*(\mathfrak{x}) = \left(\sum\_{i=1}^k \nu\_i^\* \, ||\mathfrak{x}||^{s\_i}\right)^{\frac{1}{r-1}}, \quad \text{for all } \mathfrak{x} \in \mathbb{S}.$$

The parameters *ν* ∗ 1 , · · · , *ν* ∗ *k* are nonnegative and the *i*th moment constraint holds with equality for all *i* such that *ν* ∗ *i* is strictly positive—that is, *ν* ∗ *i* > 0 =⇒ *µsi* (*f* ∗ ) = *mi* . Consequently, the maximum can be expressed in terms of a linear combination of the moments:

$$\|f^\*\|\_r^r = \|(f^\*)^r\|\_1 = \|f^\*(f^\*)^{r-1}\|\_1 = \sum\_{i=1}^k \nu\_i^\* m\_i.$$

For the purposes of this paper, it is useful to consider a relative inequality in terms of the moments of the function itself. Given a number 0 < *r* < 1 and vectors *s* ∈ R*k* and *ν* ∈ R*k* +, the function *cr*(*ν*,*s*) is defined according to

$$c\_r(\nu, s) = \left( \int\_0^\infty \left( \sum\_{i=1}^k \nu\_i \, \mathbf{x}^{s\_i} \right)^{-\frac{r}{1-r}} \, \mathbf{d}x \right)^{\frac{1-r}{r}}\Big|$$

if the integral exists. Otherwise, *cr*(*ν*,*s*) is defined to be positive infinity. It can be verified that *cr*(*ν*,*s*) is finite provided that there exists *i*, *j* such that *νi* and *νj* are strictly positive and *si* < (1 − *r*)/*r* < *sj* .

The following result can be viewed as a consequence of the constrained optimization problem described above. We provide a different and very simple proof that depends only on Hölder's inequality.

**Proposition 1.** *Let f be a nonnegative Lebesgue measurable function defined on the positive reals* R+*. For any number* 0 < *r* < 1 *and vectors s* ∈ R*k and ν* ∈ R*k* +*, we have*

$$\|f\|\_{r} \le c\_{r}(\nu, s) \sum\_{i=1}^{k} \nu\_{i} \mathcal{M}\_{s\_{i}}(f).$$

**Proof.** Let *g*(*x*) = ∑ *k i*=1 *νi x si* . Then, we have

$$\|f\|\_{r}^{r} = \|g^{-r}(f\mathfrak{g})^{r}\|\_{1} \le \|\mathfrak{g}^{-r}\|\_{\frac{1}{1-r}}\|(\mathfrak{g}f)^{r}\|\_{\frac{1}{r}} = \|g^{\frac{-r}{1-r}}\|\_{1}^{1-r}\|\mathfrak{g}f\|\_{1}^{r} = \left(c\_{r}(\nu\_{\prime}\mathfrak{s})\sum\_{i=1}^{k}\nu\_{i}\mathcal{M}\_{\mathfrak{s}\_{i}}(f)\right)^{r}.$$

where the second step is Hölder's inequality with conjugate exponents 1/(1 − *r*) and 1/*r*.

#### *2.2. Two Moments*

For *a*, *b* > 0, the beta function B(*a*, *b*) and gamma function Γ(*a*) are given by

$$\begin{aligned} \mathbf{B}(a,b) &= \int\_0^1 t^{a-1}(1-t)^{b-1} \, \mathbf{d}t \\ \Gamma(a) &= \int\_0^\infty t^{a-1} e^{-t} \, \mathbf{d}t \end{aligned}$$

and satisfy the relation B(*a*, *b*) = Γ(*a*)Γ(*b*)/Γ(*a* + *b*), *a*, *b* > 0. To lighten the notation, we define the normalized beta function

$$
\widetilde{B}(a,b) = \mathbf{B}(a,b)(a+b)^{a+b}a^{-a}b^{-b}.\tag{6}
$$

Properties of these functions are provided in Appendix A.

The next result follows from Proposition 1 for the case of two moments.

**Proposition 2.** *Let f be a nonnegative Lebesgue measurable function defined on* [0, ∞)*. For any numbers p*, *q*,*r with* 0 < *r* < 1 *and p* < 1/*r* − 1 < *q,*

$$||f||\_r \le \left[\psi\_r(p,q)\right]^{\frac{1-r}{r}} \left[\mathcal{M}\_p(f)\right]^\lambda [\mathcal{M}\_q(f)]^{1-\lambda}.$$

*where λ* = (*q* + 1 − 1/*r*)/(*q* − *p*) *and*

$$
\psi\_r(p,q) = \frac{1}{(q-p)} \widetilde{\mathcal{B}}\left(\frac{r\lambda}{1-r}, \frac{r(1-\lambda)}{1-r}\right),\tag{7}
$$

*where B*e(·, ·) *is defined in Equation* (6)*.*

**Proof.** Letting *s* = (*p*, *q*) and *ν* = (*γ* 1−*λ* , *γ* −*λ* ) with *λ* > 0, we have

$$\left[c\_r(\nu, s)\right]^{\frac{r}{1-r}} = \int\_0^\infty \left(\gamma^{1-\lambda} x^p + \gamma^{-\lambda} x^q\right)^{-\frac{r}{1-r}} \,\mathrm{d}x.$$

Making the change of variable *x* 7→ (*γu*) 1 *q*−*p* leads to

$$[c\_r(\nu, s)]^{\frac{r}{1-r}} = \frac{1}{q-p} \int\_0^\infty \frac{u^{b-1}}{(1+u)^{a+b}} \, \mathrm{d}u = \frac{\mathrm{B}\,(a, b)}{q-p} \, \mathrm{d}u$$

where *a* = *r* 1−*r λ* and *b* = *r* 1−*r* (1 − *λ*) and the second step follows from recognizing the integral representation of the beta function given in Equation (A3). Therefore, by Proposition 1, the inequality

$$||f||\_r \le \left(\frac{\mathsf{B}\left(a,b\right)}{q-p}\right)^{\frac{1-r}{r}} \left(\gamma^{1-\lambda}\mathcal{M}\_p(f) + \gamma^{-\lambda}\mathcal{M}\_q(f)\right)\lambda$$

holds for all *γ* > 0. Evaluating this inequality with

$$\gamma = \frac{\lambda \, \mathcal{M}\_{\emptyset}(f)}{(1 - \lambda)\mathcal{M}\_{p}(f)},$$

leads to the stated result.

The special case *r* = 1/2 admits the simplified expression

$$
\psi\_{1/2}(p,q) = \frac{\pi \lambda^{-\lambda} (1-\lambda)^{-(1-\lambda)}}{(q-p)\sin(\pi\lambda)},\tag{8}
$$

where we have used Euler's reflection formula for the beta function ([25], [Theorem 1.2.1]).

Next, we consider an extension of Proposition 2 for functions defined on R*n* . Given any measurable subset *S* of R*n* , we define

$$
\omega(\mathcal{S}) = \text{Vol}(\mathcal{B}^n \cap \text{cone}(\mathcal{S})),
\tag{9}
$$

where *B n* = {*u* ∈ R*n* : k*u*k ≤ 1} is the *n*-dimensional Euclidean ball of radius one and

$$\text{cone}(\mathcal{S}) = \{ \mathfrak{x} \in \mathbb{R}^n \, : \, t\mathfrak{x} \in \mathcal{S} \text{ for some } t > 0 \}.$$

The function *ω*(*S*) is proportional to the surface measure of the projection of *S* on the Euclidean sphere and satisfies

$$
\omega(\mathcal{S}) \le \omega(\mathbb{R}^n) = \frac{\pi^{\frac{\mu}{2}}}{\Gamma(\frac{\mu}{2} + 1)},\tag{10}
$$

for all *S* ⊆ R*n* . Note that *ω*(R+) = 1 and *ω*(R) = 2. **Proposition 3.** *Let f be a nonnegative Lebesgue measurable function defined on a subset S of* R*n . For any numbers p*, *q*,*r with* 0 < *r* < 1 *and p* < 1/*r* − 1 < *q,*

$$\|f\|\_{r} \le \left[\omega(\mathcal{S})\,\psi\_r(p,q)\right]^{\frac{1-r}{r}}\left[\mathcal{M}\_{np}(f)\right]^\lambda \left[\mathcal{M}\_{nq}(f)\right]^{1-\lambda}$$

*where λ* = (*q* + 1 − 1/*r*)/(*q* − *p*) *and ψr*(*p*, *q*) *is given by Equation* (7)*.*

**Proof.** Let *f* be extended to R*n* using the rule *f*(*x*) = 0 for all *x* outside of *S* and let *g* : R+ → R+ be defined according to

$$\operatorname{g}(y) = \frac{1}{n} \int\_{\mathbb{S}^{n-1}} f(y^{1/n} u) \operatorname{d} \sigma(u),$$

where S *n*−1 = {*u* ∈ R*n* : k*u*k = 1} is the Euclidean sphere of radius one and *σ*(*u*) is the surface measure of the sphere. In the following, we will show that

$$\|f\|\_{r} \le \left(\omega(\mathcal{S})\right)^{\frac{1-r}{r}} \|g\|\_{r} \tag{11}$$

$$\mathcal{M}\_{\rm ns}(f) = \mathcal{M}\_{\rm s}(\mathbf{g}).\tag{12}$$

Then, the stated inequality then follows from applying Proposition 2 to the function *g*.

To prove Inequality (11), we begin with a transformation into polar coordinates:

$$\|f\|\|\_{r}^{r} = \int\_{0}^{\infty} \int\_{\mathbb{S}^{n-1}} |f(tu)|^{r} \, t^{n-1} \, \mathrm{d}\sigma(u) \, \mathrm{d}t. \tag{13}$$

Letting **1**cone(*S*) (*x*) denote the indicator function of the set cone(*S*), the integral over the sphere can be bounded using:

$$\begin{split} \int\_{\mathbb{S}^{n-1}} |f(tu)|^{r} \, \mathrm{d}\sigma(u) &= \int\_{\mathbb{S}^{n-1}} \mathbf{1}\_{\mathrm{cone}(\mathbb{S})}(u) \, |f(tu)|^{r} \, \mathrm{d}\sigma(u) \\ &\stackrel{(a)}{\leq} \left( \int\_{\mathbb{S}^{n-1}} \mathbf{1}\_{\mathrm{cone}(\mathbb{S})}(u) \, \mathrm{d}\sigma(u) \right)^{1-r} \left( \int\_{\mathbb{S}^{n-1}} |f(tu)| \, \mathrm{d}\sigma(u) \right)^{r} \\ &\stackrel{(b)}{=} n \, (\omega(\mathbb{S}))^{1-r} \, \mathrm{g}^{r}(t^{n}), \end{split} \tag{14}$$

where: (a) follows from Hölder's inequality with conjugate exponents 1 1−*r* and 1 *r* , and (b) follows from the definition of *g* and the fact that

$$\begin{split} \omega(\mathcal{S}) &= \int\_0^1 \int\_{\mathbb{S}^{n-1}} \mathbf{1}\_{\text{cone}(\mathcal{S})} (\boldsymbol{\mu}) \, t^{n-1} \, \mathrm{d}\sigma(\boldsymbol{\mu}) \, \mathrm{d}t \\ &= \frac{1}{n} \int\_{\mathbb{S}^{n-1}} \mathbf{1}\_{\text{cone}(\mathcal{S})} (\boldsymbol{\mu}) \, \mathrm{d}\sigma(\boldsymbol{\mu}). \end{split}$$

Plugging Inequality (14) back into Equation (13) and then making the change of variable *t* → *y* 1 *n* yields

$$\|f\|\_{r}^{r} \le n \left(\omega(\mathcal{S})\right)^{1-r} \int\_{0}^{\infty} \mathcal{g}^{r}(t^{n}) t^{n-1} \, \mathbf{d}t = \left(\omega(\mathcal{S})\right)^{1-r} \|\mathcal{g}\|\_{r}^{r}.$$

The proof of Equation (12) follows along similar lines. We have

$$\begin{split} \mathcal{M}\_{\text{ns}}(f) & \stackrel{(a)}{=} \int\_{0}^{\infty} \int\_{\mathbb{S}^{n-1}} t^{\text{ns}} f(tu) \, t^{n-1} \, \mathrm{d}\sigma(u) \, \mathrm{d}t \\ & \stackrel{(b)}{=} \frac{1}{n} \int\_{0}^{\infty} \int\_{\mathbb{S}^{n-1}} y^{s} f(y^{\frac{1}{n}} u) \, \mathrm{d}\sigma(u) \, \mathrm{d}y \\ & = \mathcal{M}\_{\text{s}}(g) \end{split}$$

where (a) follows from a transformation into polar coordinates and (b) follows from the change of variable *t* 7→ *y* 1 *n* .

Having established Inequality (11) and Equation (12), an application of Proposition 2 completes the proof.

#### **3. Rényi Entropy Bounds**

Let *X* be a random vector that has a density *f*(*x*) with respect to the Lebesgue measure on R*n* . The differential Rényi entropy of order *r* ∈ (0, 1) ∪ (1, ∞) is defined according to [11]:

$$h\_r(\mathbf{X}) = \frac{1}{1-r} \log \left( \int\_{\mathbb{R}^n} f^r(\mathbf{x}) \, d\mathbf{x} \right).$$

Throughout this paper, it is assumed that the logarithm is defined with respect to the natural base and entropy is measured in nats. The Rényi entropy is continuous and nonincreasing in *r*. If the support set *S* = {*x* ∈ R*n* : *f*(*x*) > 0} has finite measure, then the limit as *r* converges to zero is given by *h*0(*X*) = log Vol(*S*). If the support does not have finite measure, then *hr*(*X*) increases to infinity as *r* decreases to zero. The case *r* = 1 is given by the Shannon differential entropy:

$$h\_1(\mathbf{X}) = -\int\_S f(\mathbf{x}) \log f(\mathbf{x}) \, \mathrm{d}x.$$

Given a random variable *X* that is not identical to zero and numbers *p*, *q*,*r* with 0 < *r* < 1 and *p* < 1/*r* − 1 < *q*, we define the function

$$L\_r(X;p,q) = \frac{r\lambda}{1-r}\log\mathbb{E}\left[|X|^p\right] + \frac{r(1-\lambda)}{1-r}\log\mathbb{E}\left[|X|^q\right],$$

where *λ* = (*q* + 1 − 1/*r*)/(*q* − *p*).

The next result, which follows directly from Proposition 3, provides an upper bound on the Rényi entropy.

**Proposition 4.** *Let X be a random vector with a density on* R*n . For any numbers p*, *q*,*r with* 0 < *r* < 1 *and p* < 1/*r* − 1 < *q, the Rényi entropy satisfies*

$$h\_r(X) \le \log \omega(S) + \log \psi\_r(p, q) + L\_r(\|X\|^\eta; p, q), \tag{15}$$

*where ω*(*S*) *is defined in Equation* (9) *and ψr*(*p*, *q*) *is defined in Equation* (7)*.*

**Proof.** This result follows immediately from Proposition 3 and the definition of Rényi entropy.

The relationship between Proposition 4 and previous results depends on whether the moment *p* is equal to zero:

*Entropy* **2020**, *22*, 1244

The contribution of two-moment inequalities is that they lead to tighter bounds. To quantify the tightness, we define ∆*r*(*X*; *p*, *q*) to be the gap between the right-hand side and left-hand side of Inequality (15) corresponding to the pair (*p*, *q*)—that is,

$$
\Delta\_r(X; p, q) = \log \omega(S) + \log \psi\_r(p, q) + L\_r(||X||^n; p, q) - h\_r(X).
$$

The gaps corresponding to the optimal two-moment and one-moment inequalities are defined according to

$$
\Delta\_r(X) = \inf\_{p,q} \Delta\_r(X; p, q)
$$

$$
\widetilde{\Delta}\_r(X) = \inf\_q \Delta\_r(X; 0, q).
$$

#### *3.1. Some Consequences of These Bounds*

By Lyapunov's inequality, the mapping *s* 7→ 1 *s* logE [|*X*| *s* ] is nondecreasing on [0, ∞), and thus

$$L\_r(X; p, q) \le L\_r(X; 0, q) = \frac{1}{q} \log \mathbb{E}\left[|X|^q\right], \quad p \ge 0. \tag{16}$$

In other words, the case *p* = 0 provides an upper bound on *Lr*(*X*; *p*, *q*) for nonnegative *p*. Alternatively, we also have the lower bound

$$L\_r(X; p, q) \ge \frac{r}{1 - r} \log \mathbb{E}\left[|X|^{\frac{1 - r}{r}}\right],\tag{17}$$

which follows from the convexity of logE [|*X*| *s* ].

A useful property of *Lr*(*X*; *p*, *q*) is that it is additive with respect to the product of independent random variables. Specifically, if *X* and *Y* are independent, then

$$L\_r(XY; p, q) = L\_r(X; p, q) + L\_r(Y; p, q). \tag{18}$$

One consequence is that multiplication by a bounded random variable cannot increase the Rényi entropy by an amount that exceeds the gap of the two-moment inequality with nonnegative moments.

**Proposition 5.** *Let Y be a random vector on* R*n with finite Rényi entropy of order* 0 < *r* < 1*, and let X be an independent random variable that satisfies* 0 < *X* ≤ *t. Then,*

$$h\_r(XY) \le h\_r(tY) + \Delta\_r(Y; p, q)\_{rt}$$

*for all* 0 < *p* < 1/*r* − 1 < *q.*

**Proof.** Let *Z* = *XY* and let *SZ* and *SY* denote the support sets of *Z* and *Y*, respectively. The assumption that *X* is nonnegative means that cone(*SZ*) = cone(*SY*). We have

$$h\_r(Z) \stackrel{(a)}{\leq} \log \omega(S\_Z) + \log \psi\_r(p\_\prime q) + L\_r(||Z||^n; p, q),$$

$$\stackrel{(b)}{=} h\_r(Y) + L\_r(|X|^n; p; q) + \Delta\_r(Y; p, q)$$

$$\stackrel{(c)}{\leq} h\_r(Y) + n \log t + \Delta\_r(Y; p, q),$$

where (a) follows from Proposition 4, (b) follows from Equation (18) and the definition of ∆*r*(*Y*; *p*, *q*), and (c) follows from Inequality (16) and the assumption |*X*| ≤ *t*. Finally, recalling that *hr*(*tY*) = *hr*(*Y*) + *n* log *t* completes the proof.

#### *3.2. Example with Log-Normal Distribution*

If *W* ∼ N (*µ*, *σ* 2 ), then the random variable *X* = exp(*W*) has a log-normal distribution with parameters (*µ*, *σ* 2 ). The Rényi entropy is given by

$$h\_r(\mathbf{X}) = \mu + \frac{1}{2} \left( \frac{1 - r}{r} \right) \sigma^2 + \frac{1}{2} \log(2\pi r^{\frac{1}{r - 1}} \sigma^2) \mu$$

and the logarithm of the *s*th moment is given by

$$\log \mathbb{E}\left[|X|^s\right] = \mu \mathbf{s} + \frac{1}{2}\sigma^2 s^2.$$

With a bit of work, it can be shown that the gap of the optimal two-moment inequality does not depend on the parameters (*µ*, *σ* 2 ) and is given by

$$\Delta\_{\mathcal{V}}(X) = \log\left(\widetilde{\mathcal{B}}\left(\frac{r}{2(1-r)}, \frac{r}{2(1-r)}\right) \sqrt{\frac{r}{4(1-r)}}\right) + \frac{1}{2} - \frac{1}{2}\log(2\pi r^{\frac{1}{r-1}}).\tag{19}$$

The details of this derivation are given in Appendix B.1. Meanwhile, the gap of the optimal one-moment inequality is given by

$$\widetilde{\Delta}\_{\mathbf{r}}(X) = \inf\_{q} \left[ \log \left( \widetilde{\mathbf{B}} \left( \frac{r}{1-r} - \frac{1}{q}, \frac{1}{q} \right) \frac{1}{q} \right) + \frac{1}{2} q \sigma^{2} \right] - \frac{1}{2} \left( \frac{1-r}{r} \right) \sigma^{2} - \frac{1}{2} \log(2 \pi r^{\frac{1}{r-1}} \sigma^{2}). \tag{20}$$

The functions ∆*r*(*X*) and ∆e*r*(*X*) are illustrated in Figure 1 as a function of *r* for various *σ* 2 . The function ∆*r*(*X*) is bounded uniformly with respect to *r* and converges to zero as *r* increases to one. The tightness of the two-moment inequality in this regime follows from the fact that the log-normal distribution maximizes Shannon entropy subject to a constraint on E [log *X*]. By contrast, the function ∆e*r*(*X*) varies with the parameter *σ* 2 . For any fixed *r* ∈ (0, 1), it can be shown that ∆e*r*(*X*) increases to infinity if *σ* 2 converges to zero or infinity.

**Figure 1.** Comparison of upper bounds on Rényi entropy in nats for the log-normal distribution as a function of the order *r* for various *σ* 2 .

#### *3.3. Example with Multivariate Gaussian Distribution*

Next, we consider the case where *Y* ∼ N (0, *In*) is an *n*-dimensional Gaussian vector with mean zero and identity covariance. The Rényi entropy is given by

$$h\_r(Y) = \frac{n}{2} \log(2\pi r^{\frac{1}{r-1}}),$$

and the *s*th moment of the magnitude k*Y*k is given by

$$\mathbb{E}\left[||Y||^s\right] = \frac{2^{\frac{s}{2}}\Gamma(\frac{n+s}{2})}{\Gamma(\frac{n}{2})}.$$

The next result shows that as the dimension *n* increases, the gap of the optimal two-moment inequality converges to the gap for the log-normal distribution. Moreover, for each *r* ∈ (0, 1), the following choice of moments is optimal in the large-*n* limit:

$$p\_n = \frac{1-r}{r} - \frac{1}{r} \sqrt{\frac{2(1-r)}{n+1}}, \qquad q\_n = \frac{1-r}{r} + \frac{1}{r} \sqrt{\frac{2(1-r)}{n+1}}.\tag{21}$$

The proof is given in Appendix B.3.

**Proposition 6.** *If Y* ∼ N (0, *In*)*, then, for each r* ∈ (0, 1)*,*

$$\lim\_{n \to \infty} \Delta\_r(Y) = \lim\_{n \to \infty} \Delta\_r(Y; p\_n, q\_n) = \Delta\_r(X)\_{\prime}$$

*where X has a log-normal distribution and* (*pn*, *qn*) *are given by* (21)*.*

Figure 2 provides a comparison of ∆*r*(*Y*), ∆*r*(*Y*; *pn*, *qn*), and ∆e*r*(*Y*) as a function of *n* for *r* = 0.1. Here, we see that both ∆*r*(*Y*) and ∆*r*(*Y*; *pn*, *qn*) converge rapidly to the asymptotic limit given by the gap of the log-normal distribution. By contrast, the gap of the optimal one-moment inequality ∆e*r*(*Y*) increases without bound.

**Figure 2.** Comparison of upper bounds on Rényi entropy in nats for the multivariate Gaussian distribution N (0, *In*) as a function of the dimension *n* with *r* = 0.1. The solid black line is the gap of the optimal two-moment inequality for the log-normal distribution.

#### *3.4. Inequalities for Differential Entropy*

Proposition 4 can also be used to recover some known inequalities for differential entropy by considering the limiting behavior as *r* converges to one. For example, it is well known that the differential entropy of an *n*-dimensional random vector *X* with finite second moment satisfies

$$h(X) \le \frac{1}{2} \log \left( 2\pi e \mathbb{E} \left[ \frac{1}{n} \|X\|^2 \right] \right),\tag{22}$$

with equality if and only if the entries of *X* are i.i.d. zero-mean Gaussian. A generalization of this result in terms of an arbitrary positive moment is given by

$$h(X) \le \log \frac{\Gamma\left(\frac{n}{s} + 1\right)}{\Gamma\left(\frac{n}{2} + 1\right)} + \frac{n}{2} \log \pi + \frac{n}{s} \log \left(es \operatorname{E}\left[\frac{1}{n} \|X\|^s\right]\right),\tag{23}$$

for all *s* > 0. Note that Inequality (22) corresponds to the case *s* = 2.

Inequality (23) can be proved as an immediate consequence of Proposition 4 and the fact that *hr*(*X*) is nonincreasing in *r*. Using properties of the beta function given in Appendix A, it is straightforward to verify that

$$\lim\_{r \to 1} \psi\_r(0, q) = (e \, q)^{\frac{1}{q}} \, \Gamma \left( \frac{1}{q} + 1 \right), \quad \text{for all } q > 0.$$

Combining this result with Proposition 4 and Inequality (16) leads to

$$h(X) \le \log \omega(S) + \log \Gamma\left(\frac{1}{q} + 1\right) + \frac{1}{q} \log \left(eq\mathbb{E}\left[||X||^{nq}\right]\right).$$

Using Inequality (10) and making the substitution *s* = *nq* leads to Inequality (23).

Another example follows from the fact that the log-normal distribution maximizes the differential entropy of a positive random variable *X* subject to constraints on the mean and variance of log(*X*), and hence

$$h(X) \le \mathbb{E}\left[\log(X)\right] + \frac{1}{2}\log\left(2\pi e \mathsf{Var}(\log(X))\right),\tag{24}$$

with equality if and only if *X* is log-normal. In Appendix B.4, it is shown how this inequality can be proved using our two-moment inequalities by studying the behavior as both *p* and *q* converge to zero as *r* increases to one.

#### **4. Bounds on Mutual Information**

#### *4.1. Relative Entropy and Chi-Squared Divergence*

Let *P* and *Q* be distributions defined on a common probability space that have densities *p* and *q* with respect to a dominating measure *µ*. The relative entropy (or Kullback–Leibler divergence) is defined according to

$$D\left(P\parallel Q\right) = \int p \log \left(\frac{p}{q}\right) \,\mathrm{d}\mu\_{\prime}$$

and the chi-squared divergence is defined as

$$\chi^2(P \parallel Q) = \int \frac{\left(p - q\right)^2}{q} \mathbf{d}\mu.$$

Both of these divergences can be seen as special cases of the general class of *f*-divergence measures and there exists a rich literature on comparisons between different divergences [8,26–32]. The chi-squared divergence can also be viewed as the squared *L*2 distance between *p*/ √*q* and √*q*. The chi-square can also be interpreted as the first non-zero term in the power series expansion of the relative entropy ([26], [Lemma 4]). More generally, the chi-squared divergence provides an upper bound on the relative entropy via

$$D\left(P\|\|Q\right) \le \log(1 + \chi^2(P\|\|Q)).\tag{25}$$

The proof of this inequality follows straightforwardly from Jensen's inequality and the concavity of the logarithm; see [27,31,32] for further refinements.

Given a random pair (*X*,*Y*), the mutual information between *X* and *Y* is defined according to

$$I(X;Y) = D\left(P\_{X,Y} \parallel P\_X P\_Y\right).$$

From Inequality (25), we see that the mutual information can always be upper bounded using

$$I(X;Y) \le \log(1 + \chi^2(P\_{X,Y}||P\_X P\_Y)).\tag{26}$$

The next section provides bounds on the mutual information that can improve upon this inequality.

#### *4.2. Mutual Information and Variance of Conditional Density*

Let (*X*,*Y*) be a random pair such that the conditional distribution of *Y* given *X* has a density *fY*|*X*(*y*|*x*) with respect to the Lebesgue measure on R*n* . Note that the marginal density of *Y* is given by *fY*(*y*) = E h *fY*|*X*(*y*|*X*) i . To simplify notation, we will write *f*(*y*|*x*) and *f*(*y*) where the subscripts are implicit. The support set of *Y* is denoted by *SY*.

The measure of the dependence between *X* and *Y* that is used in our bounds can be understood in terms of the variance of the conditional density. For each *y*, the conditional density *f*(*y*|*X*) evaluated with a random realization of *X* is a random variable. The variance of this random variable is given by

$$\mathsf{Var}(f(y|X)) = \mathbb{E}\left[ (f(y|X) - f(y))^2 \right],\tag{27}$$

where we have used the fact that the marginal density *f*(*y*) is the expectation of *f*(*y*|*X*). The *s*th moment of the variance of the conditional density is defined according to

$$V\_s(Y|X) = \int\_{S\_Y} \|y\|^s \mathsf{Var}(f(y|X)) \,\mathrm{d}y.\tag{28}$$

The variance moment *Vs*(*Y*|*X*) is nonnegative and equal to zero if and only if *X* and *Y* are independent.

The function *κ*(*t*) is defined according to

$$\kappa(t) = \sup\_{u \in (0,\infty)} \frac{\log(1+u)}{u^t}, \qquad t \in (0,1]. \tag{29}$$

The proof of the following result is given in Appendix C. The behavior of *κ*(*t*) is illustrated in Figure 3.

**Figure 3.** Graphs of *κ*(*t*) and *tκ*(*t*) as a function of *t*.

**Proposition 7.** *The function κ*(*t*) *defined in Equation* (29) *can be expressed as*

$$\kappa(t) = \frac{\log(1+u)}{u^t}, \qquad t \in (0,1]$$

*where*

$$\mu = \exp\left(W\left(-\frac{1}{\bar{t}}\exp\left(-\frac{1}{\bar{t}}\right)\right) + \frac{1}{\bar{t}}\right) - 1$$

*and W*(·) *denotes Lambert's W- function, i.e., W*(*z*) *is the unique solution to the equation z* = *w* exp(*w*) *on the interval* [−1, ∞)*. Furthermore, the function g*(*t*) = *tκ*(*t*) *is strictly increasing on* (0, 1] *with* lim*t*→0 *g*(*t*) = 1/*e and g*(1) = 1*, and thus*

$$\frac{1}{et} \le \kappa(t) \le \frac{1}{t'} \qquad t \in (0,1]\_\ell$$

*where the lower bound* 1/(*et*) *is tight for small values of t* ∈ (0, 1) *and the upper bound* 1/*t is tight for values of t close to 1.*

We are now ready to give the main results of this section, which are bounds on the mutual information. We begin with a general upper bound in terms of the variance of the conditional density.

**Proposition 8.** *For any* 0 < *t* ≤ 1*, the mutual information satisfies*

$$I(\mathbf{X}; \mathbf{Y}) \le \kappa(t) \int\_{S\_Y} \left[ f(y) \right]^{1-2t} \left[ \mathsf{Var}(f(y \mid \mathbf{X})) \right]^t \, \mathrm{d}y.$$

**Proof.** We use the following series of inequalities:

$$\begin{aligned} I(X;Y) & \stackrel{(a)}{=} \int f(y) \, D\left(P\_{X|Y=y} \, \Big|\, P\_X\right) \, \mathrm{d}y \\ & \stackrel{(b)}{\leq} \int f(y) \, \log\left(1 + \chi^2(P\_{X|Y=y} || P\_X)\right) \, \mathrm{d}y \\ & \stackrel{(c)}{=} \int f(y) \, \log\left(1 + \frac{\mathsf{Var}(f(y \mid X))}{f^2(y)}\right) \, \mathrm{d}y \\ & \stackrel{(d)}{\leq} \kappa(t) \int f(y) \left(\frac{\mathsf{Var}(f(y \mid X))}{f^2(y)}\right)^t \, \mathrm{d}y, \end{aligned}$$

where (a) follows from the definition of mutual information, (b) follows from Inequality (25), and (c) follows from Bayes' rule, which allows us to write the chi-square in terms of the variance of the conditional density:

$$\chi^2(P\_{X|Y=y}||P\_X) = \mathbb{E}\left[\left(\frac{f(y|X)}{f(y)} - 1\right)^2\right] = \frac{\mathsf{Var}(f(y|X))}{f^2(y)}.$$

Inequality (d) follows from the nonnegativity of the variance and the definition of *κ*(*t*).

Evaluating Proposition 8 with *t* = 1 recovers the well-known inequality *I*(*X*;*Y*) ≤ *χ* 2 (*PX*,*Y*k*PXPY*). The next two results follow from the cases 0 < *t* < 1 2 and *t* = 1 2 , respectively.

**Proposition 9.** *For any* 0 < *r* < 1*, the mutual information satisfies*

$$I(X;Y) \le \kappa(t) \left( e^{h\_r(Y)} \, \_t V\_0(Y|X) \right)^t \nu$$

*where t* = (1 − *r*)/(2 − *r*)*.*

**Proof.** Starting with Proposition 8 and applying Hölder's inequality with conjugate exponents 1/(1 − *t*) and 1/*t* leads to

$$I(\mathbf{X}; \mathbf{Y}) \le \mathfrak{x}(t) \left( \int f^r(\mathbf{y}) \, \mathbf{d}\mathbf{y} \right)^{1-t} \left( \int \mathsf{Var}(f(\mathbf{y} \mid \mathbf{X})) \, \mathbf{d}\mathbf{y} \right)^t = \mathfrak{x}(t) \, e^{t h\_l^r(\mathbf{Y})} V\_0^t(\mathbf{Y}|\mathbf{X}) .$$

where we have used the fact that *r* = (1 − 2*t*)/(1 − *t*).

**Proposition 10.** *For any p* < 1 < *q, the mutual information satisfies*

$$I(\mathbf{X}; \mathbf{Y}) \le \mathcal{C}(\lambda) \sqrt{\frac{\omega(\mathcal{S}\_Y) V\_{np}^{\lambda}(\mathbf{Y}|\mathbf{X}) V\_{nq}^{1-\lambda}(\mathbf{Y}|\mathbf{X})}{(q-p)}}$$

*where λ* = (*q* − 1)/(*q* − *p*) *and*

$$\mathcal{C}(\lambda) = \kappa(\frac{1}{2}) \sqrt{\frac{\pi \lambda^{-\lambda} (1 - \lambda)^{-(1 - \lambda)}}{\sin(\pi \lambda)}}$$

*with κ*( 1 2 ) = 0.80477 . . . *.*

**Proof.** Evaluating Proposition 8 with *t* = 1/2 gives

$$I(\mathbf{X}; \mathbf{Y}) \le \kappa(\frac{1}{2}) \int\_{\mathbb{S}\_Y} \sqrt{\mathsf{Var}(f(y \mid \mathbf{X}))} \, \mathrm{d}y.$$

Evaluating Proposition 3 with *r* = 1 2 leads to

$$\left(\int\_{\mathcal{S}\_Y} \sqrt{\mathsf{Var}(f(y \mid X))} \, d\mathcal{y}\right)^2 \le \omega(\mathcal{S}\_Y) \, \psi\_{1/2}(p, q) V\_{np}^\lambda(Y|X) V\_{nq}^{1-\lambda}(Y|X).$$

Combining these inequalities with the expression for *ψ*1/2(*p*, *q*) given in Equation (8) completes the proof.

The contribution of Propositions 9 and 10 is that they provide bounds on the mutual information in terms of quantities that can be easy to characterize. One application of these bounds is to establish conditions under which the mutual information corresponding to a sequence of random pairs (*Xk* ,*Yk* ) converges to zero. In this case, Proposition 9 provides a sufficient condition in terms of the Rényi entropy of *Yn* and the function *V*0(*Yn*|*Xn*), while Proposition 10 provides a sufficient condition in terms of *Vs*(*Yn*|*Xn*) evaluated with two difference values of *s*. These conditions are summarized in the following result.

**Proposition 11.** *Let* (*Xk* ,*Yk* ) *be a sequence of random pairs such that the conditional distribution of Yk given Xk has a density on* R*n . The following are sufficient conditions under which the mutual information of I*(*Xk* ;*Yk* ) *converges to zero as k increases to infinity:*

*1. There exists* 0 < *r* < 1 *such that*

$$\lim\_{k \to \infty} e^{h\_l(\mathbf{Y}\_k)} V\_0(\mathbf{Y}\_k | \mathbf{X}\_k) = \mathbf{0}.$$

*2. There exists p* < 1 < *q such that*

$$\lim\_{k \to \infty} V\_{np}^{q-1}(\mathbf{Y}\_k | \mathbf{X}\_k) V\_{nq}^{1-p}(\mathbf{Y}\_k | \mathbf{X}\_k) = \mathbf{0}.$$

#### *4.3. Properties of the Bounds*

The variance moment *Vs*(*Y*|*X*) has a number of interesting properties. The variance of the conditional density can be expressed in terms of an expectation with respect to two independent random variables *X*1 and *X*2 with the same distribution as *X* via the decomposition:

$$\mathsf{Var}(f(y|X)) = \mathbb{E}\left[f(y|X)f(y|X) - f(y|X\_1)f(y|X\_2)\right].$$

Consequently, by swapping the order of the integration and expectation, we obtain

$$V\_{\mathbf{s}}(Y|X) = \mathbb{E}\left[\mathbf{K}\_{\mathbf{s}}(X,X) - \mathbf{K}\_{\mathbf{s}}(X\_1, X\_2)\right],\tag{50}$$

where

$$\mathcal{K}\_s(\mathbf{x}\_1, \mathbf{x}\_2) = \int ||y||^s f(y|\mathbf{x}\_1) f(y|\mathbf{x}\_2) \,\mathrm{d}y \,\mathrm{d}y.$$

The function *Ks*(*x*1, *x*2) is a positive definite kernel that does not depend on the distribution of *X*. For *s* = 0, this kernel has been studied previously in the machine learning literature [33], where it is referred to as the expected likelihood kernel.

The variance of the conditional density also satisfies a data processing inequality. Suppose that *U* → *X* → *Y* forms a Markov chain. Then, the square of the conditional density of *Y* given *U* can be expressed as

$$f\_{Y|U}^2(y|\mu) = \mathbb{E}\left[f\_{Y|X}(y|X\_1')f\_{Y|X}(y|X\_2') \mid \mathcal{U} = \mu\right].$$

where (*U*, *X* 0 1 , *X* 0 2 ) ∼ *PUPX*1 |*UPX*2|*U*. Combining this expression with Equation (30) yields

$$N\_s(Y|\mathcal{U}) = \mathbb{E}\left[\mathcal{K}\_s(X\_1', X\_2') - \mathcal{K}\_s(X\_1, X\_2)\right],\tag{31}$$

where we recall that (*X*1, *X*2) are independent copies of *X*.

Finally, it is easy to verify that the function *Vs*(*Y*) satisfies

$$V\_s(aY|X) = |a|^{s-n} V\_s(Y|X), \quad \text{for all } a \neq 0.$$

Using this scaling relationship, we see that the sufficient conditions in Proposition 11 are invariant to scaling of *Y*.

#### *4.4. Example with Additive Gaussian Noise*

We now provide a specific example of our bounds on the mutual information. Let *X* ∈ R*n* be a random vector with distribution *PX* and let *Y* be the output of a Gaussian noise channel

$$Y = X + W\_{\prime} \tag{32}$$

where *W* ∼ N (0, *In*) is independent of *X*. If k*X*k has finite second moment, then the mutual information satisfies

$$I(X;Y) \le \frac{n}{2} \log \left( 1 + \frac{1}{n} \mathbb{E} \left[ \|X\|^2 \right] \right),\tag{33}$$

where equality is attained if and only if *X* has zero-mean isotropic Gaussian distribution. This inequality follows straightforwardly from the fact that the Gaussian distribution maximizes differential entropy subject to a second moment constraint [11]. One of the limitations of this bound is that it can be loose when the second moment is dominated by events that have small probability. In fact, it is easy to construct examples for which k*X*k does not have a finite second moment, and yet *I*(*X*;*Y*) is arbitrarily close to zero.

Our results provide bounds on *I*(*X*;*Y*) that are less sensitive to the effects of rare events. Let *φn*(*x*) = (2*π*) −*n*/2 exp(−k*x*k 2/2) denote the density of the standard Gaussian distribution on R*n* . The product of the conditional densities can be factored according to

$$\begin{split} f(y \mid \mathbf{x\_1}) f(y \mid \mathbf{x\_2}) &= \phi\_{2n} \left( \begin{bmatrix} y - \mathbf{x\_1} \\ y - \mathbf{x\_2} \end{bmatrix} \right) = \phi\_{2n} \left( \begin{bmatrix} \sqrt{2}y - (\mathbf{x\_1} + \mathbf{x\_2})/\sqrt{2} \\ (\mathbf{x\_1} - \mathbf{x\_2})/\sqrt{2} \end{bmatrix} \right) \\ &= \phi\_n \left( \sqrt{2}y - \frac{\mathbf{x\_1} + \mathbf{x\_2}}{\sqrt{2}} \right) \phi\_n \left( \frac{\mathbf{x\_1} - \mathbf{x\_2}}{\sqrt{2}} \right), \end{split}$$

where the second step follows because *φ*2*n*(·) is invariant to orthogonal transformations. Integrating with respect to *y* leads to

$$K\_{\mathbf{s}}(\boldsymbol{\chi}\_{1}, \boldsymbol{\chi}\_{2}) = 2^{-\frac{n+s}{2}} \mathbb{E}\left[ \left\| \boldsymbol{W} + \frac{\boldsymbol{\chi}\_{1} + \boldsymbol{\chi}\_{2}}{\sqrt{2}} \right\|^{s} \right] \phi\_{n}\left(\frac{\boldsymbol{\chi}\_{1} - \boldsymbol{\chi}\_{2}}{\sqrt{2}}\right) \boldsymbol{\chi}\_{1}$$

where we recall that *W* ∼ N (0, *In*). For the case *s* = 0, we see that *K*0(*x*1, *x*2) is a Gaussian kernel, thus

$$V\_0(Y|X) = (4\pi)^{-\frac{n}{2}} \left[ 1 - \mathbb{E} \left[ e^{-\frac{1}{4} \|X\_1 - X\_2\|^2} \right] \right]. \tag{34}$$

A useful property of *V*0(*Y*|*X*) is that the conditions under which it converges to zero are weaker than the conditions needed for other measures of dependence. Observe that the expectation in Equation (34) is bounded uniformly with respect to (*X*1, *X*2). In particular, for every *e* > 0 and *x* ∈ R, we have

$$\left[1 - \mathbb{E}\left[e^{-\frac{1}{4}(X\_1 - X\_2)^2}\right] \le \mathfrak{e}^2 + 2\mathbb{P}\left[\left|X - \mathfrak{x}\right| \ge \mathfrak{e}\right], \mathfrak{y}\right]$$

where we have used the inequality 1 − *e* −*x* ≤ *x* and the fact that P [|*X*1 − *X*2| ≥ 2*e*] ≤ 2P [|*X* − *x*| ≥ *e*]. Consequently, *V*0(*Y*|*X*) converges to zero whenever *X* converges to a constant value *x* in probability.

To study some further properties of these bounds, we now focus on the case where *X* is a Gaussian scalar mixture generated according to

$$X = A\sqrt{\mathcal{U}}, \quad A \sim \mathcal{N}(0, 1), \quad \mathcal{U} \ge 0,\tag{35}$$

with *A* and *U* independent. In this case, the expectations with respect to the kernel *Ks*(*x*1, *x*2) can be computed explicitly, leading to

$$V\_s(Y|X) = \frac{\Gamma(\frac{1+s}{2})}{2\pi} \mathbb{E}\left[ (1+2\mathcal{U})^{\frac{s}{2}} - \frac{(1+\mathcal{U}\_1)^{\frac{s}{2}}(1+\mathcal{U}\_2)^{\frac{s}{2}}}{\left(1+\frac{1}{2}(\mathcal{U}\_1+\mathcal{U}\_2)\right)^{\frac{s+1}{2}}} \right],\tag{36}$$

where (*U*1, *U*2) are independent copies of *U*. It can be shown that this expression depends primarily on the magnitude of *U*. This is not surprising given that *X* converges to a constant if and only if *U* converges to zero.

Our results can also be used to bound the mutual information *I*(*U*;*Y*) by noting that *U* → *X* → *Y* forms a Markov chain, and taking advantage of the characterization provided in Equation (31). Letting *X* 0 1 = *A*1 √ *U* and *X* 0 2 = *A*2 √ *U* with (*A*1, *A*2, *U*) be mutually independent leads to

$$V\_s(Y|\mathcal{U}) = \frac{\Gamma(\frac{1+s}{2})}{2\pi} \mathbb{E}\left[ (1+\mathcal{U})^{\frac{s-1}{2}} - \frac{(1+\mathcal{U}\_1)^{\frac{s}{2}}(1+\mathcal{U}\_2)^{\frac{s}{2}}}{(1+\frac{1}{2}(\mathcal{U}\_1+\mathcal{U}\_2))^{\frac{s+1}{2}}} \right],\tag{37}$$

In this case, *Vs*(*Y*|*U*) is a measure of the variation in *U*. To study its behavior, we consider the simple upper bound

$$V\_s(Y|\mathcal{U}) \le \frac{\Gamma(\frac{1+s}{2})}{2\pi} \mathbb{P}\left[\mathcal{U}\_1 \ne \mathcal{U}\_2\right] \mathbb{E}\left[\left(1+\mathcal{U}\right)^{\frac{s-1}{2}}\right],\tag{38}$$

which follows from noting that the term inside the expectation in Equation (37) is zero on the event *U*1 = *U*2. This bound shows that if *s* ≤ 1 then *Vs*(*Y*|*U*) is bounded uniformly with respect to distributions on *U*, and if *s* > 1, then *Vs*(*Y*|*U*) is bounded in terms of the ( *s*−1 2 )th moment of *U*.

In conjunction with Propositions 9 and 10, the function *Vs*(*Y*|*U*) provides bounds on the mutual information *I*(*U*;*Y*) that can be expressed in terms of simple expectations involving two independent copies of *U*. Figure 4 provides an illustration of the upper bound in Proposition 10 for the case where *U* is a discrete random variable supported on two points, and *X* and *Y* are generated according to Equations (32) and (35). This example shows that there exist sequences of distributions for which our upper bounds on the mutual information converge to zero while the chi-squared divergence between *PXY* and *PXPY* is bounded away from zero.

**Figure 4.** Bounds on the mutual information *I*(*U*;*Y*) in nats when *U* ∼ (1 − *e*)*δ*1 + *eδa*(*e*) , with *a*(*e*) = 1 + 1/ √ *e*, and *X* and *Y* are generated according to Equations (32) and (35). The bound from Proposition 10 is evaluated with *p* = 0 and *q* = 2.

#### **5. Conclusions**

This paper provides bounds on Rényi entropy and mutual information that are based on a relatively simple two-moment inequality. Extensions to inequalities with more moments are worth exploring. Another potential application is to provide a refined characterization of the "all-or-nothing" behavior seen in a sparse linear regression problem [34,35], where the current methods of analysis depend on a complicated conditional second moment method.

**Funding:** This research was supported in part by the National Science Foundation under Grant 1750362 and in part by the Laboratory for Analytic Sciences (LAS). Any opinions, findings, conclusions, and recommendations expressed in this material are those of the author and do not necessarily reflect the views of the sponsors.

**Conflicts of Interest:** The author declares no conflict of interest. The funders had no role in the design of the study; in the collection, analyses, or interpretation of data; in the writing of the manuscript, or in the decision to publish the results.

#### **Appendix A. The Gamma and Beta Functions**

This section reviews some properties of the gamma and beta functions. For *x* > 0, the gamma function is defined according to Γ(*x*) = R ∞ 0 *t x*−1 *e* −*t* d*t*. Binet's formula for the logarithm for the gamma function ([25], [Theorem 1.6.3]) gives

$$
\log \Gamma(\mathbf{x}) = \left(\mathbf{x} - \frac{1}{2}\right) \log \mathbf{x} - \mathbf{x} + \frac{1}{2} \log(2\pi) + \theta(\mathbf{x}),
\tag{A1}
$$

where the remainder term *θ*(*x*) is convex and nonincreasing with lim*x*→0 *θ*(*x*) = ∞ and lim*x*→∞ *θ*(*x*) = 0. Euler's reflection formula ([25], [Theorem 1.2.1]) gives

$$
\Gamma(\mathbf{x})\Gamma(1-\mathbf{x}) = \frac{\pi}{\sin(\pi\mathbf{x})}, \quad \mathbf{0} < \mathbf{x} < 1. \tag{A2}
$$

For *x*, *y* > 0, the beta function can be expressed as follows

$$\mathbf{B}(\mathbf{x}, y) = \frac{\Gamma(\mathbf{x})\Gamma(y)}{\Gamma(\mathbf{x} + y)} = \int\_0^1 t^{\mathbf{x} - 1}(1 - t)^{y - 1} \, \mathbf{d}t = \int\_0^\infty \frac{u^{a - 1}}{(1 + u)^{a + b}} \, \mathbf{d}u,\tag{A3}$$

where the second integral expression follows from the change of variables *t* 7→ *u*/(1 + *u*). Recall that Be(*x*, *y*) = B(*x*, *y*)(*x* + *y*) *x*+*yx* −*xy* −*y* . Using Equation (A1) leads to

$$\log\left(\tilde{\mathcal{B}}(\mathbf{x},\boldsymbol{y})\sqrt{\frac{\mathbf{x}\cdot\mathbf{y}}{2\pi(\mathbf{x}+\boldsymbol{y})}}\right) = \theta(\mathbf{x}) + \theta(\boldsymbol{y}) - \theta(\mathbf{x}+\boldsymbol{y}).\tag{A4}$$

It can also be shown that ([36], [Equation (2), pg. 2])

$$
\widetilde{\mathcal{B}}(x, y) \ge \frac{x + y}{xy}. \tag{A5}
$$

#### **Appendix B. Details for Rényi Entropy Examples**

This appendix studies properties of the two-moment inequalities for Rényi entropy described in Section 3.

#### *Appendix B.1. Log-Normal Distribution*

Let *X* be a log-normal random variable with parameters (*µ*, *σ* 2 ) and consider the parametrization

$$p = \frac{1-r}{r} - (1-\lambda)\sqrt{\frac{(1-r)\,\mu}{r\lambda(1-\lambda)}}$$

$$q = \frac{1-r}{r} + \lambda\sqrt{\frac{(1-r)\,\mu}{r\lambda(1-\lambda)}}$$

where *λ* ∈ (0, 1) and *u* ∈ (0, ∞). Then, we have

$$\begin{aligned} \psi\_r(p,q) &= \widetilde{\mathcal{B}}\left(\frac{r\lambda}{1-r}, \frac{r(1-\lambda)}{1-r}\right) \sqrt{\frac{r\lambda(1-\lambda)}{(1-r)\,\,u}},\\ L\_r(X;p,q) &= \mu + \frac{1}{2}\left(\frac{1-r}{r}\right)\sigma^2 + \frac{1}{2}\mu\sigma^2. \end{aligned}$$

Combining these expressions with Equation (A4) leads to

$$\Delta\_{\mathbf{r}}(\mathbf{X};p,q) = \theta\left(\frac{r\lambda}{1-r}\right) + \theta\left(\frac{r(1-\lambda)}{1-r}\right) - \theta\left(\frac{r}{1-r}\right) + \frac{1}{2}\mu\sigma^2 - \frac{1}{2}\log\left(\mu\sigma^2\right) - \frac{1}{2}\log\left(r^{\frac{1}{\tau-1}}\right). \tag{A6}$$

We now characterize the minimum with respect to the parameters (*λ*, *u*). Note that the mapping *λ* 7→ *θ*( *rλ* 1−*r* ) + *θ*( *r*(1−*λ*) 1−*r* ) is convex and symmetric about the point *λ* = 1/2. Therefore, the minimum with respect to *λ* is attained at *λ* = 1/2. Meanwhile, mapping *u* 7→ *uσ* 2 − log(*uσ* 2 ) is convex and attains its minimum at *u* = 1/*σ* 2 . Evaluating Equation (A6) with these values, we see that the optimal two-moment inequality can be expressed as

$$\Delta\_r(X) = 2\theta\left(\frac{r}{2(1-r)}\right) - \theta\left(\frac{r}{1-r}\right) + \frac{1}{2}\log\left(e^{\frac{1}{1-r}}\right).$$

By Equation (A4), this expression is equivalent to Equation (A1). Moreover, the fact that ∆*r*(*X*) decreases to zero as *r* increases to one follows from the fact that *θ*(*x*) decreases to zero and *x* increases to infinity.

Next, we express the gap in terms of the pair (*p*, *q*). Comparing the difference between ∆*r*(*X*; *p*, *q*) and ∆*r*(*X*) leads to

$$
\Delta\_r(\mathbf{X}; p, q) = \Delta\_r(\mathbf{X}) + \frac{1}{2}\varrho \left( \frac{r\lambda(1-\lambda)}{1-r}(q-p)^2\sigma^2 \right) + \theta \left( \frac{r\lambda}{1-r} \right) + \theta \left( \frac{r(1-\lambda)}{1-r} \right) - 2\theta \left( \frac{r}{2(1-r)} \right),
$$

where *ϕ*(*x*) = *x* − log(*x*) − 1. In particular, if *p* = 0, then we obtain the simplified expression

$$
\Delta\_r(X;0,q) = \Delta\_r(X) + \frac{1}{2}\varrho\left(\left(q-\frac{1-r}{r}\right)\sigma^2\right) + \theta\left(\frac{r}{1-r}-\frac{1}{q}\right) + \theta\left(\frac{1}{q}\right) - 2\theta\left(\frac{r}{2(1-r)}\right).
$$

This characterization shows that the gap of the optimal one-moment inequality ∆e*r*(*X*) increases to infinity in the limit as either *σ* 2 → 0 or *σ* 2 → ∞.

#### *Appendix B.2. Multivariate Gaussian Distribution*

Let *Y* ∼ N (0, *In*) be an *n*-dimensional Gaussian vector and consider the parametrization

$$\begin{aligned} p &= \frac{1-r}{r} - \frac{1-\lambda}{r} \sqrt{\frac{2(1-r)z}{\lambda(1-\lambda)}}, \\ q &= \frac{1-r}{r} + \frac{\lambda}{r} \sqrt{\frac{2(1-r)z}{\lambda(1-\lambda)}}. \end{aligned}$$

where *λ* ∈ (0, 1) and *z* ∈ (0, ∞). We can write

$$\begin{aligned} \log \omega(S\_Y) &= \frac{n}{2} \log \pi - \log \left(\frac{n}{2}\right) - \log \Gamma \left(\frac{n}{2}\right) \\ \psi\_r(p,q) &= \tilde{\mathcal{B}} \left(\frac{r\lambda}{1-r}, \frac{r(1-\lambda)}{1-r}\right) \sqrt{\frac{r\lambda(1-\lambda)}{(1-r)}} \sqrt{\frac{nr}{2z}}. \end{aligned}$$

Furthermore, if

$$2(1 - \lambda)\sqrt{\frac{2(1 - r)z}{\lambda(1 - \lambda)n}} < 1,\tag{A7}$$

then *Lr*(k*Y*k *n* ; *p*, *q*) is finite and is given by

$$L\_{\Gamma}(\|Y\|^{\
\eta};p,q) = Q\_{r,\eta}(\lambda,z) + \frac{n}{2}\log 2 + \frac{r}{1-r}\left[\log \Gamma\left(\frac{n}{2r}\right) - \log \Gamma\left(\frac{n}{2}\right)\right],$$

where

$$\begin{split} Q\_{r,\mathbb{I}}(\lambda,z) &= \frac{r\lambda}{1-r} \log \Gamma\left(\frac{n}{2r} - \frac{1-\lambda}{r} \sqrt{\frac{(1-r)nz}{2\lambda(1-\lambda)}}\right) + \frac{r(1-\lambda)}{1-r} \log \Gamma\left(\frac{n}{2r} + \frac{\lambda}{r} \sqrt{\frac{(1-r)nz}{2\lambda(1-\lambda)}}\right) \\ &\quad - \frac{r}{1-r} \log \Gamma\left(\frac{n}{2r}\right). \end{split} \tag{A8}$$

Here, we note that the scaling in Equation (21) corresponds to *λ* = 1/2 and *z* = *n*/(*n* + 1), and thus the condition Inequality (A7) is satisfied for all *n* ≥ 1. Combining the above expressions and then using Equations (A1) and (A4) leads to

$$\begin{split} \Delta\_{l}(Y;p,q) &= \theta\left(\frac{r\lambda}{1-r}\right) + \theta\left(\frac{r(1-\lambda)}{1-r}\right) - \theta\left(\frac{r}{1-r}\right) + Q\_{r,\mathfrak{n}}(z,\lambda) - \frac{1}{2}\log z - \frac{1}{2}\log\left(r^{\frac{1}{r-1}}\right) \\ &\quad + \frac{r}{1-r}\theta\left(\frac{n}{2r}\right) - \frac{1}{1-r}\theta\left(\frac{n}{2}\right). \end{split} \tag{A9}$$

Next, we study some properties of *Qr*,*n*(*λ*, *z*). By Equation (A1), the logarithm of the gamma function can be expressed as the sum of convex functions:

$$\log \Gamma(\mathbf{x}) = \varphi(\mathbf{x}) + \frac{1}{2} \log \left(\frac{1}{\mathbf{x}}\right) + \frac{1}{2} \log(2\pi) - 1 + \theta(\mathbf{x})\_{\mathbf{x}}$$

where *ϕ*(*x*) = *x* log *x* + 1 − *x*. Starting with the definition of *Q*(*λ*, *z*) and then using Jensen's inequality yields

$$\begin{split} Q\_{r,n}(z,\lambda) &\geq \frac{r\lambda}{1-r} \varphi\left(\frac{n}{2r} - \frac{1-\lambda}{r} \sqrt{\frac{(1-r)nz}{2\lambda(1-\lambda)}}\right) \\ &\quad + \frac{r(1-\lambda)}{1-r} \varphi\left(\frac{n}{2r} + \frac{\lambda}{r} \sqrt{\frac{(1-r)nz}{2\lambda(1-\lambda)}}\right) - \frac{r}{1-r} \varphi\left(\frac{n}{2r}\right) \\ &= \frac{\lambda}{a} \varphi\left(1 - \sqrt{\left(\frac{1-\lambda}{\lambda}\right)az}\right) + \frac{(1-\lambda)}{a} \varphi\left(1 + \sqrt{\left(\frac{\lambda}{1-\lambda}\right)az}\right). \end{split}$$

where *a* = 2(1 − *r*)/*n*. Using the inequality *ϕ*(*x*) ≥ (3/2)(*x* − 1) 2/(*x* + 2) leads to

$$\begin{split} Q\_{r,n}(\lambda, z) &\geq \frac{z}{2} \left[ \left( 1 - \sqrt{\left( \frac{1-\lambda}{\lambda} \right) bz} \right) \left( 1 + \sqrt{\left( \frac{\lambda}{1-\lambda} \right) bz} \right) \right]^{-1} \\ &\geq \frac{z}{2} \left( 1 + \sqrt{\left( \frac{\lambda}{1-\lambda} \right) bz} \right)^{-1} \end{split} \tag{A10}$$

where *b* = 2(1 − *r*)/(9*n*).

Observe that the right-hand side of Inequality (A10) converges to *z*/2 as *n* increases to infinity. It turns out this limiting behavior is tight. Using Equation (A1), it is straightforward to show that *Qn*(*λ*, *z*) converges pointwise to *z*/2 as *n* increases to infinity—that is,

$$\lim\_{n \to \infty} Q\_{r,n}(\lambda, z) = \frac{1}{2} z,\tag{A11}$$

for any fixed pair (*λ*, *z*) ∈ (0, 1) × (0, ∞).

#### *Appendix B.3. Proof of Proposition 6*

Let *D* = (0, 1) × (0, ∞). For fixed *r* ∈ (0, 1), we use *Qn*(*λ*, *z*) to denote the function *Qr*,*n*(*λ*, *z*) defined in Equation (A8) and we use *Gn*(*λ*, *z*) to denote the right-hand side of Equation (A9). These functions are defined to be equal to positive infinity for any pair (*λ*, *z*) ∈ *D* such that Inequality (A7) does not hold.

Note that the terms *θ*(*n*/(2*r*)) and *θ*(*n*/2) converge to zero in the limit as *n* increases to infinity. In conjunction with Equation (A11), this shows that *Gn*(*λ*, *z*) converges pointwise to a limit *G*(*λ*, *z*) given by

$$G(\lambda, z) = \theta\left(\frac{r\lambda}{1 - r}\right) + \theta\left(\frac{r(1 - \lambda)}{1 - r}\right) - \theta\left(\frac{r}{1 - r}\right) + \frac{1}{2}z - \frac{1}{2}\log\left(z\right) - \frac{1}{2}\log\left(r^{\frac{1}{r - 1}}\right).$$

At this point, the correspondence with the log-normal distribution can be seen from the fact that *G*(*λ*, *z*) is equal to the right-hand side of Equation (A6) evaluated with *uσ* 2 = *z*.

To show that the gap corresponding to the log-normal distribution provides an upper bound on the limit, we use

$$\begin{aligned} \limsup\_{n \to \infty} \Delta\_{\mathbb{T}}(\mathbf{Y}) &= \limsup\_{n \to \infty} \inf\_{(\lambda, z) \in D} \mathcal{G}\_{\mathbb{N}}(\lambda, z) \\ &\le \inf\_{(\lambda, z) \in D} \limsup\_{n \to \infty} \mathcal{G}\_{\mathbb{N}}(\lambda, z) \\ &= \inf\_{(\lambda, z) \in D} \mathcal{G}(\lambda, z) \\ &= \Delta\_{\mathbb{T}}(\mathbf{X}). \end{aligned} \tag{A12}$$

Here, the last equality follows from the analysis in Appendix B.1, which shows that the minimum of *G*(*λ*, *z*) is a attained at *λ* = 1/2 and *z* = 1.

To prove the lower bound requires a bit more work. Fix any *e* ∈ (0, 1) and let *De* = (0, 1 − *e*] × (0, ∞). Using the lower bound on *Qn*(*λ*, *z*) given in Inequality (A10), it can be verified that

$$\liminf\_{n \to \infty} \inf\_{(\lambda, z) \in D\_{\mathfrak{c}}} \left[ Q\_{r, n}(z, \lambda) - \frac{1}{2} \log z \right] \ge \frac{1}{2}.$$

Consequently, we have

$$\liminf\_{\substack{\eta \to \infty \ (\lambda, z) \in D\_{\mathfrak{c}}}} \inf\_{\mathcal{G}\_{\mathfrak{U}}} \mathcal{G}\_{\mathfrak{U}}(\lambda, z) = \inf\_{(\lambda, z) \in D\_{\mathfrak{c}}} \mathcal{G}(\lambda, z) \ge \Delta\_{\mathfrak{I}}(X). \tag{A13}$$

To complete the proof we will show that for any sequence *λn* that converges to one as *n* increases to infinity, we have

$$\liminf\_{n \to \infty} \inf\_{z \in (0, \infty)} G\_{\mathbb{II}}(\lambda\_n z) = \infty. \tag{A14}$$

To see why this is the case, note that by Equation (A4) and Inequality (A5),

$$\theta\left(\frac{r\lambda}{1-r}\right) + \theta\left(\frac{r(1-\lambda)}{1-r}\right) - \theta\left(\frac{r}{1-r}\right) \ge \frac{1}{2}\log\left(\frac{1-r}{2\pi r\lambda\left(1-\lambda\right)}\right).$$

Therefore, we can write

$$\mathcal{G}\_{\mathbb{H}}(\lambda, z) \ge Q\_{\mathbb{H}}(\lambda, z) - \frac{1}{2} \log \left( \lambda (1 - \lambda) z \right) + c\_{n\nu} \tag{A15}$$

where *cn* is bounded uniformly for all *n*. Making the substitution *u* = *λ*(1 − *λ*)*z*, we obtain

$$\inf\_{z>0} G\_{\mathbb{N}}\left(\lambda, z\right) \geq \inf\_{\mu>0} \left[Q\_{\mathbb{N}}\left(\lambda, \frac{\mu}{\lambda\left(1-\lambda\right)}\right) - \frac{1}{2}\log\mu\right] + c\_{\mathbb{N}}.$$

Next, let *bn* = 2(1 − *r*)/(9*n*). The lower bound in Inequality (A10) leads to

$$\inf\_{u>0} \left[ Q\_{\mathbb{H}} \left( \lambda, \frac{u}{\lambda(1-\lambda)} \right) - \frac{1}{2} \log u \right] \ge \inf\_{u>0} \left[ \frac{u}{2\lambda} \left( \frac{1}{1-\lambda + \sqrt{b\_{\mathbb{H}} u}} \right) - \frac{1}{2} \log u \right]. \tag{A16}$$

The limiting behavior in Equation (A14) can now be seen as a consequence of Inequality (A15) and the fact that, for any sequence *λn* converging to one, the right-hand side of Inequality (A16) increases without bound as *n* increases. Combining Inequality (A12), Inequality (A13), and Equation (A14) establishes that the large *n* limit of ∆*r*(*Y*) exists and is equal to ∆*r*(*X*). This concludes the proof of Proposition 6.

*Appendix B.4. Proof of Inequality* (24)

Given any *λ* ∈ (0, 1) and *u* ∈ (0, ∞) let

$$p(r) = \frac{1-r}{r} - \sqrt{\frac{1-r}{r} \left(\frac{1-\lambda}{\lambda}\right) \mu}$$

$$q(r) = \frac{1-r}{r} + \sqrt{\frac{1-r}{r} \left(\frac{\lambda}{1-\lambda}\right) \mu}$$

We need the following results, which characterize the terms in Proposition 4 in the limit as *r* increases to one.

**Lemma A1.** *The function ψr*(*p*(*r*), *q*(*r*)) *satisfies*

$$\lim\_{r \to 1} \psi\_r(p(r), q(r)) = \sqrt{\frac{2\pi}{u}}.$$

**Proof.** Starting with Equation (A4), we can write

$$
\psi\_r(p,q) = \frac{1}{q-p} \sqrt{\frac{2\pi(1-r)}{r\lambda(1-\lambda}} \exp\left(\theta\left(\frac{r\lambda}{1-r}\right) + \theta\left(\frac{r(1-\lambda)}{1-r}\right) - \theta\left(\frac{r}{1-r}\right)\right).
$$

As p *r* converges to one, the terms in the exponent converge to zero. Note that *q*(*r*) − *p*(*r*) = *rλ*(1 − *λ*)/(1 − *r*) completes the proof.

**Lemma A2.** *If X is a random variable such that s* 7→ E [|*X*| *s* ] *is finite in a neighborhood of zero, then* E [log(*X*)] *and* Var(log(*X*)) *are finite, and*

$$\lim\_{r \to 1} L\_r(X; p(r), q(r)) = \mathbb{E}\left[\log|X|\right] + \frac{\mu}{2} \operatorname{Var}(\log|X|).$$

**Proof.** Let Λ(*s*) = log(E [|*X*| *s* ]). The assumption that E [|*X*| *s* ] is finite in a neighborhood of zero means that E [(log |*X*|) *m*] is finite for all positive integers *m*, and thus Λ(*s*) is real analytic in a neighborhood of zero. Hence, there exist constants *δ* > 0 and *C* < ∞, depending on the distribution of *X*, such that

$$\left|\Lambda(s) - as + bs^2\right| \le \mathcal{C} \left|s\right|^3, \quad \text{for all } |s| \le \delta\_\prime$$

where *a* = E [log |*X*|] and *b* = 1 2 Var(|*X*|). Consequently, for all *r* such that 1 − *δ* < *p*(*r*) < (1 − *r*)/*r* < *q*(*r*) < 1 + *δ*, it follows that

$$\left| L\_r(X; p(r), q(r)) - a - \left( \frac{1 - r}{r} + u \right) b \right|\_{} \le \mathbb{C} \frac{r}{1 - r} \left( \lambda |p(r)|^3 + (1 - \lambda) |q(r)|^3 \right).$$

Taking the limit as *r* increases to one completes the proof.

We are now ready to prove Inequality (24). Combining Proposition 4 with Lemma A1 and Lemma A2 yields

$$\limsup\_{r \to \infty} h\_r(X) \le \frac{1}{2} \log \left( \frac{2\pi}{u} \right) + \mathbb{E} \left[ \log X \right] + \frac{u}{2} \operatorname{Var}(\log X).$$

The stated inequality follows from evaluating the right-hand side with *u* = 1/ Var(log *X*), recalling that *h*(*X*) corresponds to the limit of *hr*(*X*) as *r* increases to one.

#### **Appendix C. Proof of Proposition 7**

The function *κ* : (0, 1] → R+ can be expressed as

$$\kappa(t) = \sup\_{\boldsymbol{\mu} \in (0, \infty)} \rho\_l(\boldsymbol{\mu}), \tag{A17}$$

where *ρt*(*u*) = log(1 + *u*)/*u t* . For *t* = 1, the bound log(1 + *u*) ≤ *u* implies that *ρ*1(*u*) ≤ 1. Noting that lim*u*→0 *ρ*1(*u*) = 1, we conclude that *κ*(1) = 1.

Next, we consider the case *t* ∈ (0, 1). The function *ρt* is continuously differentiable on (0, ∞) with

$$\operatorname{sgn}(\rho\_t'(\mu)) = \operatorname{sgn}\left(\mu - t(1+\mu)\log(1+\mu)\right). \tag{A18}$$

Under the assumption *t* ∈ (0, 1), we see that *ρt*(*u*) is increasing for all *u* sufficiently close to zero and decreasing for all *u* sufficiently large, and thus the supremum is attained at a stationary point of *ρt*(*u*) on (0, ∞). Making the substitution *w* = log(1 + *u*) − 1/*t* leads to

$$
\rho\_t'(\mu) = 0 \quad \Longleftrightarrow \quad we^w = -\frac{1}{t}e^{-\frac{1}{T}}.
$$

For *t* ∈ (0, 1), it follows that −1 *t e* − 1 *t* ∈ (−*e* −1 , 0), and thus *ρ* 0 *t* (*u*) has a unique root that can be expressed as

$$\mu\_t^\* = \exp\left(\mathcal{W}\left(-\frac{1}{t}\exp\left(-\frac{1}{t}\right)\right) + \frac{1}{t}\right) - 1\_{\prime\prime}$$

where Lambert's function *W*(*z*) is the solution to the equation *z* = *wew* on the interval on [−1, ∞).

**Lemma A3.** *The function g*(*t*) = *tκ*(*t*) *is strictly increasing on* (0, 1] *with* lim*t*→0 *g*(*t*) = 1/*e and g*(1) = 1*.*

**Proof.** The fact that *g*(1) = 1 follows from *κ*(1) = 1. By the envelope theorem [37], the derivative of *g*(*t*) can be expressed as

$$g'(t) = \frac{\mathbf{d}}{\mathbf{d}t} \, t\rho\_t(u)\Big|\_{u=u\_t^\*} = \frac{\log(1+u\_t^\*)}{(u\_t^\*)^t} - t\log(u\_t^\*)\frac{\log(1+u\_t^\*)}{(u\_t^\*)^t}$$

In view of Equation (A18), it follows that *ρ* 0 *t* (*u* ∗ *t* ) = 0 can be expressed equivalently as

$$\frac{u\_t^\*}{(1+u\_t^\*)\log(1+u\_t^\*)} = t,\tag{A19}$$

and thus

$$\operatorname{sgn}(g'(t)) = \operatorname{sgn}\left(1 - \frac{u\_t^\* \log u\_t^\*}{(1 + u\_t^\*) \log(1 + u\_t^\*)}\right). \tag{A20}$$

Noting that *u* log *u* < (1 + *u*)log(1 + *u*) for all *u* ∈ (0, ∞), it follows that *g* 0 (*t*) > 0 is strictly positive, and thus *g*(*t*) is strictly increasing.

To prove the small *t* limit, we use Equation (A19) to write

$$\log(g(t)) = \log\left(\frac{u\_t^\*}{1 + u\_t^\*}\right) - \frac{u\_t^\* \log u\_t^\*}{(1 + u\_t^\*) \log(1 + u\_t^\*)}.\tag{A21}$$

Now, as *t* decreases to zero, Equation (A19) shows that *u* ∗ *t* increases to infinity. By Equation (A21), it then follows that log(*g*(*t*)) converges to negative one, which proves the desired limit.

#### **References**

© 2020 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).

## *Article* **Strongly Convex Divergences**

#### **James Melbourne**

Department of Electrical and Computer Engineering, University of Minnesota-Twin Cities, Minneapolis, MN 55455, USA<EMAIL_ADDRESS>
Received: 2 September 2020; Accepted: 9 November 2020; Published: 21 November 2020

**Abstract:** We consider a sub-class of the *f*-divergences satisfying a stronger convexity property, which we refer to as strongly convex, or *κ*-convex divergences. We derive new and old relationships, based on convexity arguments, between popular *f*-divergences.

**Keywords:** information measures; *f*-divergence; hypothesis testing; total variation; skew-divergence; convexity; Pinsker's inequality; Bayes risk; Jensen–Shannon divergence

#### **1. Introduction**

The concept of an *f*-divergence, introduced independently by Ali-Silvey [1], Morimoto [2], and Csisizár [3], unifies several important information measures between probability distributions, as integrals of a convex function *f* , composed with the Radon–Nikodym of the two probability distributions. (An additional assumption can be made that *f* is strictly convex at 1, to ensure that *Df* (*µ*||*ν*) > 0 for *µ* 6= *ν*. This obviously holds for any *f* 00(1) > 0, and can hold for some *f*-divergences without classical derivatives at 0, for instance the total variation is strictly convex at 1. An example of an *f*-divergence not strictly convex is provided by the so-called "hockey-stick" divergence, where *f*(*x*) = (*x* − *γ*)+, see [4–6].) For a convex function *f* : (0, ∞) → R such that *f*(1) = 0, and measures *P* and *Q* such that *P Q*, the *f*-divergence from *P* to *Q* is given by *Df* (*P*||*Q*) := R *f dP dQ dQ*. The canonical example of an *f*-divergence, realized by taking *f*(*x*) = *x* log *x*, is the relative entropy (often called the KL-divergence), which we denote with the subscript *f* omitted. *f*-divergences inherit many properties enjoyed by this special case; non-negativity, joint convexity of arguments, and a data processing inequality. Other important examples include the total variation, the *χ* 2 -divergence, and the squared Hellinger distance. The reader is directed to Chapter 6 and 7 of [7] for more background.

We are interested in how stronger convexity properties of *f* give improvements of classical *f*-divergence inequalities. More explicitly, we consider consequences of *f* being *κ*-convex, in the sense that the map *x* 7→ *f*(*x*) − *κx* 2/2 is convex. This is in part inspired by the work of Sason [8], who demonstrated that divergences that are *κ*-convex satisfy "stronger than *χ* 2" data-processing inequalities.

Perhaps the most well known example of an *f*-divergence inequality is Pinsker's inequality, which bounds the square of the total variation above by a constant multiple of the relative entropy. That is for probability measures *P* and *Q*, |*P* − *Q*| 2 *TV* ≤ *c D*(*P*||*Q*). The optimal constant is achieved for Bernoulli measures, and under our conventions for total variation, *c* = 1/2 log *e*. Many extensions and sharpenings of Pinsker's inequality exist (for examples, see [9–11]). Building on the work of Guntuboyina [9] and Topsøe [11], we achieve a further sharpening of Pinsker's inequality in Theorem 9.

Aside from the total variation, most divergences of interest have stronger than affine convexity, at least when *f* is restricted to a sub-interval of the real line. This observation is especially relevant to the situation in which one wishes to study *Df* (*P*||*Q*) in the existence of a bounded Radon–Nikodym derivative *dP dQ* ∈ (*a*, *b*) ( (0, ∞). One naturally obtains such bounds for skew divergences. That is divergences of the form (*P*, *Q*) 7→ *Df* ((1 − *t*)*P* + *tQ*||(1 − *s*)*P* + *sQ*) for *t*,*s* ∈ [0, 1], as in this case,

(1−*t*)*P*+*tQ* (1−*s*)*P*+*sQ* ≤ max n 1−*t* 1−*s* , *t s* o . Important examples of skew-divergences include the skew divergence [12] based on the relative entropy and the Vincze–Le Cam divergence [13,14], called the triangular discrimination in [11] and its generalization due to Györfi and Vajda [15] based on the *χ* 2 -divergence. The Jensen–Shannon divergence [16] and its recent generalization [17] give examples of *f*-divergences realized as linear combinations of skewed divergences.

Let us outline the paper. In Section 2, we derive elementary results of *κ*-convex divergences and give a table of examples of *κ*-convex divergences. We demonstrate that *κ*-convex divergences can be lower bounded by the *χ* 2 -divergence, and that the joint convexity of the map (*P*, *Q*) 7→ *Df* (*P*||*Q*) can be sharpened under *κ*-convexity conditions on *f* . As a consequence, we obtain bounds between the mean square total variation distance of a set of distributions from its barycenter, and the average *f*-divergence from the set to the barycenter.

In Section 3, we investigate general skewing of *f*-divergences. In particular, we introduce the skew-symmetrization of an *f*-divergence, which recovers the Jensen–Shannon divergence and the Vincze–Le Cam divergences as special cases. We also show that a scaling of the Vincze–Le Cam divergence is minimal among skew-symmetrizations of *κ*-convex divergences on (0, 2). We then consider linear combinations of skew divergences and show that a generalized Vincze–Le Cam divergence (based on skewing the *χ* 2 -divergence) can be upper bounded by the generalized Jensen–Shannon divergence introduced recently by Nielsen [17] (based on skewing the relative entropy), reversing the classical convexity bounds *D*(*P*||*Q*) ≤ log(1 + *χ* 2 (*P*||*Q*)) ≤ log *e χ* 2 (*P*||*Q*). We also derive upper and lower total variation bounds for Nielsen's generalized Jensen–Shannon divergence.

In Section 4, we consider a family of densities {*pi*} weighted by *λi* , and a density *q*. We use the Bayes estimator *T*(*x*) = arg max*i λi pi*(*x*) to derive a convex decomposition of the barycenter *p* = ∑*i λi pi* and of *q*, each into two auxiliary densities. (Recall, a Bayes estimator is one that minimizes the expected value of a loss function. By the assumptions of our model, that P(*θ* = *i*) = *λi* , and P(*X* ∈ *A*|*θ* = *i*) = R *A pi*(*x*)*dx*, we have E`(*θ*, <sup>ˆ</sup>*θ*) = <sup>1</sup> <sup>−</sup> R *λ*ˆ*θ*(*x*) *p* <sup>ˆ</sup>*θ*(*x*) (*x*)*dx* for the loss function `(*i*, *j*) = 1 − *δi*(*j*) and any estimator ˆ*θ*. It follows that E`(*θ*, <sup>ˆ</sup>*θ*) <sup>≥</sup> <sup>E</sup>`(*θ*, *T*) by *λ*ˆ*θ*(*x*) *p* ˆ*θ*(*x*) (*x*) ≤ *λT*(*x*) *pT*(*x*) (*x*). Thus, *T* is a Bayes estimator associated to `. ) We use this decomposition to sharpen, for *κ*-convex divergences, an elegant theorem of Guntuboyina [9] that generalizes Fano and Pinsker's inequality to *f*-divergences. We then demonstrate explicitly, using an argument of Topsøe, how our sharpening of Guntuboyina's inequality gives a new sharpening of Pinsker's inequality in terms of the convex decomposition induced by the Bayes estimator.

#### *Notation*

Throughout, *f* denotes a convex function *f* : (0, ∞) → R ∪ {∞}, such that *f*(1) = 0. For a convex function defined on (0, ∞), we define *f*(0) := lim*x*→0 *f*(*x*). We denote by *f* ∗ , the convex function *f* ∗ : (0, ∞) → R ∪ {∞} defined by *f* ∗ (*x*) = *x f*(*x* −1 ). We consider Borel probability measures *P* and *Q* on a Polish space X and define the *f*-divergence from *P* to *Q*, via densities *p* for *P* and *q* for *Q* with respect to a common reference measure *µ* as

$$\begin{split} D\_f(p||q) &= \int\_{\mathcal{X}} f\left(\frac{p}{q}\right) q d\mu \\ &= \int\_{\{pq>0\}} q f\left(\frac{p}{q}\right) d\mu + f(0)Q(\{p=0\}) + f^\*(0)P(\{q=0\}). \end{split} \tag{1}$$

We note that this representation is independent of *µ*, and such a reference measure always exists, take *µ* = *P* + *Q* for example.

For *t*,*s* ∈ [0, 1], define the binary *f*-divergence

$$D\_f(t||s) \coloneqq sf\left(\frac{t}{s}\right) + (1-s)f\left(\frac{1-t}{1-s}\right) \tag{2}$$

with the conventions, *f*(0) = lim*t*→0+ *f*(*t*), 0 *f*(0/0) = 0, and 0 *f*(*a*/0) = *a* lim*t*→∞ *f*(*t*)/*t*. For a random variable *X* and a set *A*, we denote the probability that *X* takes a value in *A* by P(*X* ∈ *A*), the expectation of the random variable by E*X*, and the variance by Var(*X*) := E|*X* − E*X*| 2 . For a probability measure *µ* satisfying *µ*(*A*) = P(*X* ∈ *A*) for all Borel *A*, we write *X* ∼ *µ*, and, when there exists a probability density function such that P(*X* ∈ *A*) = R *A f*(*x*)*dγ*(*x*) for a reference measure *γ*, we write *X* ∼ *f* . For a probability measure *µ* on X , and an *L* 2 function *f* : X → R, we denote Var*µ*(*f*) := Var(*f*(*X*)) for *X* ∼ *µ*.

#### **2. Strongly Convex Divergences**

**Definition 1.** *A* R ∪ {∞}*-valued function f on a convex set K* ⊆ R *is κ-convex when x*, *y* ∈ *K and t* ∈ [0, 1] *implies*

$$f((1-t)\mathbf{x}+ty) \le (1-t)f(\mathbf{x}) + tf(y) - \kappa t(1-t)(\mathbf{x}-y)^2/2. \tag{3}$$

For example, when *f* is twice differentiable, (3) is equivalent to *f* 00(*x*) ≥ *κ* for *x* ∈ *K*. Note that the case *κ* = 0 is just usual convexity.

**Proposition 1.** *For f* : *K* → R ∪ {∞} *and κ* ∈ [0, ∞)*, the following are equivalent:*

$$f'\_+(t) \ge f'\_+(s) + \kappa(t - s)$$

*for t* ≥ *s.*

**Proof.** Observe that it is enough to prove the result when *κ* = 0, where the proposition is reduced to the classical result for convex functions.

**Definition 2.** *An f-divergence Df is κ-convex on an interval K for κ* ≥ 0 *when the function f is κ-convex on K.*

Table 1 lists some *κ*-convex *f*-divergences of interest to this article.

**Table 1.** Examples of Strongly Convex Divergences.

Observe that we have taken the normalization convention on the total variation (the total variation for a signed measure *µ* on a space *X* can be defined through the Hahn-Jordan decomposition of the measure into non-negative measures *µ* + and *µ* − such that *µ* = *µ* + − *µ* −, as k*µ*k = *µ* +(*X*) + *µ* −(*X*) (see [18]); in our notation, |*µ*|*TV* = k*µ*k/2) which we denote by |*P* − *Q*|*TV*, such that |*P* − *Q*|*TV* = sup*A* |*P*(*A*) − *Q*(*A*)| ≤ 1. In addition , note that the *α*-divergence interpolates Pearson's *χ* 2 -divergence when *α* = 3, one half Neyman's *χ* 2 -divergence when *α* = −3, the squared Hellinger divergence when *α* = 0, and has limiting cases, the relative entropy when *α* = 1 and the reverse relative entropy when *α* = −1. If *f* is *κ*-convex on [*a*, *b*], then recalling its dual divergence *f* ∗ (*x*) := *x f*(*x* −1 ) is *κa* 3 -convex on [ 1 *b* , 1 *a* ]. Recall that *f* ∗ satisfies the equality *Df* ∗ (*P*||*Q*) = *Df* (*Q*||*P*). For brevity, we use *χ* 2 -divergence to refer to the Pearson *χ* 2 -divergence, and we articulate Neyman's *χ* 2 explicitly when necessary.

The next lemma is a restatement of Jensen's inequality.

**Lemma 1.** *If f is κ-convex on the range of X,*

$$\mathbb{E}f(X) \ge f(\mathbb{E}(X)) + \frac{\kappa}{2}\operatorname{Var}(X).$$

**Proof.** Apply Jensen's inequality to *f*(*x*) − *κx* 2/2.

For a convex function *f* such that *f*(1) = 0 and *c* ∈ R, the function ˜ *f*(*t*) = *f*(*t*) + *c*(*t* − 1) remains a convex function, and what is more satisfies

$$D\_f(P||Q) = D\_f(P||Q)$$

since R *c*(*p*/*q* − 1)*qdµ* = 0.

**Definition 3** (*χ* 2 -divergence)**.** *For f*(*t*) = (*t* − 1) 2 *, we write*

$$
\chi^2(P||Q) := D\_f(P||Q).
$$

We pursue a generalization of the following bound on the total variation by the *χ* 2 -divergence [19–21].

**Theorem 1** ([19–21])**.** *For measures P and Q,*

$$|P - Q|\_{TV}^2 \le \frac{\chi^2(P||Q)}{2}.\tag{4}$$