that for any $r>0$ and any $g\in\mathcal{O}_{10r}$ and
$\bm{v}\in{\mathbb{R}}^{d}$,

$\displaystyle\|g\bm{v}\|\leq\|\bm{v}\|+\|(g-I_{d})\bm{v}\|\leq(1+r)\|\bm{v}\|.$

Hence for all $r>0$,

$\displaystyle\Delta(\Lambda)-\Delta(g\Lambda)\leq\log(1+r)<r,\qquad\forall\
g\in\mathcal{O}_{10r},\ \Lambda\in X_{d}.$

Similarly, since $\mathcal{O}_{10r}$ is invariant under inversion, we also
have

$\displaystyle\Delta(g\Lambda)-\Delta(\Lambda)=\Delta(g\Lambda)-\Delta(g^{-1}g\Lambda)<r,\qquad\forall\
g\in\mathcal{O}_{10r},\ \Lambda\in X_{d}.$

Thus

$\displaystyle\mathcal{O}_{10r_{1}}\Delta^{-1}[0,r_{2}]\subset\Delta^{-1}[0,r_{1}+r_{2}],\qquad\forall\
r_{1},r_{2}>0.$

Now to prove the relation (6.12), in view of the definition of
$\widetilde{\Delta}_{r}$, it suffices to show that for any
$g\in\mathcal{O}_{r_{1}}$, $0\leq s<1$ and $\Lambda\in\Delta^{-1}[0,r_{2}]$
there exists some $0\leq s^{\prime}<1$ such that

$gg_{-s}\Lambda\in g_{-s^{\prime}}\Delta^{-1}[0,r_{1}+r_{2}],$

or equivalently, $g_{s^{\prime}}gg_{-s}\Lambda\in\Delta^{-1}[0,r_{1}+r_{2}]$.
We take $s^{\prime}=s$. By direct computation and using
$\alpha_{i},\beta_{j}\in(0,1)$ for all $1\leq i\leq m,1\leq j\leq n$, we have

$\displaystyle\max\big{\\{}\|g_{s}gg_{-s}-I_{d}\|,\|(g_{s}gg_{-s})^{-1}-I_{d}\|\big{\\}}$
$\displaystyle=\max\big{\\{}\|g_{s}(g-I_{d})g_{-s}\|,\|g_{s}(g^{-1}-I_{d})g_{-s}\|\big{\\}}$
$\displaystyle<e^{2}\max\big{\\{}\|g-I_{d}\|,\|g^{-1}-I_{d}\|\big{\\}}<e^{2}\frac{r_{1}}{10d}<\frac{r_{1}}{d}.$

Thus $g_{s}gg_{-s}\in\mathcal{O}_{10r_{1}}$, implying that

$\displaystyle
g_{s}gg_{-s}\Lambda\in\mathcal{O}_{10r_{1}}\Delta^{-1}[0,r_{2}]\subset\Delta^{-1}[0,r_{1}+r_{2}].$

This finishes the proof of (6.12).

Given any $0<r<1$, we now choose ${\theta}_{r}$ as in Lemma 6.6, and then
define our approximating function $\phi_{r}\in C_{c}^{\infty}(X_{d})$ as the
convolution

$\displaystyle\phi_{r}(x):={\theta}_{r}\ast\chi_{\widetilde{\Delta}_{3r/2}}(x)=\int_{G}{\theta}_{r}(g)\chi_{\widetilde{\Delta}_{3r/2}}(g^{-1}x)\,\operatorname{d}\\!{\nu}(g).$

It follows from ${\theta}_{r}\geq 0$ and $\int_{G}{\theta}_{r}\,d\nu=1$ that
$\phi_{r}$ takes values in $[0,1]$. Moreover, for any
$g\in\operatorname{supp}({\theta}_{r})\subset\mathcal{O}_{r/2}$ (so that
$g^{-1}$ is also contained in $\mathcal{O}_{r/2}$ since $\mathcal{O}_{r/2}$ is
invariant under inversion) and for any $x\in\widetilde{\Delta}_{r}$, we have
by (6.12)

$\displaystyle
g^{-1}x\in\mathcal{O}_{r/2}\widetilde{\Delta}_{r}\subset\widetilde{\Delta}_{3r/2},$

implying that for any $x\in\widetilde{\Delta}_{r}$,

$\displaystyle\phi_{r}(x)=\int_{\operatorname{supp}({\theta}_{r})}{\theta}_{r}(g)\chi_{\widetilde{\Delta}_{3r/2}}(g^{-1}x)\,\operatorname{d}\\!{\nu}(g)=\int_{\operatorname{supp}({\theta}_{r})}{\theta}_{r}(g)\,\operatorname{d}\\!{\nu}(g)=1.$

Thus $\chi_{\widetilde{\Delta}_{r}}\leq\phi_{r}$. Next, we claim that
$\operatorname{supp}(\phi_{r})\subset\widetilde{\Delta}_{2r}$. To prove this,
note that since $\operatorname{supp}({\theta}_{r})$ is compact and contained
in ${\mathcal{O}}_{r/2}$, there exists some ${\epsilon}\in(0,r/2)$ such that
$\operatorname{supp}({\theta}_{r})\subset{\mathcal{O}}_{r/2-{\epsilon}}$. Now
if $x\in\operatorname{supp}(\phi_{r})$ then there exists some
$x^{\prime}\in{\mathcal{O}}_{{\epsilon}}\,x$ with $\phi_{r}(x^{\prime})>0$;
and by the definition of $\phi_{r}$ there then exists some
$g\in\operatorname{supp}({\theta}_{r})\subset{\mathcal{O}}_{r/2-{\epsilon}}$
such that $g^{-1}x^{\prime}\in\widetilde{\Delta}_{3r/2}$. Hence
$x^{\prime}\in{\mathcal{O}}_{r/2-{\epsilon}}\widetilde{\Delta}_{3r/2}\subset\widetilde{\Delta}_{2r-{\epsilon}}$,
and (since ${\mathcal{O}}_{{\epsilon}}$ is invariant under inversion)
$x\in{\mathcal{O}}_{{\epsilon}}\,x^{\prime}\subset{\mathcal{O}}_{{\epsilon}}\,\widetilde{\Delta}_{2r-{\epsilon}}\subset\widetilde{\Delta}_{2r}$.
We have thus proved that
$\operatorname{supp}(\phi_{r})\subset\widetilde{\Delta}_{2r}$. Using this
inclusion together with the fact that $\phi_{r}$ takes values in $[0,1]$, we
conclude that $\phi_{r}\leq\chi_{\widetilde{\Delta}_{2r}}$. (Note that
$\phi_{r}\leq\chi_{\widetilde{\Delta}_{2r}}$ follows already from the easier
fact that for any $x^{\prime}\in X_{d}$, $\phi_{r}(x^{\prime})>0$ implies
$x^{\prime}\in\widetilde{\Delta}_{2r}$. However we need some control on
$\operatorname{supp}(\phi_{r})$ below when we discuss derivatives of
$\phi_{r}$.)

For the norm bounds, we first note that using the invariance of the Haar
measure, for any $Y\in\mathfrak{g}$ we have
$\mathcal{D}_{Y}(\phi_{r})=\big{(}\mathcal{D}_{Y}({\theta}_{r})\big{)}\ast\chi_{\widetilde{\Delta}_{3r/2}}$.
More generally, for any monomial $Z$ in $\\{Y_{1},\ldots,Y_{a}\\}$,

(6.13)
$\displaystyle\mathcal{D}_{Z}(\phi_{r})=\big{(}\mathcal{D}_{Z}({\theta}_{r})\big{)}\ast\chi_{\widetilde{\Delta}_{3r/2}}.$

Recall from Lemma 6.6 that
$\operatorname{supp}({\theta}_{r})\subset\mathcal{O}_{r/2}$ and
$\|\mathcal{D}_{Z}({\theta}_{r})\|_{L^{\infty}(G)}\ll_{\ell_{Z},d}r^{1-d^{2}-\ell_{Z}}$,
where $\ell_{Z}$ is the degree of $Z$. Furthermore, it is easily verified that
$\nu(\mathcal{O}_{r/2})\asymp_{d}r^{d^{2}-1}$. Using these facts, we have, for
every $x\in X_{d}$ and every monomial $Z$ of degree $\ell_{Z}\leq\ell$,

(6.14)
$\displaystyle|{\mathcal{D}}_{Z}(\phi_{r})(x)|=\left|\int_{\operatorname{supp}({\theta}_{r})}\mathcal{D}_{Z}({\theta}_{r})(g)\chi_{\widetilde{\Delta}_{3r/2}}(g^{-1}x)\,\operatorname{d}\\!{\nu}(g)\right|\ll_{d,\ell}r^{1-d^{2}-\ell_{Z}}\nu(\mathcal{O}_{r/2})\ll_{d}r^{-\ell_{Z}}.$

Hence, using also
$\operatorname{supp}(\phi_{r})\subset\widetilde{\Delta}_{2r}$ and Theorem 5.1,
we get

$\displaystyle\|\phi_{r}\|_{{H^{\ell}}}\ll_{d,\ell}\mu_{d}(\widetilde{\Delta}_{2r})^{1/2}r^{-\ell}\ll_{d,\varepsilon}r^{\frac{d^{2}}{4}+\frac{d}{4}-1-\ell-\varepsilon}.$

Finally, using the fact that for any $0<r<1$, the support of $\phi_{r}$ is
contained in the fixed precompact set $\widetilde{\Delta}_{2}$, we have
$\|\phi_{r}\|_{\operatorname{Lip}}\ll_{d}\sup_{x\in
X_{d}}\sup_{j\in\\{1,\ldots,a\\}}\left|{\mathcal{D}}_{Y_{j}}(\phi_{r})(x)\right|$,
and hence by (6.14),

$\displaystyle\|\phi_{r}\|_{\operatorname{Lip}}\ll_{d}r^{-1}.$

Now the bound in (6.11) follows, via the definition of the norm
$\mathcal{N}_{\ell}$. ∎

##  7\. Proof of Theorem 1.2

In this section, building on the analysis developed in the previous section,
we give the proof of Theorem 1.2. We keep the notation as in the previous
section. In particular, throughout this section, we fix constants $\delta>0$
and $\ell\in{\mathbb{N}}$ as in Corollary 6.4, and for each $0<r<1$ we fix a
function $\phi_{r}\in C^{\infty}_{c}(X_{d})$ as in Lemma 6.5. Taking
$\varepsilon=1$ in Lemma 6.5 we have that the norm bound in (6.11) holds for
$L:=1+\max\left\\{0,\ell+1-\frac{d^{2}}{4}-\frac{d}{4}\right\\}$.

###  7.1. Application of effective equidistribution

For any $0<r<1$, taking $f=\phi_{r}$ in the effective equidistribution result
(6.9) and applying the norm estimate (6.11), we get for any $s>0$,

(7.1)
$\displaystyle\int_{{\mathcal{Y}}}\phi_{r}(g_{s}\Lambda_{A})\,\operatorname{d}\\!{A}=\mu_{d}(\phi_{r})+O_{d}\left(e^{-\delta
s}r^{-L}\right).$

When $r$ is small, the above integral should be expected to be small as well;
however, the error term in (7.1) blows up as $r\to 0^{+}$. To remedy this
issue, for $r$ very small we will instead prove an upper bound on the
integral, obtained by applying (7.1) for a suitable enlargened $r$-value. The
result is as follows:

######  Lemma 7.1.

Let $\eta:=\frac{\delta}{\varkappa_{d}+L}$. There exists
$r_{0}\in(0,\frac{1}{2})$ such that for any $0<r<r_{0}$ and $s>0$,

(7.2)
$\displaystyle\int_{{\mathcal{Y}}}\phi_{r}(g_{s}\Lambda_{A})\,\operatorname{d}\\!{A}\>\>\begin{cases}{\displaystyle\asymp_{d}r^{\varkappa_{d}}\log^{\lambda_{d}}\bigl{(}\tfrac{1}{r}\bigr{)}}&\textrm{if
$r>e^{-\eta s}$},\\\\[8.0pt] {\displaystyle\ll_{d}e^{-\varkappa_{d}\eta
s/2}}&\textrm{if $r\leq e^{-\eta s}$.}\end{cases}$

In particular, for any sequence
$\\{\rho_{k}\\}_{k\in{\mathbb{N}}}\subset(0,\frac{1}{2})$ with
$\lim_{k\to\infty}\rho_{k}=0$, we have

(7.3)
$\displaystyle\sum_{k}\int_{{\mathcal{Y}}}\phi_{\rho_{k}}(g_{k}\Lambda_{A})\,\operatorname{d}\\!{A}=\infty\quad\Longleftrightarrow\quad\sum_{k}\rho_{k}^{\varkappa_{d}}\log^{\lambda_{d}}\left(\frac{1}{\rho_{k}}\right)=\infty.$

###### Proof.

First we note that by Theorem 5.1 and the relation
$\chi_{\widetilde{\Delta}_{r}}\leq\phi_{r}\leq\chi_{\widetilde{\Delta}_{2r}}$
(see Lemma 6.5), we have
$\mu_{d}(\phi_{r})\asymp_{d}r^{\varkappa_{d}}\log^{\lambda_{d}}\left(\frac{1}{r}\right)$.
Furthermore, if $r>e^{-\eta s}$, then the ratio of the main term and the error
term in (7.1) satisfies:

$\displaystyle\frac{\mu_{d}(\phi_{r})}{e^{-\delta
s}r^{-L}}\asymp_{d}\frac{r^{\varkappa_{d}}\log^{\lambda_{d}}\bigl{(}\tfrac{1}{r}\bigr{)}}{e^{-\delta
s}r^{-L}}>\log^{\lambda_{d}}\bigl{(}\tfrac{1}{r}\bigr{)},$

which we can force to be as large as we like by taking the constant $r_{0}$
sufficiently small (in a way which only depends on $d$). Hence it follows from
(7.1) that (7.2) holds in the case $r>e^{-\eta s}$.

Next assume $r\leq e^{-\eta s}$. Set $\rho:=2e^{-\eta s}$. If $\rho<r_{0}$,
then by what we proved in the previous paragraph,

$\displaystyle\int_{{\mathcal{Y}}}\phi_{\rho}(g_{s}\Lambda_{A})\,\operatorname{d}\\!{A}\asymp_{d}\rho^{\varkappa_{d}}\log^{\lambda_{d}}\bigl{(}\tfrac{1}{\rho}\bigr{)}\ll_{d}e^{-\varkappa_{d}\eta
s/2},$

and hence the bound in (7.2) follows, since
$\phi_{r}\leq\chi_{\widetilde{\Delta}_{2r}}\leq\chi_{\widetilde{\Delta}_{\rho}}\leq\phi_{\rho}$
(again see Lemma 6.5). In the remaining case when $\rho\geq r_{0}$, we have
$s\ll_{d}1$ and $e^{-\varkappa_{d}\eta s/2}\gg_{d}1$, and hence the bound in
(7.2) holds simply because of $\phi_{r}\leq 1$. This completes the proof of
(7.2).

For the last part of the lemma, since $\lim_{k\to\infty}\rho_{k}=0$, after
possibly deleting finitely many terms from the two sums in (7.3), we may
assume $\rho_{k}<r_{0}$ for all appearing terms. Next, using the second bound
in (7.2) and the fact that both of the series $\sum_{k}e^{-\varkappa_{d}\eta
k/2}$ and $\sum_{\rho_{k}\leq e^{-\eta
k}}\rho_{k}^{\varkappa_{d}}\log^{\lambda_{d}}\left(\frac{1}{\rho_{k}}\right)$
converge, it follows that the two divergence statements in (7.3) remain
unaffected if all the terms for which $\rho_{k}\leq e^{-\eta k}$ are removed
from the respective series. After this operation, the equivalence in (7.3) is
an immediate consequence of the first relation in (7.2). ∎

###  7.2. The convergence case

This case is now easily handled using Lemma 7.1.

###### Proof of the convergence case of Theorem 1.2.

Let $r=r_{\psi}:[s_{0},\infty)\to(0,\infty)$ be the continuous, decreasing
function corresponding to $\psi$ as in Lemma 6.1. First note that since the
series (1.10) converges, we have $\lim_{t\to\infty}t\psi(t)=1$ (or
equivalently, $\lim_{s\to\infty}r(s)=0$ as seen from the proof of Lemma 6.1).
Moreover, by the last part of Lemma 6.1, the fact that the series (1.10)
converges implies that the series
$\sum_{k}r(k)^{\varkappa_{d}}\log^{\lambda_{d}}\bigl{(}1+\tfrac{1}{r(k)}\bigr{)}$
also converges.

Now for each $k>s_{0}$ let us define

$\displaystyle\overline{B}_{k}:=\bigcup_{0\leq
s<1}g_{-s}\Delta^{-1}[0,\omega_{1}r(k+s)]\quad\textrm{and}\quad\overline{E}_{k}:=\left\\{\Lambda_{A}\in\mathcal{Y}\>:\>g_{k}\Lambda_{A}\in\overline{B}_{k}\right\\},$

where $\omega_{1}:=\max\\{m\alpha_{i},n\beta_{j}\>:\>1\leq i\leq m,1\leq j\leq
n\\}$ is as in Proposition 6.2. In view of Proposition 6.2 (and the paragraph
after it), it suffices to show that for Leb-a.e. $\Lambda_{A}\in\mathcal{Y}$,
$g_{k}\Lambda_{A}\in\overline{B}_{k}$ for only finitely many $k>s_{0}$, or
equivalently, that the limsup set $\limsup_{k\to\infty}\overline{E}_{k}$ is of
zero measure. Thus in view of the Borel-Cantelli lemma, it suffices to show
that $\sum_{k}\textrm{Leb}(\overline{E}_{k})<\infty$.

To prove this, we will approximate the shrinking targets
$\\{\overline{B}_{k}\\}_{k>s_{0}}$ from above. Since $r(s)\to 0$ as
$s\to\infty$, by enlarging $s_{0}$ if necessary (equivalently, enlarging
$t_{0}$ as in Lemma 6.1), we may assume $\omega_{1}r(s)\in(0,\frac{1}{2})$ for
all $s>s_{0}$. Moreover, by Lemma 6.1, $r(\cdot)$ is decreasing; thus with
$\rho_{k}:=\omega_{1}r(k)$, we have

(7.4)
$\displaystyle\overline{B}_{k}\subset\widetilde{\Delta}_{\rho_{k}},\qquad\forall\,k>s_{0}.$

Recall that for each $0<r<1$ we have fixed a function $\phi_{r}\in
C^{\infty}_{c}(X_{d})$ as in Lemma 6.5. Now for each $k>s_{0}$, we have
$\chi_{\overline{B}_{k}}\leq\chi_{\widetilde{\Delta}_{\rho_{k}}}\leq\phi_{\rho_{k}}$
(by (7.4) and Lemma 6.5), implying that

(7.5) $\displaystyle\textrm{Leb}(\overline{E}_{k})$
$\displaystyle=\int_{\mathcal{Y}}\chi_{\overline{B}_{k}}(g_{k}\Lambda_{A})\,\operatorname{d}\\!{A}\leq\int_{\mathcal{Y}}\phi_{\rho_{k}}(g_{k}\Lambda_{A})\,\operatorname{d}\\!{A}.$

Next, it follows from $\rho_{k}=\omega_{1}r(k)$ and the convergence of
$\sum_{k}r(k)^{\varkappa_{d}}\log^{\lambda_{d}}\bigl{(}1+\tfrac{1}{r(k)}\bigr{)}$
that the series
$\sum_{k}\rho_{k}^{\varkappa_{d}}\log^{\lambda_{d}}\bigl{(}\tfrac{1}{\rho_{k}}\bigr{)}$
also converges; in addition, $\lim_{k\to\infty}\rho_{k}=0$ since
$\lim_{k\to\infty}r(k)=0$. Hence by the last part of Lemma 7.1 combined with
(7.5), we have $\sum_{k}\textrm{Leb}\left(\overline{E}_{k}\right)<\infty$,
finishing the proof. ∎

######  Remark 14.

Let $(m,n)=(2,1)$; thus $d=3$. In [6, Theorem 1.1], Chow and Yang proved an
effective equidistribution result for certain Diophantine lines in
${\mathcal{Y}}$ translated under the full (two dimensional) diagonal subgroup
of $G$ along certain restricted directions. In particular, their result
implies the following: Let $(a,b)\in{\mathbb{R}}^{2}$ be a Diophantine vector
(see [6, p. 2] for the definition), and let $J\subset{\mathbb{R}}$ be a
compact subinterval. Then these exist constants
$\ell^{\prime}\in{\mathbb{N}}$, $c\in(0,1)$ and $\delta^{\prime}>0$ such that
for any pair of weights
$\bm{\alpha}=(\frac{c^{\prime}}{1+c^{\prime}},\frac{1}{1+c^{\prime}})$ with
$0<c^{\prime}\leq c$, for any $f\in C_{c}^{\infty}(X_{3})$ and for any $s>0$

(7.6)
$\displaystyle\frac{1}{|J|}\int_{J}f\left(g_{s}\Lambda_{\bm{v}(x)}\right)\operatorname{d}\\!{x}=\mu_{3}(f)+O\left(e^{-\delta^{\prime}s}\|f\|_{L^{\infty}_{\ell^{\prime}}}\right),$

where
$g_{s}=g^{\bm{\alpha},1}_{s}=\operatorname{diag}(e^{\alpha_{1}s},e^{\alpha_{2}s},e^{-s})$
with $\bm{\alpha}$ as above, $\bm{v}(x):=(ax+b,x)^{t}\in{\mathbb{R}}^{2}$ and
$\|\cdot\|_{L^{\infty}_{\ell^{\prime}}}$ the “$L^{\infty}$, degree
$\ell^{\prime}$” Sobolev norm defined by

$\displaystyle\|f\|_{L^{\infty}_{\ell^{\prime}}}:=\sum_{\deg(Z)\leq\ell^{\prime}}\|\mathcal{D}_{Z}(f)\|_{C^{0}}.$

Here $\|\cdot\|_{C^{0}}$ is the uniform norm on $C^{\infty}_{c}(X_{3})$ as
before. On the other hand, it is easy to see from Lemma 6.6 and the relation
(6.13) that there exists $L^{\prime}>1$ such that

(7.7)
$\displaystyle\|\phi_{r}\|_{L^{\infty}_{\ell^{\prime}}}\ll_{\ell^{\prime}}r^{-L^{\prime}},\qquad\forall\
0<r<1.$

Using a similar analysis with (7.6) and (7.7) in place of (6.9) and (6.11)
respectively, we can conclude that if the series (1.10) (with $d=3$)
converges, then for Leb-a.e. $x\in J$ the column vector
$\bm{v}(x)=(ax+b,x)^{t}$ is $\psi_{\bm{\alpha},1}$-Dirichlet.

###  7.3. The divergence case

In this subsection we prove the divergence case of Theorem 1.2. We first
record from [30] the following divergence Borel-Cantelli lemma which we will
use.

######  Lemma 7.2.

Let $(\mathcal{X},\mu)$ be a probability space. Let
$\\{h_{k}\\}_{k\in{\mathbb{N}}}$ be a sequence of measurable functions on
$\mathcal{X}$ taking values in $[0,1]$. Let $b_{k}:=\mu(h_{k})$. Suppose
$\sum_{k}b_{k}=\infty$ and

(7.8)
$\displaystyle\liminf_{k_{2}\to\infty}\frac{\int_{\mathcal{X}}\left(\sum_{i=k_{1}}^{k_{2}}h_{i}(x)-\sum_{i=k_{1}}^{k_{2}}b_{i}\right)^{2}\operatorname{d}\\!{\mu}(x)}{\left(\sum_{i=k_{1}}^{k_{2}}b_{i}\right)^{2}}=0\quad\textrm{for
some $k_{1}\in{\mathbb{N}}$}.$

Then for $\mu$-a.e. $x\in\mathcal{X}$, $h_{k}(x)>0$ infinitely often.

###### Proof.

Let $Y_{1},Y_{2},\ldots$ be the sequence of random variables defined by
$Y_{k}(x)=\sum_{i=k_{1}}^{k+k_{1}-1}h_{i}(x)$ ($k\in{\mathbb{N}}$). Note that

$\displaystyle\int_{\mathcal{X}}\biggl{(}\sum_{i=k_{1}}^{k_{2}}h_{i}(x)-\sum_{i=k_{1}}^{k_{2}}b_{i}\biggr{)}^{2}\,\operatorname{d}\\!{\mu}(x)=\mu\bigl{(}Y_{k_{2}-k_{1}+1}^{2}\bigr{)}-\mu\bigl{(}Y_{k_{2}-k_{1}+1}\bigr{)}^{2};$

hence (7.8) implies that
$\limsup_{k\to\infty}\mu(Y_{k})^{2}/\mu(Y_{k}^{2})=1$. Therefore by part (iii)
of the main theorem in [30], for $\mu$-a.e. $x$ we have
$\limsup_{k\to\infty}Y_{k}(x)/\mu(Y_{k})>0$. Also
$\mu(Y_{k})=\sum_{i=k_{1}}^{k+k_{1}-1}b_{i}\to\infty$ as $k\to\infty$. Hence
it follows that for $\mu$-a.e. $x$ we have
$\lim_{k\to\infty}Y_{k}(x)=+\infty$, and in particular $h_{i}(x)>0$ for
infinitely many $i$. ∎

######  Remark 15.

If one replaces the assumption (7.8) by the weaker assumption that

$\displaystyle\liminf_{k_{2}\to\infty}\frac{\int_{\mathcal{X}}\left(\sum_{i=k_{1}}^{k_{2}}h_{i}(x)-\sum_{i=k_{1}}^{k_{2}}b_{i}\right)^{2}\operatorname{d}\\!{\mu}(x)}{\left(\sum_{i=k_{1}}^{k_{2}}b_{i}\right)^{2}}=:C<\infty\quad\textrm{for
some $k_{1}\in{\mathbb{N}}$},$

then by the application of part (iii) of the main theorem in [30] we get
instead

$\mu\left(\left\\{x\in
X\>:\>\limsup_{k\to\infty}\frac{Y_{k}(x)}{\mu(Y_{k})}>0\right\\}\right)\geq\frac{1}{1+C}.$

In particular, there is a positive measure set of $x\in X$ such that
$h_{i}(x)>0$ infinitely often.

###### Proof of the divergence case of Theorem 1.2.

First we note that in view of Remark 2 we may assume
$\lim_{t\to\infty}t\psi(t)=1$. Let $r=r_{\psi}$ be the continuous, decreasing
function corresponding to $\psi$ as in Lemma 6.1; then from the proof of that
lemma we have $\lim_{s\to\infty}r(s)=0$. Also by Lemma 6.1, since the series
(1.10) diverges, the series
$\sum_{k}r(k)^{\varkappa_{d}}\log^{\lambda_{d}}\bigl{(}1+\tfrac{1}{r(k)}\bigr{)}$
also diverges. Moreover, by Remark 12, condition (1.11) is equivalent to

(7.9) $\displaystyle\liminf_{s_{1}\to\infty}\frac{\sum_{s_{0}<k\leq
s_{1}}r(k)^{\varkappa_{d}}\log^{\lambda_{d}+1}\bigl{(}1+\frac{1}{r(k)}\bigr{)}}{\Bigl{(}\sum_{s_{0}<k\leq
s_{1}}r(k)^{\varkappa_{d}}\log^{\lambda_{d}}\bigl{(}1+\frac{1}{r(k)}\bigr{)}\Bigr{)}^{2}}=0.$

Now for any $k>s_{0}$ let

$\displaystyle\underline{B}_{k}=\bigcup_{0\leq
s<1}g_{-s}\Delta^{-1}[0,\omega_{2}r(k+s)]$

be as before with $\omega_{2}:=\min\\{m\alpha_{i},n\beta_{j}\>:\>1\leq i\leq
m,1\leq j\leq n\\}$ as in Proposition 6.2. In view of Proposition 6.2 (and the
paragraph after it) it suffices to show that for $\operatorname{Leb}$-a.e.
$\Lambda_{A}\in\mathcal{Y}$, $g_{k}\Lambda_{A}\in\underline{B}_{k}$ infinitely
often.

In this case we will approximate the shrinking targets
$\left\\{\underline{B}_{k}\right\\}_{k>s_{0}}$ from below. Recall that for
each $0<r<1$ we have fixed a function $\phi_{r}\in C^{\infty}_{c}(X_{d})$ as
in Lemma 6.5. Again we may assume $\omega_{2}r(s)\in(0,\frac{1}{2})$ for all
$s>s_{0}$, and since $r(\cdot)$ is decreasing, we have
$\widetilde{\Delta}_{\omega_{2}r(k+1)}\subset\underline{B}_{k}$, implying
that, with $\rho_{k}:=\omega_{2}r(k+1)/2$:

(7.10) $\displaystyle
f_{k}:=\phi_{\rho_{k}}\leq\chi_{\widetilde{\Delta}_{\omega_{2}r(k+1)}}\leq\chi_{\underline{B}_{k}},\qquad\forall\,k>s_{0}.$

Let us set, for each $k>s_{0}$,

$b_{k}:=\int_{{\mathcal{Y}}}f_{k}(g_{k}\Lambda_{A})\,\operatorname{d}\\!{A}=\int_{{\mathcal{Y}}}\phi_{\rho_{k}}(g_{k}\Lambda_{A})\,\operatorname{d}\\!{A}.$

Then, similarly to the proof of the convergence case, by applying the last
part of Lemma 7.1 and using the relation $\rho_{k}=\omega_{2}r(k+1)/2$ and the
facts that the series
$\sum_{k}r(k)^{\varkappa_{d}}\log^{\lambda_{d}}\bigl{(}1+\tfrac{1}{r(k)}\bigr{)}$
diverges and $\lim_{s\to\infty}r(s)=0$, it follows that the series
$\sum_{k}b_{k}$ also diverges.

Now for each $k>s_{0}$, let $h_{k}$ be the function on $\mathcal{Y}$ defined
by $h_{k}(\Lambda_{A}):=f_{k}(g_{k}\Lambda_{A})$. Then in view of the
definition of $f_{k}:=\phi_{\rho_{k}}$ and the relation
$\chi_{\widetilde{\Delta}_{\rho_{k}}}\leq\phi_{\rho_{k}}\leq\chi_{\widetilde{\Delta}_{2\rho_{k}}}$,
the function $h_{k}$ takes values in $[0,1]$, and

$\displaystyle\int_{\mathcal{Y}}h_{k}(\Lambda_{A})\,\operatorname{d}\\!{A}=\int_{\mathcal{Y}}f_{k}(g_{k}\Lambda_{A})\,\operatorname{d}\\!{A}=b_{k}.$

We will apply Lemma 7.2 to the probability space $(\mathcal{Y},\textrm{Leb})$
and the sequence $\\{h_{k}\\}_{k>s_{0}}$. We have already seen that
$\sum_{k}b_{k}=\infty$; thus in view of Lemma 7.2 it suffices to show that
$\\{h_{k}\\}_{k>s_{0}}$ satisfies condition (7.8).

Let us take $C>0$ sufficiently large so that for all $k>C$,
$\rho_{k}\in(0,r_{0})$, where $r_{0}$ is the constant as in Lemma 7.1. For any
$k_{2}>k_{1}>C$, let us denote

$\displaystyle Q_{k_{1},k_{2}}$
$\displaystyle:=\int_{\mathcal{Y}}\left(\sum_{i=k_{1}}^{k_{2}}h_{i}(\Lambda_{A})-\sum_{i=k_{1}}^{k_{2}}b_{i}\right)^{2}\,\operatorname{d}\\!{A}=\sum_{k_{1}\leq
i,j\leq
k_{2}}\int_{\mathcal{Y}}\left(h_{i}(\Lambda_{A})h_{j}(\Lambda_{A})-b_{i}b_{j}\right)\,\operatorname{d}\\!{A}.$

Using the fact that for each $k_{1}\leq i\leq k_{2}$,

$\int_{\mathcal{Y}}\left(h^{2}_{i}(\Lambda_{A})-b_{i}^{2}\right)\,\operatorname{d}\\!{A}\leq\int_{\mathcal{Y}}h_{i}(\Lambda_{A})\,\operatorname{d}\\!{A}=b_{i},$

we have

$\displaystyle Q_{k_{1},k_{2}}$
$\displaystyle\leq\sum_{i=k_{1}}^{k_{2}}b_{i}+2\sum_{k_{1}\leq i<j\leq
k_{2}}\int_{\mathcal{Y}}\left(h_{i}(\Lambda_{A})h_{j}(\Lambda_{A})-b_{i}b_{j}\right)\,\operatorname{d}\\!{A}.$

Fix $k_{1}\leq i<j\leq k_{2}$; we will use two different estimates for the
term
$\int_{\mathcal{Y}}\big{(}h_{i}(\Lambda_{A})h_{j}(\Lambda_{A})-b_{i}b_{j}\big{)}\,\operatorname{d}\\!{A}$
depending on whether $\min\\{i,j-i\\}$ is large or small. First, applying the
effective doubly mixing (6.10) to the pair $(f_{i},f_{j})$, we get

(7.11)
$\displaystyle\int_{\mathcal{Y}}f_{i}(g_{i}\Lambda_{A})f_{j}(g_{j}\Lambda_{A})\,\operatorname{d}\\!{A}=\mu_{d}(f_{i})\mu_{d}(f_{j})+O\left(e^{-\delta\min\\{i,j-i\\}}\mathcal{N}_{\ell}(f_{i})\mathcal{N}_{\ell}(f_{j})\right).$

On the other hand, by (6.9) we have

(7.12) $\displaystyle b_{k}=\mu_{d}(f_{k})+O\left(e^{-\delta
k}\mathcal{N}_{\ell}(f_{k})\right),\quad\forall\ k>C.$

Combining (7.12), (7.11), the norm estimate
${\mathcal{N}}_{\ell}(f_{k})={\mathcal{N}}_{\ell}(\phi_{\rho_{k}})\ll_{d}\rho_{k}^{-L}$
(by (6.11)) and noting that
$\int_{{\mathcal{Y}}}h_{i}(\Lambda_{A})h_{j}(\Lambda_{A})\,\operatorname{d}\\!{A}=\int_{\mathcal{Y}}f_{i}(g_{i}\Lambda_{A})f_{j}(g_{j}\Lambda_{A})\,\operatorname{d}\\!{A}$,
we have

$\displaystyle\left|\int_{{\mathcal{Y}}}\left(h_{i}(\Lambda_{A})h_{j}(\Lambda_{A})-b_{i}b_{j}\right)\,\operatorname{d}\\!{A}\right|\ll_{d}e^{-\delta\min\\{i,j-i\\}}\rho_{i}^{-L}\rho_{j}^{-L}<e^{-\delta\min\\{i,j-i\\}}\rho_{j}^{-2L}.$

On the other hand, using the trivial estimate $|h_{i}h_{j}-b_{i}b_{j}|\leq
h_{i}h_{j}+b_{i}b_{j}\leq h_{j}+b_{j}$ we have

(7.13)
$\displaystyle\biggl{|}\int_{\mathcal{Y}}\left(h_{i}(\Lambda_{A})h_{j}(\Lambda_{A})-b_{i}b_{j}\right)\,\operatorname{d}\\!{A}\biggr{|}$
$\displaystyle\leq\int_{\mathcal{Y}}\left(h_{j}(\Lambda_{A})+b_{j}\right)\,\operatorname{d}\\!{A}=2b_{j}.$

Combining these two bounds, we conclude:

(7.14) $\displaystyle
Q_{k_{1},k_{2}}\ll_{d}\sum_{i=k_{1}}^{k_{2}}b_{i}+\sum_{j=k_{1}+1}^{k_{2}}\sum_{i=k_{1}}^{j-1}\min\left\\{e^{-\delta\min\\{i,j-i\\}}\rho_{j}^{-2L},b_{j}\right\\}.$

In order to bound the above inner sum we replace $k_{1}$ by $1$ and use the
symmetry $i\mapsto j-i$ to get

(7.15)
$\displaystyle\sum_{i=k_{1}}^{j-1}\min\left\\{e^{-\delta\min\\{i,j-i\\}}\rho_{j}^{-2L},b_{j}\right\\}\leq
2\sum_{i=1}^{j-1}\min\left\\{e^{-\delta\,i}\rho_{j}^{-2L},b_{j}\right\\}.$

In the last sum, all terms are $\leq b_{j}$, and there are at most
$O_{d}\bigl{(}\log\bigl{(}2+b_{j}^{-1}\rho_{j}^{-2L}\bigr{)}\bigr{)}$ terms
which are equal to $b_{j}$ (indeed, remember that $\delta$ depends only on
$d$). Furthermore, if there are any terms which are less than $b_{j}$, then
these are bounded above by $b_{j},b_{j}e^{-\delta},b_{j}e^{-2\delta},\ldots$,
and so their sum is $O_{d}(b_{j})$. It follows that the last sum in (7.15) is
$O_{d}\left(b_{j}\min\left\\{j,\log\bigl{(}2+b_{j}^{-1}\rho_{j}^{-2L}\bigr{)}\right\\}\right)$,
and hence from (7.14) we get

$\displaystyle
Q_{k_{1},k_{2}}\ll_{d}\sum_{i=k_{1}}^{k_{2}}b_{i}+\sum_{j=k_{1}+1}^{k_{2}}b_{j}\min\left\\{j,\log\bigl{(}2+b_{j}^{-1}\rho_{j}^{-2L}\bigr{)}\right\\}\ll\sum_{j=k_{1}}^{k_{2}}b_{j}\min\left\\{j,\log\bigl{(}2+b_{j}^{-1}\rho_{j}^{-2L}\bigr{)}\right\\}.$

Let $\eta=\frac{\delta}{\varkappa_{d}+L}$ be as in Lemma 7.1 and set
$\kappa:=\frac{\varkappa_{d}\eta}{2}$. Then by (7.2) we have for each
$k_{1}\leq j\leq k_{2}$:

(7.16) $\displaystyle
b_{j}=\int_{{\mathcal{Y}}}\phi_{\rho_{j}}(g_{j}\Lambda_{A})\,\operatorname{d}\\!{A}\>\>\begin{cases}{\displaystyle\asymp_{d}\rho_{j}^{\varkappa_{d}}\log^{\lambda_{d}}\bigl{(}\tfrac{1}{\rho_{j}}\bigr{)}}&\textrm{if
$\rho_{j}>e^{-\eta j}$,}\\\\[8.0pt] {\displaystyle\ll_{d}e^{-\kappa
j}}&\textrm{if $\rho_{j}\leq e^{-\eta j}$.}\end{cases}$

Thus for any $k_{2}>k_{1}>C$ we have (recalling that
$\rho_{j}=\omega_{2}r(j+1)/2$)

$\displaystyle Q_{k_{1},k_{2}}$
$\displaystyle\ll_{d}\sum_{\begin{subarray}{c}k_{1}\leq j\leq k_{2}\\\
(\rho_{j}\leq e^{-\eta j})\end{subarray}}e^{-\kappa
j}j+\sum_{\begin{subarray}{c}k_{1}\leq j\leq k_{2}\\\ (\rho_{j}>e^{-\eta
j})\end{subarray}}\rho_{j}^{\varkappa_{d}}\log^{\lambda_{d}}\Bigl{(}\frac{1}{\rho_{j}}\Bigr{)}\log\left(2+\rho_{j}^{-\varkappa_{d}-2L}\log^{-\lambda_{d}}\Bigl{(}\frac{1}{\rho_{j}}\Bigr{)}\right)\hskip
40.0pt$
$\displaystyle\ll_{d}1+\sum_{j=k_{1}}^{k_{2}}\rho_{j}^{\varkappa_{d}}\log^{\lambda_{d}+1}\Bigl{(}\frac{1}{\rho_{j}}\Bigr{)}\ll
1+\sum_{j=k_{1}+1}^{k_{2}+1}r(j)^{\varkappa_{d}}\log^{\lambda_{d}+1}\Bigl{(}1+\frac{1}{r(j)}\Bigr{)}.$

Similarly, by (7.16), for any fixed $k_{1}>C$ we have as $k_{2}\to\infty$:

$\displaystyle\sum_{j=k_{1}}^{k_{2}}b_{j}\gg_{d}\sum_{\begin{subarray}{c}k_{1}\leq
j\leq k_{2}\\\ (\rho_{j}>e^{-\eta
j})\end{subarray}}\rho_{j}^{\varkappa_{d}}\log^{\lambda_{d}}\Bigl{(}\frac{1}{\rho_{j}}\Bigr{)}\gg\sum_{j=k_{1}}^{k_{2}}\rho_{j}^{\varkappa_{d}}\log^{\lambda_{d}}\Bigl{(}\frac{1}{\rho_{j}}\Bigr{)}\gg_{\omega_{2}}\sum_{j=k_{1}+1}^{k_{2}+1}r(j)^{\varkappa_{d}}\log^{\lambda_{d}}\Bigl{(}1+\frac{1}{r(j)}\Bigr{)},$

where the second relation holds since the series
$\sum_{j=k_{1}}^{\infty}\rho_{j}^{\varkappa_{d}}\log^{\lambda_{d}}\bigl{(}\frac{1}{\rho_{j}}\bigr{)}$
diverges (this follows from the relation $\rho_{j}=\omega_{2}r(j+1)/2$ and the
fact that the series
$\sum_{j}r(j)^{\varkappa_{d}}\log^{\lambda_{d}}\bigl{(}1+\tfrac{1}{r(j)}\bigr{)}$
diverges), while the same sum restricted to those $j$ for which $\rho_{j}\leq
e^{-\eta j}$ is convergent.

Combining the last two bounds, we conclude that for any $k_{1}>C$, and for
$k_{2}$ sufficiently large,

(7.17)
$\displaystyle\frac{Q_{k_{1},k_{2}}}{\left(\sum_{j=k_{1}}^{k_{2}}b_{j}\right)^{2}}\ll_{d,\omega_{2}}\frac{1+\sum_{j=k_{1}+1}^{k_{2}+1}r(j)^{\varkappa_{d}}\log^{\lambda_{d}+1}\bigl{(}1+\frac{1}{r(j)}\bigr{)}}{\Bigl{(}\sum_{j=k_{1}+1}^{k_{2}+1}r(j)^{\varkappa_{d}}\log^{\lambda_{d}}\bigl{(}1+\frac{1}{r(j)}\bigr{)}\Bigr{)}^{2}}.$

Since the series
$\sum_{j}r(j)^{\varkappa_{d}}\log^{\lambda_{d}}\bigl{(}1+\frac{1}{r(j)}\bigr{)}$
diverges, condition (7.9) implies that the limit inferior of the expression in
the right hand side of (7.17) tends to zero as $k_{2}\to\infty$. Hence (7.8)
holds. We have also noted that $\sum_{k}b_{k}=\infty$. Hence by Lemma 7.2, for
Leb-a.e. $A\in M_{m,n}({\mathbb{R}}/{\mathbb{Z}})$ we have
$h_{k}(\Lambda_{A})=f_{k}(g_{k}\Lambda_{A})>0$ infinitely often. Together with
(7.10), this implies that for Leb-a.e. $A\in
M_{m,n}({\mathbb{R}}/{\mathbb{Z}})$, the lattice $g_{k}\Lambda_{A}$ belongs to
$\operatorname{supp}(f_{k})\subset\underline{B}_{k}$ for infinitely many
$k\in{\mathbb{N}}$. This finishes the proof. ∎

######  Remark 16.

For the divergence case in Theorem 1.2, we note that if one replaces the
assumption (1.11) by the weaker assumption that

$\displaystyle\liminf_{t_{1}\to\infty}\frac{\sum_{t_{0}\leq k\leq
t_{1}}k^{-1}F_{\psi}(k)^{\varkappa_{d}}\log^{\lambda_{d}+1}\left(\frac{1}{F_{\psi}(k)}\right)}{\left(\sum_{t_{0}\leq
k\leq
t_{1}}k^{-1}F_{\psi}(k)^{\varkappa_{d}}\log^{\lambda_{d}}\Bigl{(}\frac{1}{F_{\psi}(k)}\Bigr{)}\right)^{2}}<\infty,$

then, in view of Remark 12, Remark 15 and the estimate (7.17), we can conclude
that, under this weaker assumption,
$\operatorname{\mathbf{DI}}_{\bm{\alpha},\bm{\beta}}(\psi)^{c}$ is of positive
Lebesgue measure. It is an interesting question, to which we do not know the
answer, whether $\operatorname{\mathbf{DI}}_{\bm{\alpha},\bm{\beta}}(\psi)$
must always be of zero or full Lebesgue measure. See [1] for an analogous
question in the setting of asymptotic approximation.

## References

  * [1] V. Beresnevich and S. Velani.  A note on zero-one laws in metrical Diophantine approximation.  Acta Arith., 133(4):363–374, 2008.
  * [2] M. Björklund and A. Gorodnik.  Central limit theorems for Diophantine approximants.  Math. Ann., 374(3-4):1371–1437, 2019.
  * [3] J. W. S. Cassels.  An introduction to Diophantine approximation.  Cambridge Tracts in Mathematics and Mathematical Physics, No. 45. Cambridge University Press, New York, 1957.
  * [4] Y. Cheung.  Hausdorff dimension of the set of singular pairs.  Ann. of Math. (2), 173(1):127–167, 2011.
  * [5] Y. Cheung and N. Chevallier.  Hausdorff dimension of singular vectors.  Duke Math. J., 165(12):2273–2329, 2016.
  * [6] S. Chow and L. Yang.  An effective ratner equidistribution theorem for multiplicative Diophantine approximation on planar lines.  arXiv preprint arXiv:1902.06081, 2019.
  * [7] S. G. Dani.  Divergent trajectories of flows on homogeneous spaces and Diophantine approximation.  J. Reine Angew. Math., 359:55–89, 1985.
  * [8] T. Das, L. Fishman, D. Simmons, and M. Urbański.  A variational principle in the parametric geometry of numbers, with applications to metric Diophantine approximation.  C. R. Math. Acad. Sci. Paris, 355(8):835–846, 2017.
  * [9] T. Das, L. Fishman, D. Simmons, and M. Urbański.  A variational principle in the parametric geometry of numbers.  arXiv preprint arXiv:1901.06602, 2019.
  * [10] H. Davenport and W. M. Schmidt.  Dirichlet’s theorem on diophantine approximation. II.  Acta Arith., 16:413–424, 1969/70.
  * [11] H. Davenport and W. M. Schmidt.  Dirichlet’s theorem on diophantine approximation.  In Symposia Mathematica, Vol. IV (INDAM, Rome, 1968/69), pages 113–132. Academic Press, London, 1970.
  * [12] G. Hajós.  Über einfache und mehrfache Bedeckung des $n$-dimensionalen Raumes mit einem Würfelgitter.  Math. Z., 47:427–467, 1941.
  * [13] M. Hussain, D. Y. Kleinbock, N. Wadleigh, and B. Wang.  Hausdorff measure of sets of Dirichlet non-improvable numbers.  Mathematika, 64(2):502–518, 2018.
  * [14] S. Kadyrov, D. Y. Kleinbock, E. Lindenstrauss, and G. A. Margulis.  Singular systems of linear forms and non-escape of mass in the space of lattices.  J. Anal. Math., 133:253–277, 2017.
  * [15] O. Khalil and M. Luethi.  Random walks, spectral gaps, and Khintchine’s theorem on fractals.  arXiv preprint arXiv:2101.05797, 2021.
  * [16] T. Kim and W. Kim.  Hausdorff measure of sets of Dirichlet non-improvable affine forms.  Adv. Math., 403:Paper No. 108353, 2022.
  * [17] D. Y. Kleinbock.  Flows on homogeneous spaces and Diophantine properties of matrices.  Duke Math. J., 95(1):107–124, 1998.
  * [18] D. Y. Kleinbock, N. de Saxcé, N. A. Shah, and P. Yang.  Equidistribution in the space of $3$-lattices and Dirichlet-improvable vectors on planar lines.  arXiv preprint arXiv:2106.08860, 2021.
  * [19] D. Y. Kleinbock and G. A. Margulis.  Bounded orbits of nonquasiunipotent flows on homogeneous spaces.  In Sinaĭ’s Moscow Seminar on Dynamical Systems, volume 171 of Amer. Math. Soc. Transl. Ser. 2, pages 141–172. Amer. Math. Soc., Providence, RI, 1996.
  * [20] D. Y. Kleinbock and G. A. Margulis.  Logarithm laws for flows on homogeneous spaces.  Invent. Math., 138(3):451–494, 1999.
  * [21] D. Y. Kleinbock and G. A. Margulis.  On effective equidistribution of expanding translates of certain orbits in the space of lattices.  In Number theory, analysis and geometry, pages 385–396. Springer, New York, 2012.
  * [22] D. Y. Kleinbock and G. A. Margulis.  Erratum to: Logarithm laws for flows on homogeneous spaces.  Invent. Math., 211(2):855–862, 2018.
  * [23] D. Y. Kleinbock and S. Mirzadeh.  On the dimension drop conjecture for diagonal flows on the space of lattices.  arXiv preprint arXiv:2010.14065, 2020.
  * [24] D. Y. Kleinbock and A. Rao.  A zero-one law for uniform Diophantine approximation in Euclidean norm.  Int. Math. Res. Not. IMRN, (8):5617–5657, 2022.
  * [25] D. Y. Kleinbock, R. Shi, and B. Weiss.  Pointwise equidistribution with an error rate and with respect to unbounded functions.  Math. Ann., 367(1-2):857–879, 2017.
  * [26] D. Y. Kleinbock and N. Wadleigh.  A zero-one law for improvements to Dirichlet’s Theorem.  Proc. Amer. Math. Soc., 146(5):1833–1844, 2018.
  * [27] D. Y. Kleinbock and N. Wadleigh.  An inhomogeneous Dirichlet theorem via shrinking targets.  Compos. Math., 155(7):1402–1423, 2019.
  * [28] D. Y. Kleinbock and B. Weiss.  Dirichlet’s theorem on Diophantine approximation and homogeneous flows.  J. Mod. Dyn., 2(1):43–62, 2008.
  * [29] D. Y. Kleinbock and S. Yu.  A dynamical Borel–Cantelli lemma via improvements to Dirichlet’s theorem.  Mosc. J. Comb. Number Theory, 9(2):101–122, 2020.
  * [30] S. Kochen and C. Stone.  A note on the Borel-Cantelli lemma.  Illinois J. Math., 8:248–251, 1964.
  * [31] G. A. Margulis.  On some aspects of the theory of Anosov systems.  Springer Monographs in Mathematics. Springer-Verlag, Berlin, 2004.  With a survey by Richard Sharp: Periodic orbits of hyperbolic flows, Translated from the Russian by Valentina Vladimirovna Szulikowska.
  * [32] M. Ratner.  On Raghunathan’s measure conjecture.  Ann. of Math. (2), 134(3):545–607, 1991.
  * [33] N. A. Shah.  Equidistribution of expanding translates of curves and Dirichlet’s theorem on Diophantine approximation.  Invent. Math., 177(3):509–532, 2009.
  * [34] N. A. Shah.  Expanding translates of curves and Dirichlet-Minkowski theorem on linear forms.  J. Amer. Math. Soc., 23(2):563–589, 2010.
  * [35] N. A. Shah and L. Yang.  Equidistribution of curves in homogeneous spaces and Dirichlet’s approximation theorem for matrices.  Discrete Contin. Dyn. Syst., 40(9):5247–5287, 2020.
  * [36] S. K. Stein and S. Szabó.  Algebra and tiling, volume 25 of Carus Mathematical Monographs.  Mathematical Association of America, Washington, DC, 1994.  Homomorphisms in the service of geometry.
  * [37] A. Strömbergsson and A. Venkatesh.  Small solutions to linear congruences and Hecke equidistribution.  Acta Arith., 118(1):41–78, 2005.
  * [38] P. Yang.  Equidistribution of expanding translates of curves and Diophantine approximation on matrices.  Invent. Math., 220(3):909–948, 2020.


# Learning End-to-end Autonomous Driving using Guided Auxiliary Supervision

Ashish Mehta
Intel Corporation
<EMAIL_ADDRESS>
Adithya Subramanian
Intel Corporation
<EMAIL_ADDRESS>
Anbumani Subramanian
Intel Corporation
<EMAIL_ADDRESS>

###### Abstract

Learning to drive faithfully in highly stochastic urban settings remains an
open problem. To that end, we propose a Multi-task Learning from Demonstration
(MT-LfD) framework which uses supervised auxiliary task prediction to guide
the main task of predicting the driving commands. Our framework involves an
end-to-end trainable network for imitating the expert demonstrator’s driving
commands. The network intermediately predicts visual affordances and action
primitives through direct supervision which provide the aforementioned
auxiliary supervised guidance. We demonstrate that such joint learning and
supervised guidance facilitates hierarchical task decomposition, assisting the
agent to learn faster, achieve better driving performance and increases
transparency of the otherwise black-box end-to-end network. We run our
experiments to validate the MT-LfD framework in CARLA, an open-source urban
driving simulator. We introduce multiple non-player agents in CARLA and induce
temporal noise in them for realistic stochasticity.

##  1 Introduction

A lot of recent work in autonomous driving has been focused on designing end-
to-end learning networks for the task of driving using input images (Bojarski
et al., 2016; Xu et al., 2016; Bojarski et al., 2017; Codevilla et al., 2017;
Chowdhuri et al., 2017a). Most works model this as an end-to-end regression
problem of arriving at the control values using the input pixel values
directly. Clearly, humans utilize a much richer hierarchical task
decomposition pipeline while carrying out visuomotor tasks like driving.
Rather than arriving at the exact throttle, brake and steering percentages
directly from high-dimensional pixel values, our brains first decompose the
image into recognizable objects and their intuitive states in the scene
(Felleman and Van, 1991; Kandel et al., 2000), then use visual abstractions
along with past experience to come up with approximately optimal high level
decisions or action primitives for the given state, and lastly use these
action primitives to arrive at the exact motor commands to be executed
(Bertenthal, 1996; Mataric, 2000; Flash and Hochner, 2005; Hart and Giszter,
2010). Such kind of hierarchical structure allows us to tackle hard problems
like driving by decomposing them into multiple sub-tasks allowing for better
generalization.

Studies in Neuroscience suggest that humans use a combintion of model-based
and model-free techniques for sequential decision making (Gläscher et al.,
2010; Evans, 2008; Fermin et al., 2010; Beierholm et al., 2011). Most of the
current end-to-end driving networks only utilize a model-free architecture,
without explicitly trying to infer the state or model of the environment which
can be used for planning. A combination of model-based and model-free
approaches could greatly improve decision-making in autonmous driving
settings.

Another challenge with such end-to-end learning systems is their black-box
nature (Marcus, 2018; Samek et al., 2017; Ribeiro et al., 2016). Deep Learning
techniques, in general, are highly opaque function approximators having
millions of self-learned parameters, each of which might not have any human-
interpretable significance. This is a huge setback in life-critical decision-
making tasks like autonomous driving, where it is essential to not only arrive
at the correct decision but also know how the decision was made by the system.

In this work, we attempt to overcome these obstacles by proposing supervised
auxiliary tasks that are learned by the network along with the main task of
driving. We propose a set of ‘visual affordances’ and ‘action primitives’,
that are annotated and used as auxiliary supervised tasks. The visual
affordances are used to form abstract description of the visual scene in front
of the vehicle and the action primitives provide an abstract description of
the possible high-level actions required for driving.

The auxiliary tasks serve a three-fold purpose. Firstly, the auxiliary tasks
allow us to infuse a rich prior in the form of human knowledge into the system
that assists the final prediction task instead of expecting the network to
learn all relevant knowledge from scratch. While trying to demonstrate the
driving task, a human identifies the visual affordances and action primitives
that are essential for driving, and provides the network with the auxiliary
task of predicting these abstractions, thus assisting its decision-making with
human knowledge. Secondly, the joint learning of the auxiliary task along with
the main task of driving provides auxiliary guided supervision by forcing the
network to predict intermediate representations like distance to vehicles and
orientation with respect to the lane among others, that can be crucial to
arrive at the final driving decision. This guided supervision allows the
network to learn superior internal features thus allowing it to learn faster
and generalize better. The auxiliary visual affordances can also be seen as an
abstract description of the agent’s environment and the joint-learning
technique as a method of efficiently combining model-based and model-free
techniques. Lastly, predicting the visual affordances and action primitives
allows for better transparency in the network’s learned internal
representations, helping us better understand its decision-making process.

We demonstrate our hypothesis in the CARLA simulator (Dosovitskiy et al.,
2017). All trials are started with random initialization of the player and
non-player positions and goals and the non-player vehicles are induced with
temporal noise to increase stochasticity of the multi-player dynamics. Such a
stochastic setting makes it more difficult for the agent to overfit to a
deterministic non-player policy and requires that the player have temporal
information about the non-player agents to infer their present state without
which it cannot derive it’s own policy. The demonstrated expert policy is also
extremely opportunistic as the demonstrator tries to not only avoid all
collisions but also reach the goal locations in the least amount of time,
generously overtaking vehicles if required. These changes make the driving
conditions more realistic and challenging for a machine learning algorithm.

##  2 Related Work

Learning from Demonstration or Imitation Learning (Schaal, 1997; Argall et
al., 2009; Atkeson and Schaal, 1997) and Reinforcement Learning (Sutton and
Barto, 1998; Kaelbling et al., 1996) have proven to be the key techniques for
learning sequential decision-making tasks through demonstration and experience
respectively. Reinforcement Learning requires a hand-crafted reward signal
which the agent tries to maximize over time and in turn learns the desired
policy through trial-and-error. The agent has to learn to assign credit to
past actions faithfully in case of delayed rewards and also balance
exploration and exploitation effectively to arrive at an ideal policy, both of
which are extremely challenging (Sutton, 1984, 1992). In Learning from
Demonstration, on the other hand, the agent learns by directly trying to mimic
the decisions of an expert demonstrator. Though originally designed for low-
dimensional state-space tasks, with the advent of superior supervised function
approximators (Krizhevsky et al., 2012; He et al., 2015b; Szegedy et al.,
2017), a lot of progress has been made in scaling these algorithms to very
high dimensional state-space tasks enabling agents to learn policies directly
from images (Tai and Liu, 2016; Mnih et al., 2015; Ratliff et al., 2009;
Bojarski et al., 2016).

In this work, we use Learning from Demonstration framework to train an agent
to learn the task of driving using high-dimensional observations in form of
images. Learning from Demonstration along with deep function approximators
have been used to tackle a lot of problems in robotics like indoor mobile
robot navigation (Tai et al., 2016), quad-rotor control in forest trials
(Giusti et al., 2016), robot-arm manipulation (Duan et al., 2017; Finn et al.,
2017; Yu et al., 2018) among others. The closest to our work are the works of
Bojarski et al. (2016) who show autonomous lane following using a single
trained network, Codevilla et al. (2017) who demonstrate autonomous driving in
CARLA using an additional conditional input from a high-level planner, Hou et
al. (2017) who compare various contemporary networks for autonomous driving
tasks and Chowdhuri et al. (2017b) who demonstrate multi-task and multi-modal
behavior for autonomous driving.

Multi-task learning (MTL) research shows the joint training of auxiliary
related side-tasks along with the main task enhances the training performance
(Caruana, 1998; Zhang and Yeung, 2012). MTL in neural networks (Ruder, 2017)
has been successfully demonstrated in many tasks previously including text-to-
speech conversion (Seltzer and Droppo, 2013), natural language processing
(Collobert and Weston, 2008), speech processing (Deng et al., 2013) and
computer vision (Girshick, 2015; Zhang et al., 2016). In the field of
sequential decision making, Lample and Chaplot (2016) demonstrate MTL for 3D
game playing, Mirowski et al. (2016) and Jaderberg et al. (2016) demonstrate
MTL in 3D maze navigation task whereas Chowdhuri et al. (2017b) utilize the
MTL framework for autonomous driving. Instead of employing future control
outputs as auxiliary tasks as shown by Chowdhuri et al. (2017b), in this work
we employ action and visual abstractions to guide the driving behavior.

Supervised learning of visual affordances for autonomous driving was
introduced by Chen et al. (2015), though they use the predicted affordances to
plan using a set of fixed rules whereas our network uses visual affordances as
auxiliary tasks for the main task of driving. Action primitives can be
inferred as sub-policies for the desired task. Learning hierarchical policies
via demonstration is an active area of research (Byrne and Russon, 1998;
Demiris and Dearden, 2005; Le et al., 2018; Shiarlis et al., 2018) and
research in developmental psychology has also found evidence of hierarchical
task decomposition during imitation in young children (Whiten et al., 2006).
Our work decomposes the main task of driving into sub-policies which are used
as auxiliary supervision to derive the final control commands.

##  3 Multi-task Learning from Demonstration (MT-LfD) Framework

###  3.1 Learning from Demonstration

We first begin by detailing the framework used to train the agent through
expert demonstrations for the task of driving autonomously along with
auxiliary task guidance. Learning from Demonstration (LfD) involves training
an agent to try and imitate an expert demonstrator. At each time step $i$, the
expert demonstrator is provided with an observation $o_{i}$, and the
demonstrator provides the ideal action $u_{i}$ for that particular
observation. A dataset $D=\\{o_{i},u_{i}\\}_{i=1}^{N}$ comprising of multiple
episodic sequential roll-outs of the demonstrations is curated.

LfD works on the assumption that if the demonstrator could deduce the ideal
actions $u_{i}$ from the provided observations $o_{i}$, and if the
demonstrator uses a consistent policy to determine the ideal actions $u_{i}$,
there must exist a constant mapping function $F$ which maps the correlation
between actions and observations $u_{i}=F(o_{i})\forall i\in[1,N]$. In such a
scenario, an agent parameterized by $\theta$ can be trained to obtain a policy
$\pi(u_{i}/o_{i};\theta)$ which maps the observations to actions. If a
sufficiently expressive function approximator is used to train the agent, the
parameters $\theta$ can be tuned such that the learned policy $\pi$ is almost
equivalent to the demonstrated mapping function $F$. This can be done by
tuning the parameters using the following update rule:

$\theta=\arg\min_{\theta}\sum_{i=1}^{N}loss(\pi(o_{i}),u_{i})$ (1)

and since we are assuming that $u_{i}=F(o_{i})\forall i\in[1,N]$ the above
equation is reducing the loss:

$loss(\pi(o_{i}),F(o_{i}))$ (2)

and hence trying to make $\pi\approx F$.

If the collected dataset $D$ is diverse enough to cover a large support of the
distribution, the learned policy $\pi(u/o;\theta)$ could generalize to new
unseen observations $o_{j}$ which are similar to the demonstrated
observations, and predict a faithful control output $u_{j}$. Even so,
distribution-mismatch is a common problem in LfD, where the demonstrated
distribution does not match the test-time distribution, leading to unexpected
compounding errors. A lot of methods have been suggested to overcome the
distribution-mismatch problem (Ross et al., 2010; Levine and Koltun, 2013;
Bojarski et al., 2016). In this work, we employ the noise injection method
suggested by Laskey et al. (2017), by inducing noise in the agent during
demonstration (details of which are in Section 4.2) to force it to visit novel
states and improve the demonstration distribution.

###  3.2 Auxiliary Task Supervision

Input observations $o_{i}$ include high-dimensional images and thus the
underlying state $s_{i}$ of the system has to be inferred from these images
for model-based decision making. Here we form an abstract description of the
state $s_{i}$ using the visual affordances $v_{i}$. These local visual
statistics allow the agent to learn an abstract local model of it’s
environment. The control commands $u_{i}$ predicted by the agent can also be
decomposed into sub-policies. We decompose the task by enabling the agent to
predict action primitives $a_{i}$ which are abstract descriptions of the
policy. The visual affordances and action primitives are used as auxiliary
supervised sub-tasks that are predicted by the multi-task network.

The dataset $D$ is augmented with the visual affordances $v_{i}$ and action
primitives $a_{i}$ annotation and thus becomes
$D=\\{o_{i},u_{i},v_{i},a_{i}\\}_{i=1}^{N}$. Let $\pi(u_{i}/o_{i}^{\prime})$,
$\phi(v_{i}/o_{i})$ and $\psi(a_{i}/o_{i})$, jointly parameterized by $\theta$
denote the learnable functions to predict control, visual affordances and
action primitives respectively. Since the policy $\pi$ is now guided by the
predicted visual affordances and action primitives, policy $\pi$ is
conditioned on $\phi$ and $\psi$ along with observation $o_{i}$ as
$\pi(u_{i}/o_{i},\phi(o_{i}),\psi(o_{i}))$ which we denote as
$\pi(u_{i}/o_{i}^{\prime})$. We can augment the update rule defined in Eq. 1
as follows:

$\theta=\arg\min_{\theta}\sum_{i=1}^{N}loss(\pi(o_{i}^{\prime}),u_{i})+\alpha\,loss(\phi(o_{i}),v_{i})+\beta\,loss(\psi(o_{i}),a_{i})+\gamma(\theta)^{2}$
(3)

where $(\theta)^{2}$ is the L2 regularization loss and $\alpha,\beta,\gamma$
are hyperparameter coefficients for the auxiliary losses. As the learning
progresses and the network is able to learn to faithfully predict visual
affordances and action abstractions, $\phi(o_{i})\approx v_{i}$ and
$\psi(o_{i})\approx a_{i}$. Thus eventually, the policy is conditioned on the
visual affordances and action primitives as $\pi(u_{i}/o_{i},v_{i},a_{i})$ and
is guided by them.

###  3.3 Network Architecture

Figure 1: High-level representational block diagram of the proposed network.

In this sub-section we outline the network architecture we use to perform MT-
LfD. In our framework, the observations $o_{i}$ consist of stacked input
images, forward speed of the player agent and a planner input which provides
the high-level directions at intersections. To encode temporal information
which is vital for stochastic multi-agent urban scenarios that we experiment
with, we provide a history sequence of five images stacked along with the the
current image at each time-step. The control predictions $u_{i}$ include the
brake, throttle and steering percentage. Details about the visual affordances
and actions primitives and their ground truth annotation scheme is presented
in Section 4.3.

Figure 1 shows a high-level representation of our proposed network. We use
Resnet-50 (He et al., 2015a) to extract useful features from the stack of
input images. Instead of adding or concatenating the speed and planner inputs
we use a learnable soft-attention mechanism (Bahdanau et al., 2014) to attend
to the extracted image features using the speed and planner inputs as
represented by the input attention block in Figure 1. A learnable soft-
attention mechanism allows for the low dimensional speed and planner inputs to
have a large impact on the relatively higher-dimensional image feature vector.
This attention mechanism is represented by a neural network layer and is
jointly learned with the main network. The speed and planner inputs separately
attend to the extracted feature vector and a union of the two attended
features is taken to achieve a jointly attended vector. This attended
observation feature is used to predict the visual affordances and action
primitives. Lastly, in the Auxiliary Task Attention Block, a similar
aforementioned double-attention mechanism is used to attend over the
observation feature vector using the visual affordances and action primitives,
which is then used to predict the final control predictions. The dashed-line
vectors in Figure 1 represent the ground-truth positions from where the
gradients are jointly back-propagated.

##  4 Experimental Setup

###  4.1 Environment Setup

We use CARLA, an open-source 3D urban driving simulator, for our experiments.
The use of CARLA allows for high-fidelity graphics and physics simulations of
urban driving environment including diverse models of vehicles, pedestrians,
houses, static obstacles, side-walks and intersections. For training, we
collect data from a town with 2.9 km drivable urban roads. The map of the town
is as shown in Figure 2.

Figure 2: Map of the town used for collecting the demonstrations. Pink dots
indicate the initialized positions of the non-player agent vehicles where as
the red dot indicates the initialized position of the player agent.

To enable rich multi-agent behaviour, we initialize the map with 120 non-
player vehicles and 140 non-player pedestrians during demonstration. The
player as well as non-player agents are initialized at random start positions
at the beginning of each episode to increase stochasticity of the system
dynamics. The non-player pedestrians have a rich AI which enables diverse
realistic behaviours. Each pedestrian is provided a goal location at random
and assigned a random maximum walking speed. The pedestrians try to reach the
random goal point, without colliding with other static or dynamic obstacles
and use a weighted navigation mesh to decide when to walk on the footpath or
when and where to cross the road at a stochastically sampled angle to the
road. The non-player vehicles are initialized with random model and random
colours at the beginning of each episode and have a rich intelligent behaviour
as well. Each non-player vehicle drives within lane, stops at traffic lights,
follows the speed limit of the particular road it is travelling on, randomly
samples turns at intersections and actively avoids all other static and
dynamic obstacles.

Even though the non-player vehicles have rich intelligent behaviour, their
deterministic nature makes them highly unrealistic. A player agent could
easily map such deterministic behaviour and overfit it’s own behaviour. In
such cases a player agent could easily derive it’s own policy given a single
current time-step image. To increase the stochasticity of the non-player
vehicles, we induce temporal noise in the non-player vehicles. The temporal
noise is enabled with some random probability and causes the non-payer
vehicles to brake immediately and stop for a random duration sampled from a
uniform distribution. This simple noise injection, prevents the player agent
from overfitting to non-player vehicles policy and gives rise to realistic
high-level urban traffic behaviours like erratic stopping, slow moving
vehicular queues, busy multi-directional traffic at intersections, increased
probability of overtaking maneuvers among others as can be seen in Figure 3.

(a)

(b)

(c)

(d)

(e)

(f)

Figure 3: Sample simulation images as recorded by the front-facing RGB camera
showing the diverse dynamics of the urban environment. 3(a) shows the player
agent following a non-player agent vehicle. 3(b) shows a long queue of
vehicles in the opposite lane. 3(c) shows a yellow vehicle cutting into the
driving lane from a turn in front of the player agent. 3(d) shows a pedestrian
carrying a guitar crossing the road in front of the player agent. 3(e) shows a
busy intersection with agents approaching from multiple directions. 3(f) shows
an approaching left turn.

###  4.2 Data Collection

In our framework, a human driver is an expert demonstrator and provides the
ground truth driving commands using CARLA that the network is trained to
imitate. A front facing RBG mono-camera of $320X180$ pixel resolution and
$100^{\circ}$ FOV is mounted on a simulated vehicle for collecting visual
information while driving. The expert demonstrator looks at a similar first-
person view of a higher resolution ($1280X720$) while driving the vehicle to
collect data. The complications of record-mapping (Argall et al., 2009) are
prevented by having the demonstrator use a similar visual view point for
demonstration.

The demonstrator is equipped with the task of demonstrating the ideal brake,
throttle and steering commands for each observation frame. The demonstrator
does this by using a gaming steering wheel and throttle and brake pedals,
which allow for easier demonstrations and enable analog inputs into the
system. The setup we use for demonstration is shown in Figure 4 At the
beginning of each episode, a collection of random goal points are selected and
sequentially provided to an A* planner which uses the town map to plan the
shortest route to the goal locations. The A* planner provides the demonstrator
with one of the four high level commands (go left, go right, go straight,
follow lane) which the demonstrator follows while providing demonstrations.
The demonstrator avoids all static and dynamic obstacles and also tries to
stay on lane as much as possible. The demonstrator also tries to reach the
goal in the least amount of time, even overtaking long queues of vehicles if
required, while trying to drive within the above mentioned constraints. This
results in an extremely rich demonstration policy making it more realistic and
difficult to imitate in novel scenarios.

Figure 4: Setup used by expert demonstrator to provide demonstrations in
CARLA.

To overcome the distribution mismatch problem, we induce noise in the player
agent and record the demonstrations of the agent recovering from the noise as
provided by the expert demonstrator. Unlike Codevilla et al. (2017) who induce
a fixed duration temporal noise in the steering, we induce both positive and
negative temporal noises in the steering, brake and throttle; the probability
and duration of which are sampled from a uniform distribution.

We collect $150,000$ frames of images at about 6 fps resulting from
approximately 7 hours of driving demonstration spread across 82 episodes. The
player agent data collected include RGB images, high-level planner command and
speed provided by CARLA which form our input observation $o_{i}$, and
steering, brake and throttle demonstrations provided by the expert
demonstrator which forms the action $u_{i}$. Other auxiliary player and non-
player measurements are also collected which help us in annotating the ground-
truth visual affordances $v_{i}$ and action primitives $a_{i}$ as discussed
below.

Visual Affordances
---
Distance to vehicle ahead in same lane
Velocity of vehicle ahead in same lane
Distance to vehicle ahead in opposite lane
Velocity of vehicle ahead in opposite lane
Percentage player in opposite lane
Percentage player on sidewalk
Distance to approaching intersection
Distance to approaching left turn
Distance to approaching right turn
Longitudinal distance to pedestrian approaching
Lateral distance to pedestrian approaching
Longitudinal distance to pedestrian departing
Lateral distance to pedestrian departing

Action Primitives
---
Do nothing
Speed up
Slow down
Halt
Turn left
Turn right
Cut out of the lane
Cut into the lane

Table 1: List of visual affordances and action primitives used for auxiliary
supervision

###  4.3 Auxiliary Tasks and Data Annotation

In this subsection, we describe the visual affordances and action primitives
used for auxiliary supervision. We identify 13 different visual affordances
that are critical to describe the local state $s_{i}$ of the system and 8
different action primitives that decompose the policy effectively. A list of
all the visual affordances and action primitives is provided in Table 1. A few
visual affordance measurements like ’Percentage player in opposite lane’ and
’Percentage player on sidewalk’ are directly provided by CARLA, but most
others have to be derived using a fixed set of rule-based mechanism. Multiple
player and non-player measurements are collected which are not directly used
for training the network, but are used to derive the ground truth annotations
for the visual affordances. These measurements include player global position,
player yaw, non-player agent positions, non-player agent yaw and non-player
agent velocity. Action Primitives are derived using fixed rules to classify
the demonstrated steering, brake and throttle control values.

##  5 Results

We train the proposed network in Section 3.3 using the expert demonstrated
data for the control values and the automatic rule-based ground truth
generated for the visual affordances and action primitives. Out of the 150000
images collected, a sequence of 10000 images was held out for validation and
the rest was used for training. The control prediction and visual affordances
prediction are regression problems and thus we use Mean Squared Error loss for
$\pi(o_{i}^{\prime})$ and $\phi(o_{i})$ whereas the action primitives
prediction is a mutually non-exclusive multi-class classification problem and
thus we use a multi-class cross entropy loss for $\psi(o_{i})$. Data
augmentation including Gaussian blurring, Gaussian noise, pixel dropouts,
gray-scaling, contrast change and intensity shits were used to help with
better generalization. We use Adam optimizer with an initial learning rate of
$3e^{-4}$ and use learning rate decay. We set $\alpha$, $\beta$ and $\gamma$
to $0.1$, $0.2$ and $1e^{-5}$ respectively. The network is trained for
approximately 15 hours on a GPU. We call our proposed network as MT-LfD
through the rest of this section.

(a)

(b)

Figure 5: 5(a) Average MSE loss for the control prediction task on the
validation set as the training progresses (Lower is better). 5(b) Accuracy for
the multi-class classification task of predicting the action primitives on the
validation set as the training progresses (Higher is better).

Figure 5 shows the plots of our results. We compare the results of MT-LfD with
2 baselines to verify the performance of MT-LfD. The first baseline which we
call Baseline with no attention is an ablated version of the MT-LfD network
architecture in which we use simple concatenation instead of learnable soft-
attention to combine the auxiliary information along with the visual
information. The second baseline which we call Baseline with no auxiliary
guidance has the same architecture as MT-LfD with $\alpha$ and $\beta$ set to
zero. This essentially prevents the loss from the visual affordances and
action primitives from back propagating through the network and thus removes
the auxiliary guidance provided by their predictions. Figure 5(b) shows the
action primitives prediction accuracy on the held-out validation set. Since
Baseline with no auxiliary guidance does not have a loss for action
primitives, it can be observed that it performs almost on par with random
sampling whereas the others perform much better at the classification task.

We keep the rest of the hyperparameters same as MT-LfD for the baselines.
Figure 5(a) shows the mean-squared error loss for the final control prediction
task on the held-out validation set. MT-LfD has lower loss as compared to
Baseline with no attention thus validating our use of the soft-attention
mechanism. MT-LfD also has a lower loss and converges much faster than
Baseline with no auxiliary guidance thus proving the necessity of the guided
auxiliary supervision provided by the visual affordances and action
primitives, validating our hypothesis.

##  6 Summary

We demonstrate Multi-task Learning from Demonstration for end-to-end learning
of autonomous driving, jointly supervised and guided by visual affordances and
action primitives. We present our network architecture which uses ResNet-50
and learnable soft-attention mechanisms to combine the auxiliary task
predictions with the observation information for the final driving task
prediction. We show that our proposed MT-LfD framework outperforms vanilla LfD
and MT-LfD without attention in the main task of vehicle control prediction.
We thus, validate our hypothesis that the joint learning of the auxiliary
tasks and the employing their predictions to guide the final control
prediction is able to enhance the speed and the performance of learning.

Nevertheless, our network does not outperform previous work in more
deterministic setups with controlled number of agents and maximum speed.
Driving in a realistic scenario with a highly stochastic multi-agent setup and
realisitic driving demonstrations still remains a challenging open problem. It
remains to ben seen how memory-augmented networks perform in such a scenario.
A prospective future work could be to use Inverse-Reinforcement Learning to
derive the intention of the demonstrator and use model based-statistics along
with such inferred intention to derive an ideal driving policy. Another
orthogonal prospective work could be to train a network to plan on the model-
based statistics and use the planning network along with the model-free
prediction network to derive the driving policies. More work also needs to be
done in making driving simulators more realistic to human-driving scenarios
for autonomous driving experiments.

## References

  * Argall et al. [2009] Brenna D Argall, Sonia Chernova, Manuela Veloso, and Brett Browning.  A survey of robot learning from demonstration.  _Robotics and autonomous systems_ , 57(5):469–483, 2009.
  * Atkeson and Schaal [1997] Christopher G Atkeson and Stefan Schaal.  Robot learning from demonstration.  In _ICML_ , volume 97, pages 12–20, 1997.
  * Bahdanau et al. [2014] Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio.  Neural machine translation by jointly learning to align and translate.  _CoRR_ , abs/1409.0473, 2014.  URL http://arxiv.org/abs/1409.0473.
  * Beierholm et al. [2011] Ulrik R Beierholm, Cedric Anen, Steven Quartz, and Peter Bossaerts.  Separate encoding of model-based and model-free valuations in the human brain.  _Neuroimage_ , 58(3):955–962, 2011.
  * Bertenthal [1996] Bennett I. Bertenthal.  Origins and early development of perception, action, and representation.  _Annual review of psychology_ , 47:431–59, 1996.
  * Bojarski et al. [2016] Mariusz Bojarski, Davide Del Testa, Daniel Dworakowski, Bernhard Firner, Beat Flepp, Prasoon Goyal, Lawrence D Jackel, Mathew Monfort, Urs Muller, Jiakai Zhang, et al.  End to end learning for self-driving cars.  _arXiv preprint arXiv:1604.07316_ , 2016.
  * Bojarski et al. [2017] Mariusz Bojarski, Philip Yeres, Anna Choromanska, Krzysztof Choromanski, Bernhard Firner, Lawrence Jackel, and Urs Muller.  Explaining how a deep neural network trained with end-to-end learning steers a car.  _arXiv preprint arXiv:1704.07911_ , 2017.
  * Byrne and Russon [1998] Richard W Byrne and Anne E Russon.  Learning by imitation: A hierarchical approach.  _Behavioral and brain sciences_ , 21(5):667–684, 1998.
  * Caruana [1998] Rich Caruana.  Multitask learning.  In _Learning to learn_ , pages 95–133. Springer, 1998.
  * Chen et al. [2015] Chenyi Chen, Ari Seff, Alain L. Kornhauser, and Jianxiong Xiao.  Deepdriving: Learning affordance for direct perception in autonomous driving.  _CoRR_ , abs/1505.00256, 2015.  URL http://arxiv.org/abs/1505.00256.
  * Chowdhuri et al. [2017a] Sauhaarda Chowdhuri, Tushar Pankaj, and Karl Zipser.  Multi-modal multi-task deep learning for autonomous driving.  _CoRR_ , abs/1709.05581, 2017a.  URL http://arxiv.org/abs/1709.05581.
  * Chowdhuri et al. [2017b] Sauhaarda Chowdhuri, Tushar Pankaj, and Karl Zipser.  Multi-modal multi-task deep learning for autonomous driving.  _arXiv preprint arXiv:1709.05581_ , 2017b.
  * Codevilla et al. [2017] Felipe Codevilla, Matthias Müller, Alexey Dosovitskiy, Antonio López, and Vladlen Koltun.  End-to-end driving via conditional imitation learning.  _arXiv preprint arXiv:1710.02410_ , 2017.
  * Collobert and Weston [2008] Ronan Collobert and Jason Weston.  A unified architecture for natural language processing: Deep neural networks with multitask learning.  In _Proceedings of the 25th international conference on Machine learning_ , pages 160–167. ACM, 2008.
  * Demiris and Dearden [2005] Yiannis Demiris and Anthony Dearden.  From motor babbling to hierarchical learning by imitation: a robot developmental pathway.  2005\.
  * Deng et al. [2013] Li Deng, Geoffrey Hinton, and Brian Kingsbury.  New types of deep neural network learning for speech recognition and related applications: An overview.  In _Acoustics, Speech and Signal Processing (ICASSP), 2013 IEEE International Conference on_ , pages 8599–8603. IEEE, 2013.
  * Dosovitskiy et al. [2017] Alexey Dosovitskiy, German Ros, Felipe Codevilla, Antonio López, and Vladlen Koltun.  Carla: An open urban driving simulator.  _arXiv preprint arXiv:1711.03938_ , 2017.
  * Duan et al. [2017] Yan Duan, Marcin Andrychowicz, Bradly C. Stadie, Jonathan Ho, Jonas Schneider, Ilya Sutskever, Pieter Abbeel, and Wojciech Zaremba.  One-shot imitation learning.  _CoRR_ , abs/1703.07326, 2017.  URL http://arxiv.org/abs/1703.07326.
  * Evans [2008] Jonathan St BT Evans.  Dual-processing accounts of reasoning, judgment, and social cognition.  _Annu. Rev. Psychol._ , 59:255–278, 2008.
  * Felleman and Van [1991] Daniel J Felleman and DC Essen Van.  Distributed hierarchical processing in the primate cerebral cortex.  _Cerebral cortex (New York, NY: 1991)_ , 1(1):1–47, 1991.
  * Fermin et al. [2010] Alan Fermin, Takehiko Yoshida, Makoto Ito, Junichiro Yoshimoto, and Kenji Doya.  Evidence for model-based action planning in a sequential finger movement task.  _Journal of motor behavior_ , 42(6):371–379, 2010\.
  * Finn et al. [2017] Chelsea Finn, Tianhe Yu, Tianhao Zhang, Pieter Abbeel, and Sergey Levine.  One-shot visual imitation learning via meta-learning.  _CoRR_ , abs/1709.04905, 2017.  URL http://arxiv.org/abs/1709.04905.
  * Flash and Hochner [2005] Tamar Flash and Binyamin Hochner.  Motor primitives in vertebrates and invertebrates.  _Current opinion in neurobiology_ , 15(6):660–666, 2005.
  * Girshick [2015] Ross B. Girshick.  Fast R-CNN.  _CoRR_ , abs/1504.08083, 2015.  URL http://arxiv.org/abs/1504.08083.
  * Giusti et al. [2016] Alessandro Giusti, Jérôme Guzzi, Dan C Cireşan, Fang-Lin He, Juan P Rodríguez, Flavio Fontana, Matthias Faessler, Christian Forster, Jürgen Schmidhuber, Gianni Di Caro, et al.  A machine learning approach to visual perception of forest trails for mobile robots.  _IEEE Robotics and Automation Letters_ , 1(2):661–667, 2016.
  * Gläscher et al. [2010] Jan Gläscher, Nathaniel Daw, Peter Dayan, and John P O’Doherty.  States versus rewards: dissociable neural prediction error signals underlying model-based and model-free reinforcement learning.  _Neuron_ , 66(4):585–595, 2010.
  * Hart and Giszter [2010] Corey B Hart and Simon F Giszter.  A neural basis for motor primitives in the spinal cord.  _Journal of Neuroscience_ , 30(4):1322–1336, 2010\.
  * He et al. [2015a] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  _CoRR_ , abs/1512.03385, 2015a.  URL http://arxiv.org/abs/1512.03385.
  * He et al. [2015b] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Delving deep into rectifiers: Surpassing human-level performance on imagenet classification.  In _Proceedings of the IEEE international conference on computer vision_ , pages 1026–1034, 2015b.
  * Hou et al. [2017] Yiqi Hou, Sascha Hornauer, and Karl Zipser.  Fast recurrent fully convolutional networks for direct perception in autonomous driving.  _CoRR_ , abs/1711.06459, 2017.  URL http://arxiv.org/abs/1711.06459.
  * Jaderberg et al. [2016] Max Jaderberg, Volodymyr Mnih, Wojciech Marian Czarnecki, Tom Schaul, Joel Z. Leibo, David Silver, and Koray Kavukcuoglu.  Reinforcement learning with unsupervised auxiliary tasks.  _CoRR_ , abs/1611.05397, 2016.  URL http://arxiv.org/abs/1611.05397.
  * Kaelbling et al. [1996] Leslie Pack Kaelbling, Michael L Littman, and Andrew W Moore.  Reinforcement learning: A survey.  _Journal of artificial intelligence research_ , 4:237–285, 1996.
  * Kandel et al. [2000] Eric R Kandel, James H Schwartz, Thomas M Jessell, Steven A Siegelbaum, A James Hudspeth, et al.  _Principles of neural science_ , volume 4.  McGraw-hill New York, 2000.
  * Krizhevsky et al. [2012] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton.  Imagenet classification with deep convolutional neural networks.  In _Advances in neural information processing systems_ , pages 1097–1105, 2012.
  * Lample and Chaplot [2016] Guillaume Lample and Devendra Singh Chaplot.  Playing FPS games with deep reinforcement learning.  _CoRR_ , abs/1609.05521, 2016.  URL http://arxiv.org/abs/1609.05521.
  * Laskey et al. [2017] Michael Laskey, Jonathan Lee, Wesley Yu-Shu Hsieh, Richard Liaw, Jeffrey Mahler, Roy Fox, and Ken Goldberg.  Iterative noise injection for scalable imitation learning.  _CoRR_ , abs/1703.09327, 2017.  URL http://arxiv.org/abs/1703.09327.
  * Le et al. [2018] Hoang M. Le, Nan Jiang, Alekh Agarwal, Miroslav Dudík, Yisong Yue, and Hal Daumé III.  Hierarchical imitation and reinforcement learning.  _CoRR_ , abs/1803.00590, 2018.  URL http://arxiv.org/abs/1803.00590.
  * Levine and Koltun [2013] Sergey Levine and Vladlen Koltun.  Guided policy search.  In _International Conference on Machine Learning_ , pages 1–9, 2013\.
  * Marcus [2018] Gary Marcus.  Deep learning: A critical appraisal.  _CoRR_ , abs/1801.00631, 2018.  URL http://arxiv.org/abs/1801.00631.
  * Mataric [2000] Maja J. Mataric.  Sensory-motor primitives as a basis for imitation: Linking perception to action and biology to robotics.  In _Imitation in Animals and Artifacts_ , pages 391–422. MIT Press, 2000.
  * Mirowski et al. [2016] Piotr Mirowski, Razvan Pascanu, Fabio Viola, Hubert Soyer, Andrew J. Ballard, Andrea Banino, Misha Denil, Ross Goroshin, Laurent Sifre, Koray Kavukcuoglu, Dharshan Kumaran, and Raia Hadsell.  Learning to navigate in complex environments.  _CoRR_ , abs/1611.03673, 2016.  URL http://arxiv.org/abs/1611.03673.
  * Mnih et al. [2015] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al.  Human-level control through deep reinforcement learning.  _Nature_ , 518(7540):529, 2015.
  * Ratliff et al. [2009] Nathan D Ratliff, David Silver, and J Andrew Bagnell.  Learning to search: Functional gradient techniques for imitation learning.  _Autonomous Robots_ , 27(1):25–53, 2009.
  * Ribeiro et al. [2016] Marco Túlio Ribeiro, Sameer Singh, and Carlos Guestrin.  "why should I trust you?": Explaining the predictions of any classifier.  _CoRR_ , abs/1602.04938, 2016.  URL http://arxiv.org/abs/1602.04938.
  * Ross et al. [2010] Stéphane Ross, Geoffrey J. Gordon, and J. Andrew Bagnell.  No-regret reductions for imitation learning and structured prediction.  _CoRR_ , abs/1011.0686, 2010.  URL http://arxiv.org/abs/1011.0686.
  * Ruder [2017] Sebastian Ruder.  An overview of multi-task learning in deep neural networks.  _CoRR_ , abs/1706.05098, 2017.  URL http://arxiv.org/abs/1706.05098.
  * Samek et al. [2017] Wojciech Samek, Thomas Wiegand, and Klaus-Robert Müller.  Explainable artificial intelligence: Understanding, visualizing and interpreting deep learning models.  _CoRR_ , abs/1708.08296, 2017.  URL http://arxiv.org/abs/1708.08296.
  * Schaal [1997] Stefan Schaal.  Learning from demonstration.  In _Advances in neural information processing systems_ , pages 1040–1046, 1997.
  * Seltzer and Droppo [2013] Michael L Seltzer and Jasha Droppo.  Multi-task learning in deep neural networks for improved phoneme recognition.  In _Acoustics, Speech and Signal Processing (ICASSP), 2013 IEEE International Conference on_ , pages 6965–6969. IEEE, 2013.
  * Shiarlis et al. [2018] Kyriacos Shiarlis, Markus Wulfmeier, Sasha Salter, Shimon Whiteson, and Ingmar Posner.  TACO: learning task decomposition via temporal alignment for control.  _CoRR_ , abs/1803.01840, 2018.  URL http://arxiv.org/abs/1803.01840.
  * Sutton [1992] Richard S Sutton.  Introduction: The challenge of reinforcement learning.  In _Reinforcement Learning_ , pages 1–3. Springer, 1992.
  * Sutton and Barto [1998] Richard S Sutton and Andrew G Barto.  _Reinforcement learning: An introduction_ , volume 1.  MIT press Cambridge, 1998.
  * Sutton [1984] Richard Stuart Sutton.  Temporal credit assignment in reinforcement learning.  1984\.
  * Szegedy et al. [2017] Christian Szegedy, Sergey Ioffe, Vincent Vanhoucke, and Alexander A Alemi.  Inception-v4, inception-resnet and the impact of residual connections on learning.  In _AAAI_ , volume 4, page 12, 2017.
  * Tai and Liu [2016] Lei Tai and Ming Liu.  Deep-learning in mobile robotics - from perception to control systems: A survey on why and why not.  _CoRR_ , abs/1612.07139, 2016.  URL http://arxiv.org/abs/1612.07139.
  * Tai et al. [2016] Lei Tai, Shaohua Li, and Ming Liu.  A deep-network solution towards model-less obstacle avoidance.  In _Intelligent Robots and Systems (IROS), 2016 IEEE/RSJ International Conference on_ , pages 2759–2764. IEEE, 2016.
  * Whiten et al. [2006] Andrew Whiten, Emma Flynn, Katy Brown, and Tanya Lee.  Imitation of hierarchical action structure by young children.  _Developmental science_ , 9(6):574–582, 2006\.
  * Xu et al. [2016] Huazhe Xu, Yang Gao, Fisher Yu, and Trevor Darrell.  End-to-end learning of driving models from large-scale video datasets.  _arXiv preprint_ , 2016.
  * Yu et al. [2018] Tianhe Yu, Chelsea Finn, Annie Xie, Sudeep Dasari, Tianhao Zhang, Pieter Abbeel, and Sergey Levine.  One-shot imitation from observing humans via domain-adaptive meta-learning.  _CoRR_ , abs/1802.01557, 2018.  URL http://arxiv.org/abs/1802.01557.
  * Zhang and Yeung [2012] Yu Zhang and Dit-Yan Yeung.  A convex formulation for learning task relationships in multi-task learning.  _CoRR_ , abs/1203.3536, 2012.  URL http://arxiv.org/abs/1203.3536.
  * Zhang et al. [2016] Yuting Zhang, Kibok Lee, and Honglak Lee.  Augmenting supervised neural networks with unsupervised objectives for large-scale image classification.  In _International Conference on Machine Learning_ , pages 612–621, 2016.