# Engineered spin phase diagram of two interacting electrons
in semiconductor nanowire quantum dots

Yan-Ting Chen  Department of Electrophysics, National Chiao Tung University,
Hsinchu 30010, Taiwan, Republic of China  Shun-Jen Cheng
<EMAIL_ADDRESS>Department of Electrophysics, National Chiao Tung
University, Hsinchu 30010, Taiwan, Republic of China  Chi-Shung Tang
Department of Mechanical Engineering, National United University, Miaoli
36003, Taiwan, Republic of China

###### Abstract

Spin properties of two interacting electrons in a quantum dot (QD) embedded in
a nanowire with controlled aspect ratio and longitudinal magnetic fields are
investigated by using a configuration interaction (CI) method and exact
diagonalization (ED) techniques. The developed CI theory based on a three-
dimensional (3D) parabolic model provides explicit formulations of the Coulomb
matrix elements and allows for straightforward and efficient numerical
implementation. Our studies reveal fruitful features of spin singlet-triplet
transitions of two electrons confined in a nanowire quantum dot (NWQD), as a
consequence of the competing effects of geometry-controlled kinetic energy
quantization, the various Coulomb interactions, and spin Zeeman energies. The
developed theory is further employed to study the spin phase diagram of two
quantum-confined electrons in the regime of “cross over” dimensionality, from
quasi-two-dimensional (disk-like) QDs to finite one-dimensional (rod-like)
QDs.

##  I Introduction

Stimulated by recent success in coherent control of two-electron spin in
laterally coupled quantum dots (QDs), Petta05  the spin states of two
interacting electrons in semiconductor QDs have received increasingly
considerable attention. Accessible and engineerable spin states of few
electrons in QDs thus have become one of the basic features required by the
quantum information applications in which electron spins are utilized as
quantum bit. Loss98 ; Loss09  For two-dimensional (2D) epitaxial QDs, magnetic
field induced spin singlet-triplet (ST) transitions of two-electron ground
states have been studied extensively for years. Kouwenhoven97 ; Kouwenhoven01
; Reimann02 ; Ellenberger06  The underlying physics of the ST transitions is
usually associated with the energetic competition between quantized kinetic
energies, the coulomb interactions, and spin Zeeman energies. Reversely
switching the singlet and triplet spin states of a lateral two-electron QD is
feasible by utilizing electrical control. Kyriakidis02  Moreover, it has been
both theoretically and experimentally shown that more complex oscillating spin
phases can be generated either by reducing the lateral confinement or by
increasing an applied magnetic field. Wagner92 ; Hawrylak93-prl ; Peeters99 ;
Tarucha07

Recently, the local-gate electrical depletion Fasth05 ; Ensslin06 ; Ensslin07
and the bottom-up grown techniques Bjork04 ; Bjork05  have been developed for
the fabrication of few-electron QDs embedded in a nanowire. These experimental
developments open up an opportunity of exploring the cross over mechanisms
from the 2D (disk-like) to the finite 1D (rod-like) QD regimes. Such nanowire
quantum dots (NWQDs) are advantageous for geometrical control over a wide rage
of aspect ratio $a$ (typically from $a\sim 10^{-1}$ to $a\gg 1$). Bjork04 ;
Bjork05  The excellent versatility of shape and dimensionality makes NWQDs a
suitable nanomaterial for scalable quantum electronics. Very recently,
successful fabrication of single electron transistors made of InAs based gate-
defined NWQDs and observations of the singlet-triplet transitions of two
electrons in the QDs have been demonstrated. Fasth07  How the highly tunable
longitudinal confinement of NWQD affects and can be utilized to tailor the
spin properties of few electrons in NWQDs are interesting subjects worth
studying.

The above experimental efforts motivate us to perform a theoretical
investigation of the spin states of two electrons in InAs-based NWQDs Fasth07
by using a developed configuration interaction (CI) theory and exact
diagonalization techniques. Hawrylak03  The developed CI theory is based on
the 3D parabolic model with arbitrary transverse and longitudinal confinement
strengths Nazmitdinov97 ; Lin01  and provides explicit generalized
formulations of the Coulomb matrix, and thus allows for straightforward and
efficient numerical or even semi-analytical implementation widely applicable
for various cylindrically symmetric QDs. Our exact diagonalization studies of
two-electron charged NWQDs with controlled geometric aspect ratios and
longitudinal magnetic fields reveal fruitful features of spin singlet-triplet
transitions, as a consequence of the competing effects of geometry-engineered
kinetic energy quantization, the various Coulomb interactions, and spin Zeeman
energies. The developed theory is further employed to study the spin phase
diagram of two quantum-confined electrons in the regime of “cross over”
dimensionality from quasi-2D (disk-like) QDs to finite 1D (rod-like) QDs.

This article is organized as follows: Section II describes the theoretical
model and the developed configuration interaction theory for few-electron
problems of three-dimensionally confining quantum dots. In Sec. III, we
present and discuss the calculated results of magneto-energy spectrum, the ST
transitions and geometry-engineered spin phase diagrams of two-electron
charged quantum dots embedded in nanowires. Concluding remarks are presented
in Sec. IV.

##  II Model

###  II.1 Single-particle model

We begin with the problem of a single electron in a NWQD with a uniform
longitudinal magnetic field ${\bf B}=(0,0,B)$, which is described by the
single-electron Hamiltonian,

$H_{0}=\frac{1}{2m^{\ast}}({\bf p}+e{\bf A})^{2}+V_{c}(x,y,z)+H_{\rm Z}.$ (1)

Here the first term indicates the term of kinetic energy with ${{\bf
A}}=(B/2)(-y,x,0)$ being the vector potential in symmetric gauge, $m^{\ast}$
the effective mass of electron and $e$ the charge of an elctron. The second
term is the confining potential of NWQD modeled by

$V_{c}(x,y,z)=\frac{1}{2}m^{\ast}\left[\omega_{0}^{2}\left(x^{2}+y^{2}\right)+\omega_{z}^{2}z^{2}\right]$
(2)

with $\omega_{0}$ and $\omega_{z}$ parametrizing, respectively, the transverse
and the longitudinal confining strength. The last term is the spin Zeeman
energy $H_{\rm Z}=g^{\ast}\mu_{B}Bs_{z}$, in terms of the $z$-component of
electron spin $s_{z}=\pm 1/2$, the effective Lande $g$-factor of electron
$g^{\ast}$ and the Bohr magneton $\mu_{B}$.

Figure 1: (Color online) Single-electron energy spectrum as a function of
aspect ratio $a$ of a NWQD with fixed lateral confinement
$\hbar\omega_{0}=13.3$ meV at zero magnetic field obtained from the 3D
parabolic model. The considered lateral confinement strength
$\hbar\omega_{0}=13.3$ meV corresponds to the cross section diameter ${\bf
L}_{0}\sim 65$ nm for a cylindrical InAs nanowire. The low-lying $s$-,
$p^{\pm}$-, and $p^{0}$-orbitals are relevant to a two-electron problem. The
energy quantization for a short (long) NWQD with $a<1$ ( $a>1$) is
characterized by the energy difference between the lowest and first excited
orbitals $\hbar\omega_{0}$ ($\hbar\omega_{z}$).

The single-particle Hamiltonian (1) leads to the extended Fock-Darwin single-
particle spectrum

$\displaystyle\epsilon_{n,m,q,s_{z}}$ $\displaystyle=$
$\displaystyle\hbar\omega_{+}\left(n+\frac{1}{2}\right)+\hbar\omega_{-}\left(m+\frac{1}{2}\right)$
(3) $\displaystyle+\hbar\omega_{z}\left(q+\frac{1}{2}\right)+E_{\rm Z}$

where $n,m,q=0,1,2\cdots$ denote oscillator quantum numbers, $E_{\rm
Z}=g^{\ast}\mu_{B}Bs_{z}$ is the spin Zeeman energy,
$\omega_{\pm}=\omega_{h}\pm\omega_{c}/2$ are in terms of the hybridized
frequency $\omega_{h}\equiv(\omega_{0}^{2}+\omega_{c}^{2}/4)^{1/2}$ and the
cyclotron frequency $\omega_{c}={eB}/{m^{\ast}}$. The corresponding eigenstate
$|n,m,q\rangle$ possesses the orbital angular momentum projection
${\ell}_{z}=\hbar(n-m)$ and the parity $P=1$ ($P=-1$) with respect to $z$-axis
for an even (odd) $q$ number. The wave function of the lowest orbital is given
by

$\displaystyle\psi_{000}({\bf r})$ $\displaystyle=$
$\displaystyle\left[(2\pi)^{3/4}{l_{h}}\sqrt{l_{z}}\right]^{-1}$ (4)
$\displaystyle\times{\rm{exp}}\left[-\frac{1}{4}\left(\frac{x^{2}+y^{2}}{l_{h}^{2}}+\frac{z^{2}}{l_{z}^{2}}\right)\right]\,,$

with the characteristic lengths of the wave function extents
$l_{h}=\sqrt{{\hbar}/{2m^{\ast}\omega_{h}}}$ and
$l_{z}=\sqrt{{\hbar}/{2m^{\ast}\omega_{z}}}$. The wave functions of other
excited states can be generated by successively applying the following defined
raising operators Hawrylak03

$\displaystyle a^{\dagger}$ $\displaystyle=$
$\displaystyle\frac{1}{\sqrt{2}}\left[\frac{x+iy}{2l_{h}}-l_{h}(\partial_{x}+i\partial_{y})\right],$
$\displaystyle b^{\dagger}$ $\displaystyle=$
$\displaystyle\frac{1}{\sqrt{2}}\left[\frac{x-iy}{2l_{h}}-l_{h}(\partial_{x}+i\partial_{y})\right],$
(5) $\displaystyle a_{z}^{\dagger}$ $\displaystyle=$
$\displaystyle\frac{z}{2l_{z}}-l_{z}\partial_{z}$

onto the ground state $|0,0,0\rangle$, i.e.

$|n,m,q\rangle=\frac{(\hat{a}^{\dagger})^{n}(\hat{b}^{\dagger})^{m}(\hat{a_{z}}^{\dagger})^{q}}{\sqrt{n!m!q!}}|0,0,0\rangle.$
(6)

The diameter of cross section of bottom-up synthesized nanowire is typically
$\sim 50-70$ nm. By contrast, the length of a QD in a nanowire, defined by
imposed electrodes or heterostructure potential barriers, is highly tunable
over a wide range from $10$ to $300$ nm. Bjork05  For characterizing the
geometry of a NWQD, it is convenient to define the parameter of aspect ratio,

$a\equiv\frac{l_{z}}{l_{0}}=\sqrt{\frac{\omega_{0}}{\omega_{z}}}$ (7)

according to the characteristic length of the lowest orbital wave function
based on the 3D parabolic model. A rod-like (disk-like) NWQD is characterized
by the value of aspect ratio $a>1$ ($a<1$), where the longitudinal extent of
the electron wave function is longer (shorter) than the transverse one on the
cross section of the nanowire. Notably, the effective aspect ratio
$a=l_{z}/l_{0}$ defined here is not but very close to the geometric aspect
ratio $a_{\rm geom}$, namely $a\simeq a_{\rm geom}={\bf L}_{z}/{\bf L}_{0}$
with ${\bf L}_{0}$ (${\bf L}_{z}$) being the cross section diameter (length)
of NWQD.

Figure 2: (Color online) Schematic illustration of the electronic structures,
consisting of few relevant low lying orbitals (one $s$\- and three
$p$-orbitals), of long rod-like NWQDs [(a)(c)(e)] and short disk-like NWQDs
[(b)(d)(f)] with or without longitudinal magnetic field $B$ and including or
excluding the spin Zeeman splitting $E_{\rm{Z}}$ ($g^{\ast}=0$ or
$g^{\ast}\neq 0$). (a) $a>1$ and $B=0$; (b) $a<1$ and $B=0$; (c) $a>1$, $B\neq
0$ and $g^{\ast}=0$; (d)$a<1$, $B\neq 0$ and $g^{\ast}=0$; (e)$a>1$, $B\neq 0$
and $g^{\ast}\neq 0$; (f)$a<1$, $B\neq 0$ and $g^{\ast}\neq 0$.

Figure 1 presents the calculated single-electron energy spectrum as a function
of aspect ratio $a$ for a NWQD with fixed lateral confinement
$\hbar\omega_{0}=13.3$ meV at zero magnetic field according to Eq.(3). The
chosen parameter of lateral confinement $\hbar\omega_{0}=13.3$ meV is
determined by fitting the numerically calculated energy separation between the
two lowest single-electron orbitals of a cylindrical InAs/InP NWQD of cross
section diameter ${\bf L}_{0}=65$ nm by 3D finite difference simulation. In
the simulation, the $\rm{Schr\ddot{o}dinger}$ equation for a single electron
confined in a 3D cylindrical potential well is solved by using finite
difference method, with the used parameters: the effective mass
$m^{\ast}=0.023m_{0}$ of electron for InAs and the InAs/InP band edge offset
$V_{b}=0.6$ eV as the barrier height of the confining potential. Bjork05 ;
Bjork02

In a two-electron (2e) problem, the most relevant orbitals are the two lowest
ones because the kinetic energy difference between the two orbitals is the
main energy cost, in competition with the coulomb or spin Zeeman energies, for
a spin triplet state to be the ground state of two-electron. By convention, we
from now on name the lowest single electron state
$|n,m,q\rangle=|0,0,0\rangle$ as $s$-orbital, and the next three $p$-shell
states $|0,0,1\rangle$, $|1,0,0\rangle$, and $|0,1,0\rangle$ as $p^{0}$-,
$p^{+}$-, and $p^{-}$-orbitals, respectively. According to Eq. (3), the energy
of the lowest $s$-orbital is explicitly given by

$\epsilon_{s,s_{z}}=\frac{1}{2}\left(\hbar\omega_{+}+\hbar\omega_{-}+\hbar\omega_{z}\right)+g^{\ast}\mu_{B}Bs_{z},$
(8)

and those of the three p-shell orbitals are respectively given by

$\displaystyle\epsilon_{p^{0},s_{z}}=\epsilon_{s,s_{z}}+\hbar\omega_{z},$
$\displaystyle\epsilon_{p^{+},s_{z}}=\epsilon_{s,s_{z}}+\hbar\omega_{+},$
$\displaystyle\epsilon_{p^{-},s_{z}}=\epsilon_{s,s_{z}}+\hbar\omega_{-}.$ (9)

For $B=0$, we have
$\epsilon_{s,s_{z}}=\hbar\omega_{0}\left(1+1/2a^{2}\right)$,
$\epsilon_{p^{0},s_{z}}=\epsilon_{s,s_{z}}+\hbar\omega_{0}/a^{2}$, and
$\epsilon_{p^{+},s_{z}}=\epsilon_{p^{-},s_{z}}=\epsilon_{s,s_{z}}+\hbar\omega_{0}$
according to Eqs.(7) and (9). Here, the $p^{+}$\- and $p^{-}$-orbitals are
degenerate with the same energy separation from the $s$-orbital,
$\hbar\omega_{\pm}=\hbar\omega_{0}$, while the $p^{0}$-orbital is
energetically higher than $s$-orbital by
$\hbar\omega_{z}=\hbar\omega_{0}/a^{2}$. Obviously, $p^{0}$ ($p^{\pm}$) is the
second lowest orbital for a long (short) NWQD with $a>1$ ($a<1$) at zero
magnetic field, as shown in Fig. 1. For a symmetric NWQD with $a=1$, the
$p^{0}$\- and $p^{\pm}$-orbitals form a 3-fold orbital-degenerate shell.
Figure 2 (a) [(b)] schematically depicts the low-lying orbitals of a long
[short] NWQD with $a>1$ [$a<1$] at zero magnetic field.

Applying a longitudinal magnetic field onto a cylindrical NWQD breaks the
degeneracy of $p^{+}$\- and $p^{-}$-orbitals. The orbital Zeeman effect lowers
(raises) the energy level of the $p^{-}$($p^{+}$)-orbital from
$\hbar\omega_{0}$ to $\hbar\omega_{-}$ ($\hbar\omega_{+}$). Thus, if a long
NWQD is subjected to a sufficiently strong magnetic field, the second lowest
orbital of the dot could be changed from the $p^{0}$ to $p^{-}$. By contrast,
the second lowest orbital of a short NWQD is always the $p^{-}$-orbital.
Therefore, the characteristic energy quantization of the $p^{-}$-orbitals,
$\hbar\omega_{-}=\hbar\left(\omega_{0}^{2}+\omega_{c}^{2}/4\right)^{1/2}-\hbar\omega_{c}/2$,
is often a key parameter for a short NWQD or a moderately long NWQD with
strong magnetic field. Considering wide-band gap materials such as GaAs, the
$g$-factors are usually small and the spin Zeeman effect on the energy shift
of orbital is negligible. Figure 2(c) [(d)] depicts the $B$-dependent
electronic orbitals of a long [short] NWQDs, where vanishing spin Zeeman
splitting is assumed ($g^{\ast}=0$ is set).

For a low energy gap material with larger $g^{\ast}$, like InAs, the spin
Zeeman effect could be significant in the spin ST transition of two-electron
QD. Figure 2(e) [(f)] schematically shows the spin-resolved electronic
orbitals of a long [short] NWQDs with $B\neq 0$ and $g^{\ast}\neq 0$ by the
spin Zeeman splitting $2E_{\rm Z}$. With the spin Zeeman effect, all the spin-
up (spin-down) orbitals are energetically lowered (raised) by $E_{\rm
Z}=g^{\ast}\mu_{B}B/2$ according to Eq. (3). If the applied magnetic field or
the $g$-factor of material is so large that the spin Zeeman splittings exceed
the kinetic energy quantization of QD, both of the two lowest single-electron
states are the spin-up ones and the ground state of the 2e dot is ensured to
be a spin triplet state simply according to spin Pauli exclusion principle.

In this work, the following formulation for the $g$-factor of an InAs-based QD
is adopted Hermann77 ; Bjork05

$g^{\ast}=g\left[1-\frac{P^{2}}{3}\frac{\Delta_{\rm SO}}{E_{g}^{\rm
eff}\left(E_{g}^{\rm eff}+\Delta_{\rm SO}\right)}\right]\,,$ (10)

where $E_{g}^{\rm eff}$ is the effective energy gap of semiconductor QD,
$g=2.0$ is the Lande $g$-factor for free electron, $\Delta_{\rm SO}$ is the
spin-orbit splitting in the valence band, and $P$ is the parameter of
interband transition matrix element. Hermann77

Here, the effective energy gap of a QD can be estimated as $E_{g}^{\rm
eff}=E_{g}^{\rm bulk}+\epsilon_{s,s_{z}}$, where $E_{g}^{\rm bulk}$ is the
bulk energy gap and $\epsilon_{s,s_{z}}$ is the quantization energy of the
lowest electronic orbital of the QD with $B=0$ measured from the conduction
band edge. For InAs-based QDs, we take the following parameter values:
$E_{g}^{\rm bulk}=460$ meV, $\Delta_{\rm SO}=390$ meV, $P^{2}=21.5$ eV.
Bjork05  Accordingly, the value of $g^{\ast}$ for a symmetric NWQD with ${\bf
L}_{0}={\bf L}_{z}=65$ nm is estimated as large as $g^{\ast}\approx-11$.
Bjork05

###  II.2 Interacting few-electron model

The interacting Hamiltonian of few electrons in a NWQD can be expressed in the
form of second quantization as

$\displaystyle H$ $\displaystyle=$
$\displaystyle\sum_{i,\sigma}\epsilon_{i\sigma}c_{i\sigma}^{{\dagger}}c_{i\sigma}$
(11) $\displaystyle+\frac{1}{2}\sum_{ijkl,\sigma\sigma^{\prime}}\langle
ij|V|kl\rangle
c_{i\sigma}^{{\dagger}}c_{j\sigma^{\prime}}^{{\dagger}}c_{k\sigma^{\prime}}c_{l\sigma}\,,$

where $i,j,k,l$ denote the composite indices of single electron orbitals such
as $|i\rangle=|n_{i},m_{i},q_{i}\rangle$, $c_{i\sigma}^{{\dagger}}$
($c_{i\sigma}$) the electron creation (annihilation) operators, and
$\sigma=\pm$ the electron spins $s_{z}=\pm\frac{1}{2}$. The first (second)
term on the right hand side of Eq.(11) represents the kinetic energy of
electrons (the Coulomb interactions between electrons) and the Coulomb matrix
elements are defined as

$\displaystyle\langle ij|V|kl\rangle$ $\displaystyle\equiv$
$\displaystyle\frac{e^{2}}{4\pi\kappa}\int\int d{\bf r_{1}}d{\bf
r_{2}}\psi_{i}^{*}({\bf r_{1}})\psi_{j}^{*}({\bf r_{2}})$ (12)
$\displaystyle\times\frac{1}{|{\bf r_{1}}-{\bf r_{2}}|}\psi_{k}({\bf
r_{2}})\psi_{l}({\bf r_{1}})\,,$

where $\kappa$ is the dielectric constant of dot material. For InAs material,
we take $\kappa=15.15$. After lengthy derivation, one can obtain the
generalized Coulomb matrix elements for the case of $a\geq 1$:

$\displaystyle\langle
n_{i}m_{i}q_{i};n_{j}m_{j}q_{j}|V|n_{k}m_{k}q_{k};n_{l}m_{l}q_{l}\rangle=$
$\displaystyle\left(\frac{1}{\pi
l_{h}}\right)\frac{\delta_{R_{L},R_{R}}\cdot\delta_{q_{i}+q_{j}+q_{l}+q_{k},\rm{even}}}{\sqrt{n_{i}!m_{i}!q_{i}!n_{j}!m_{j}!q_{j}!n_{k}!m_{k}!q_{k}!n_{l}!m_{l}!q_{l}!}}$
$\displaystyle\times\sum_{p_{1}=0}^{\min(n_{i},n_{l})}\sum_{p_{2}=0}^{\min(m_{i},m_{l})}\sum_{p_{3}=0}^{\min(q_{i},q_{l})}\sum_{p_{4}=0}^{\min(n_{j},n_{k})}\sum_{p_{5}=0}^{\min(m_{j},m_{k})}\sum_{p_{6}=0}^{\min(q_{j},q_{k})}\
p_{1}!p_{2}!p_{3}!p_{4}!p_{5}!p_{6}!$ $\displaystyle\times{n_{i}\choose
p_{1}}{n_{l}\choose p_{1}}{m_{i}\choose p_{2}}{m_{l}\choose
p_{2}}{q_{i}\choose p_{3}}{q_{l}\choose p_{3}}{n_{j}\choose
p_{4}}{n_{k}\choose p_{4}}{m_{j}\choose p_{5}}{m_{k}\choose
p_{5}}{q_{j}\choose p_{6}}{q_{k}\choose p_{6}}$
$\displaystyle\times(-1)^{u+v/2+n_{j}+m_{j}+q_{j}+n_{k}+m_{k}+q_{k}}{\left(\frac{1}{2}\right)^{u}}x^{u+1/2}$
$\displaystyle\times\frac{\Gamma(\frac{1+2u+v}{2})\Gamma(1+u)\Gamma(\frac{1+v}{2})}{\Gamma(\frac{3+2u+v}{2})}{{}_{2}F_{1}}\left(1+u,\frac{1+2u+v}{2};\frac{3+2u+v}{2};1-x\right)\,,$
(13)

where we define $u=m_{i}+m_{j}+n_{l}+n_{k}-(p_{1}+p_{2}+p_{4}+p_{5})$,
$v=(q_{i}+q_{l}+q_{j}+q_{k})-2(p_{3}+p_{6})$,
$R_{L}=(m_{i}+m_{j})-(n_{i}+n_{j})=-(\ell_{z,i}+\ell_{z,j})$,
$R_{R}=(m_{l}+m_{k})-(n_{l}+n_{k})=-(\ell_{z,l}+\ell_{z,k})$,
$x\equiv\omega_{z}/\omega_{h}$, and ${}_{2}F_{1}$ is the hypergeometric
function. The $\delta$-functions $\delta_{q_{i}+q_{j}+q_{l}+q_{k},\rm{even}}$
and $\delta_{R_{L},R_{R}}$ in the formulation ensure the conservation of the
parity with respect to $z$-axis and the $z$-component of angular momentum of
system $L_{z}$, respectively. The formulation of Eq. (13) is confirmed by
computing the Coulomb integral numerically.

For short NWQDs with $a<1$, the formulations of the Coulomb matrix elements
are obtained by simply taking Euler’s hypergeometric transformation for the
hypergeometric function in Eq. (13), i.e., replacing

${{}_{2}F_{1}}\left(1+u,\frac{1+2u+v}{2};\frac{3+2u+v}{2};1-x\right)\,$

by

$x^{-\frac{1+2u+v}{2}}{{}_{2}F_{1}}\left(\frac{1+v}{2},\frac{1+2u+v}{2};\frac{3+2u+v}{2};1-\frac{1}{x}\right)\,.$

The generalized formulations for the Coulomb matrix elements based on the 3D
asymmetric parabolic model are probably for the first time derived, which
allows for straightforward implementation of the CI theory and is widely
applicable to arbitrary 3D confining semiconductor nanostructures.

###  II.3 Exact diagonalization

Based on the CI theory presented above, we follow the standard numerical exact
diagonalization procedure to calculate the energy spectrum of $N_{e}$
interacting electrons in a NWQD . Hawrylak03  The numerically exact results
are obtained by increasing the numbers of chosen single electron orbital basis
and the corresponding $N_{e}$-electron configurations until a numerical
convergence is achieved. In the full configuration interaction (FCI)
calculation for a 2e problem, we usually take the number of single electron
orbitals typically from $20$ to $26$ and that of the corresponding 2e
configurations from $190$ to $325$ to have a satisfactory numerical
convergence.

##  III Numerical results and discussion

###  III.1 Magnetic-field induced ST transitions

Figure 3: (Color online) Magneto-energy spectrum of two interacting electrons
in a NWQD with transverse confining strength $\hbar\omega_{0}=13.3$ meV and
aspect ratio $a=3$.

Let us first consider two interacting electrons in a rod-like NWQD with the
aspect ratio $a=3$ and the transverse confining strength
$\hbar\omega_{0}=13.3$ meV using FCI calculation. The low-lying magneto-energy
spectrum of the two-electron NWQD is shown in Fig. 3, which consists of a spin
singlet state branch, labeled by $\rm{S}$, and three triplet state branches
split by the spin Zeeman energy, labeled by $\rm{T_{L0}^{+}}$,
$\rm{T_{L0}^{0}}$ and $\rm{T_{L0}^{-}}$ according to the z-component of total
spin ($S_{z}=+1$, $S_{z}=0$ and $S_{z}=-1$), respectively. Fasth07 ; Hanson07
Since usually only triplet states with $S_{z}=+1$ are involved in ST
transitions, we shall use $\rm T_{\rm L\it{\left|L_{z}\right|}}$ to denote the
triplet states with angular momentum $L_{z}$ through out this article,
skipping the superscript $+$ of $\rm{T_{L\it{\left|L_{z}\right|}}^{+}}$ for
brevity.

The main configurations of the two-electron ground states around the critical
magnetic field are schematically shown in the lower right corner of Fig. 3. In
the weak magnetic field regime $\left(B<B_{\rm{ST_{L0}}}\sim 0.9~{}\rm
T\right)$, the two electrons in the NWQD mainly occupy the lowest $s$-orbital
simply following the Aufbau principle, and form a spin singlet ground state.
With increasing $B$, the triplet state $\rm{{T_{L0}}}$ is more energetically
favorable than the singlet state because of the increasing spin Zeeman energy,
the reduced Coulomb repulsion, and exchange energy between the two spin
polarized electrons. A crossing of the singlet branch and the triplet state
branch $\rm{T_{L0}}$ is observed at the critical magnetic field
$B_{\rm{ST_{L0}}}=0.9$ T. Such magnetic-field induced ST transitions are
attributed to the energetic competition between single particle energy
quantization, the spin Zeeman energy, and the various Coulomb interactions
including the direct, exchange, and correlation interactions as well.
Hawrylak93-prl

Other weak spin-related terms, such as the spin-orbital coupling (SOC) with
1-2 order of magnitude smaller than the kinetic quantization of QD are
neglected in the Hamiltonian of Eq.(11). The SOC mixes the spin of the
$\rm{S}$ and $\rm{{T_{L0}}}$ states and creates an anti-crossing of the S- and
$\rm{{T_{L0}}}$-branches around the $B_{ST}$ with a small energy gap,
typically only $\sim 0.1-0.5$ meV as observed in previous experiments. Fasth07

###  III.2 Spin phase diagram

Figure 4: (Color online) Spin phase diagrams of two-electron NWQDs of lateral
confinement $\hbar\omega_{0}=13.3$ meV with respect to tunable magnetic field
$B$ and aspect ratio $a$. The phases are distinguished by the curves of
critical magnetic field $B_{\rm{ST}}$ obtained from non-interacting (black
dotted), PCI (blue dashed), and FCI (red solid) calculations.

Figure 4 shows the calculated spin phase diagrams of the two-electron ground
state of the NWDQs with a fixed cross section diameter (fixed
$\hbar\omega_{0}=13.3$ meV) but various lengths (various $\hbar\omega_{z}$)
with respect to the applied magnetic field $B$ and the aspect ratio $a$. Three
phases ($\rm S$, $\rm{T_{L0}}$, and $\rm{T_{L1}}$) are distinguished by the
curves of critical magnetic field $B_{\rm{ST}}$ in Fig. 4. Correspondingly,
the main configurations of the 2e ground states are depicted inside the
colored regions of the phases. To identify the various underlying mechanisms
in the phase diagrams, including the spin Zeeman effect and the inter-particle
Coulomb interactions, the spin phase diagrams are calculated by using non-
interacting, full CI, and partial CI calculations, respectively.

In the non-interacting calculation, the coulomb interactions are artificially
disabled and the considered ST transitions are induced only by the spin Zeeman
effect. The comparison between the results of non-interacting and FCI
calculations allows us to distinguish effects of the Coulomb interaction and
spin Zeeman coupling on the ST transitions. In particular, to highlight the
Coulomb correlation effect, a partial configuration interaction (PCI)
calculation is also performed for the spin phase diagrams, in which only the
lowest energy configuration is taken as the sole basis and the couplings from
higher energy configurations are excluded.

The essential features of the phase diagrams can be realized based on the non-
interacting picture. For a not very long (small or moderate $a$) NWQD with
weak $B$, the 2e ground state is likely to be the spin singlet state $\rm S$,
simply following Aufbau principle (the yellow region in Fig. 4). Starting from
the singlet phase $\rm S$, the two-electron ground state of a NWQD might be
switched to the spin triplet phases (the pink region $\rm{T_{L0}}$ or the cyan
region $\rm{T_{L1}}$) by increasing either $a$ or $B$ (see the horizontal and
vertical dashed lines with arrows in Fig. 4 for the guidance of eyes).

Following the horizontal dashed line, the longitudinal energy quantization
$\hbar\omega_{z}$ is decreased by the increase of $a$. With the addition of
spin Zeeman term, the spin-up level of $p^{0}$-orbital could become even lower
than the spin-down level of $s$-orbital if the decreasing $\hbar\omega_{z}$ is
so small as that $\hbar\omega_{z}<2|E_{z}|$ (see the difference between the
schematic configurations for the $\rm S$ and $\rm{T_{L0}}$ states). In this
situation, the 2e ground state can transit to the spin triple states
$\rm{T_{L0}}$, simply following spin Pauli exclusion principle. On the other
hand, the transition of a 2e ground state of NWQD from the singlet state $\rm
S$ to the triple one $\rm{T_{L1}}$ is shown also possible by increasing the
strength of applied magnetic field. Following the vertical dashed line,
increasing $B$ reduce the energy separation between the $s$\- and
$p^{-}$-orbital levels, i.e. $\hbar\omega_{-}$. Similar to the case of
$\rm{S}$-$\rm{T_{L0}}$ transition, a $\rm S$-$\rm{T_{L1}}$ transition can
happen as the decreased $\hbar\omega_{-}$ is so small as that
$\hbar\omega_{-}<2|E_{z}|$ . In the non-interacting picture, $B_{\rm ST_{L0}}$
is explicitly given by $B_{\rm ST_{L0}}=\hbar\omega_{0}/g^{\ast}\mu_{B}a^{2}$,
showing a quadratic decay with $a$, while the critical magnetic field $B_{\rm
ST_{L1}}$ for $\rm{S}$-$\rm{T_{L1}}$ transitions is dependent only on
$\hbar\omega_{0}$ and remains nearly constant in the $B-a$ plot.

The Coulomb interactions are shown to reduce the singlet phase area in the
diagrams from the comparison between the non-interacting and CI results. For
example, the segment of vertical solid line at $a=3$ in Fig. 4 indicates that
the critical magnetic field is significantly reduced from $B_{\rm
ST_{L0}}(\rm{Non-interacting})=3.2$ T to $B_{\rm ST_{L0}}(\rm{FCI})=0.9$ T as
the Coulomb interactions are taken into account. This is because the spin
triplet states gain additional negative exchange energies while the singlet
state does not. We also notice that the $B_{\rm ST_{L1}}$ for the
$\rm{S}$-$\rm{T_{L1}}$ transition no longer remains constant but slightly
increases with increasing $a$ because the strength of the coulomb interactions
is reduced by the increase of dot volume.

Basically, the results obtained from the FCI and PCI calculations have similar
features except for those in the regime of high $a$ ($a>3$). While the PCI
calculation shows the vanishing $B_{\rm ST_{L0}}$ for $a\sim 3$, the FCI
calculation yields the always non-zero $B_{\rm ST_{L0}}$. This means that the
Coulomb correlations energetically favor the spin singlet state as ground
state and become more pronounced in long NWQDs.

###  III.3 Crossover from disk-like to rod-like QDs

The spin phase diagrams of Fig. 4 suggest that purposely accessing a specific
spin phase of two-electron is feasible through the geometrical control of
NWQDs. For instance, the ground state of a two-electron NWQD can be switched
from the singlet $\rm{S}$ to the triplet state $\rm{T_{L0}}$ by increasing the
aspect ratio $a$ at the fixed $B=5$ T (trace the horizontal dashed line in
Fig. 4).

Figure 5: (Color online) Spin phase diagrams of doubly charged NWQDs with
respect to the lateral and longitudinal confinements, parametrized by
$\hbar\omega_{0}$ and $\hbar\omega_{z}$, respectively, in a fixed magnetic
field $B=5$ T for (a) non-interacting two electrons with $g^{\ast}\neq 0$, (b)
interacting two electrons with $g^{\ast}\neq 0$ and (c) interacting two
electrons with $g^{\ast}=0$.

Figure 5 presents the spin phase diagrams of two-electron NWQDs with respect
to the lateral and longitudinal confinements, parametrized by
$\hbar\omega_{0}$ and $\hbar\omega_{z}$, respectively, in a fixed magnetic
field $B=5$ T for (a) non-interacting two electrons with $g^{\ast}\neq 0$, (b)
interacting two electrons with $g^{\ast}\neq 0$, and (c) interacting two
electrons with $g^{\ast}=0$. In Figs. 6(a) and (b), we present the relevant
two-electron configurations to the spin phase diagrams of Figs. 5(a) and (b)
with the inclusion of spin Zeeman effect $\left(g^{\ast}\neq 0\right)$, while
in Figs. 6(c) and (d) we present the relevant two-electron configurations to
the spin phase diagrams of Fig. 5(c) for $g^{\ast}=0$

Figure 6: (Color online) Relevant two-electron configurations possibly being
the main components in the ground states of NWQDs in an uniform magnetic field
$B=5$ T for (a) $a>1$ and $g^{\ast}\neq 0$, (b) $a<1$ and $g^{\ast}\neq 0$,
(c) $a>1$ and $g^{\ast}=0$, and (d) $a<1$ and $g^{\ast}=0$.

The non-interacting spin phase diagram is first shown in Fig. 5(a) in order to
identify the spin Zeeman effect and also contrast the Coulomb interaction
effects on the interacting spin phase diagrams presented in Fig. 5(b). In the
non-interacting case, the features of the spin phases of Fig. 5(a) are purely
determined by the competition between geometry-dependent quantized electronic
structures of dots and the spin Zeeman splitting, which is nearly a constant
here created by the fixed $B$. Three distinctive spin phases, $\rm{S}$,
$\rm{T_{L0}}$, and $\rm{T_{L1}}$, are marked in different colors in Fig. 5(a).
In the yellow region where both $\hbar\omega_{0}$ ($\hbar\omega_{-}$) and
$\hbar\omega_{z}$ are large, the kinetic quantizations in both longitudinal
and lateral directions are stronger than the spin Zeeman splitting and
$\rm{S}$ remains as a ground state. Reducing the longitudinal confinement,
$\hbar\omega_{z}$, can lead to the $\rm{S}$-$\rm{T_{L0}}$ (from the yellow to
the pink region) transition as $\hbar\omega_{z}\lesssim 2|E_{\rm{Z}}|$.
Similarly, reducing the transverse confinement leads to the
$\rm{S}$-$\rm{T_{L1}}$ transition as $\hbar\omega_{-}\lesssim
2E_{\rm{Z}}$(from the yellow to the light cyan region).

Compared with Fig. 5(a), the interacting spin phase diagram of Fig. 5(b) shows
the following additional features:

(i) Larger areas of both $\rm{T_{L0}}$ and $\rm{T_{L1}}$ phases are observed
because of the additional negative exchange energies and the reduced direct
Coulomb repulsions gained by the triplet states.

(ii) A NWQD with $\hbar\omega_{0}\approx 12$ meV could experience a three-
phase transitions from $\rm{T_{L1}}$ (cyan) to $\rm{S}$ (yellow), and then to
$\rm{T_{L0}}$ (pink) with increasing the length of wire, from
$\hbar\omega_{z}>25$ meV to $\hbar\omega_{z}<5$ meV (see the vertical line
positioned at $\hbar\omega_{0}=12$ meV in Fig. 5(b)).

(iii) In the regime of small $\hbar\omega_{0}$ and large $\hbar\omega_{z}$
(i.e. flat quasi-2D dots with $a\ll 1$), a series of transitions from the spin
single states to various triplet states, $\rm{T_{L1}}$, $\rm{T_{L3}}$,
$\rm{T_{L5}}$, etc. [see Fig. 6(b)] and a staircase increase of total orbital
angular momentum are observed with reducing the lateral confinement
$\hbar\omega_{0}$.

In the weak laterally confining regime, few electrons in the quasi-2D QD in a
high magnetic field successively fill the orbitals with negative
$z$-projection of orbital angular momentum, i.e. the orbitals of lowest Landau
level (LLL), with small kinetic energy separation $\hbar\omega_{-}$. The
inter-particle Coulomb interactions thus become particularly pronounced among
the particles on the nearly degenerate LLL orbitals with alomost quenched
kinetic energies. In order to minimize the coulomb repulsion, the particles on
the quasi-degenerate orbitals tend to spread the occupancy of orbitals as far
as possible, but in competition with the cost of increase of kinetic energy.
As a result, with reducing $\hbar\omega_{0}$ or increasing $B$, the total
angular momentum of two-electron increases, as previously discussed by Wagner
et al. Wagner92  for gated 2D QDs.

Figure 5(c) shows the phase diagram of two interacting electrons calculated by
FCI method but with the vanishing spin Zeeman term, i.e. $g^{\ast}=0$. This
allows us to distinguish the effects of spin Zeeman energy and the coulomb
interactions on the spin phase diagram of Fig. 5(b), and also to study the
spin phases of QD made of a material with small $g^{\ast}$ such as GaAs.
Without spin Zeeman splitting, the significant features of Fig. 5(c) are
completely determined by the many-body effects and geometry-engineered
electronic structures of NWQDs.

In the $a>1$ regime, unlike the result shown in Fig. 5(b), the $\rm{T_{L0}}$
phase disappears and naturally there is no $\rm{S}$-$\rm{T_{L0}}$ transition
observed. This is because the Coulomb correlations that energetically favor
spin-singlet state as mentioned previously, become dominant and compensate the
negative exchange energy gained by the $\rm{T_{L0}}$ states. Reimann02 ;
Hanson07  However, in the small $\hbar\omega_{0}$ regime, an additional
singlet-triplet state oscillation with decreasing $\hbar\omega_{0}$ is
observed. Compared with Figure 5(b), the difference is the emergences of
various singlet states between the triplet phases. This is due to the removal
of spin Zeeman splittings, which energetically favor only the triplet states.
Such a singlet-triplet state oscillation is evidenced as a main feature of a
flat 2D QD with small spin Zeeman effect, as shown both theoretically Wagner92
and experimentally Tarucha07  in the previous studies.

##  IV Summary

In conclusion, we present exact diagonalization studies of spin phase
transitions of two electrons confined in nanowire quantum dots with highly
tunable aspect ratio and external magnetic field. A configuration interaction
theory based on a 3D parabolic model for such three dimensionally confining
QDs is developed, which provides generalized explicit formulation of the
Coulomb matrix elements and allows for straightforward implementation of
direct diagonalization. The exact diagonalization study reveals fruitful
features of spin ST transitions with respect to the tunable geometric aspect
ratio and applied magnetic field.

For disk-like QDs, the ST transition behaviors may be dominated by the spin
Zeeman, the direct-Coulomb, and the exchange energies. The pronounced Coulomb
correlations are identified in rod-like QDs with aspect ratio $a>3$, which
energetically favor singlet spin states and yield the always non-zero critical
magnetic fields of ST transitions. The developed theory is further employed to
study spin phase diagram in the dimensional “cross over” regime from the 2D
(disk-like) QDs to finite 1D (rod-like) QDs. In the 2D disk-like QD regime,
various distinctive spin phases are emerged under the conditions of
appropriate lateral confinement strength and magnetic fields. In the rod-like
QD regime, switching the ST transitions is shown feasible by controlling both
lateral and/or longitudinal confinement strength.

##  V Acknowledgement

This work was financially supported by the National Science Council in Taiwan
through Contracts No. NSC-98-2112-M-009-011-MY2 (SJC) and No.
NSC97-2112-M-239-003-MY3 (CST). The authors are grateful to the facilities
supported by the National Center of Theoretical Sciences in Hsinchu and the
National Center for High-Performance Computing in Taiwan.

## References

  * (1) J. R. Petta, A.C. Johnson, J. M. Taylor, E. A. Laird, A. Yacoby, M. D. Lukin, C. M. Marcus, M. P. Hanson, and A. C. Gossard, Science 309, 2180 (2005).
  * (2) D. Loss and D. P. DiVincenzo, Phys. Rev. A 57, 120 (1998).
  * (3) J. Fischer, M. Trif, W. A. Coish, and D. Loss, Solid State Comm. 149, 1443 (2009).
  * (4) L. P. Kouwenhoven, T. H. Oosterkamp, M. W. S. Danoesastro, M. Eto, D. G. Austing, T. Honda, and S. Tarucha, Science 278, 1788 (1997).
  * (5) L. P. Kouwenhoven, D. G. Austing, and S. Tarucha, Rep. Prog. Phys. 64, 701 (2001).
  * (6) S. M. Reimann, and M. Manninen, Rev. Mod. Phys. 74, 1283 (2002).
  * (7) C. Ellenberger, T. Ihn, C. Yannouleas, U. Landman, K. Ensslin, D. Driscoll, and A. C. Gossard, Phys. Rev. Lett. 96, 126806 (2006).
  * (8) J. Kyriakidis, M. Pioro-Ladriere, M. Ciorga, A. S. Sachrajda and P. Hawrylak, Phys. Rev. B 66, 035320 (2002).
  * (9) M. Wagner, U. Merkt, and A. V. Chaplik, Phys. Rev. B 45, 1951 (1992).
  * (10) P. Hawrylak, Phys. Rev. Lett. 71, 3347 (1993).
  * (11) B. Partoens, A. Matulis, and F. M. Peeters, Phys. Rev. B 59, 1617 (1999).
  * (12) Y. Nishi, Y. Tokura, J. Gupta, G. Austing, and S. Tarucha, Phys. Rev. B 75, 121301(R) (2007).
  * (13) C. Fasth, A. Fuhrer, M. T. Björk, and L. Samuelson, Nano Lett. 5, 1487 (2005).
  * (14) A. Pfund, I. Shorubalko, R. Leturcq, and K. Ensslin, Appl. Phys. Lett. 89, 252106 (2006).
  * (15) A. Pfund, I. Shorubalko, K. Ensslin, and R. Leturcq, Phys. Rev. Lett. 99, 036801 (2007).
  * (16) M. T. Björk, C. Thelander, A. E. Hansen, L. E. Jensen, M. W. Larsson, L. R. Wallenberg, and L. Samuelson, Nano Lett. 4, 1621 (2004).
  * (17) M. T. Björk, A. Fuhrer, A. E. Hansen, M. W. Larsson, L. E. Fröberg, and L. Samuelson, Phys. Rev. B 72, 201307(R) (2005).
  * (18) C. Fasth, A. Fuhrer, L. Samuelson, V. N. Golovach, and D. Loss, Phys. Rev. Lett. 98, 266801 (2007).
  * (19) A. Wensauer, M. Korkusiński, and P. Hawrylak, Solid State Comm. 130, 115 (2004).
  * (20) M. Dineykhan, and R. G. Nazmitdinov, Phys. Rev. B 55, 13707 (1997).
  * (21) J. T. Lin, and T. F. Jiang, Phys. Rev. B 64, 195323 (2001).
  * (22) M. T. Björk, B. J. Ohlsson, C. Thelander, A. I. Persson, K. Deppert, L. R. Wallenberg, and L. Samuelson, Appl. Phys. Lett. 81, 4458 (2002).
  * (23) C. Hermann and C. Weisbuch, Phys. Rev. B 15, 823 (1977).
  * (24) R. Hanson, L. P. Kouwenhoven, J. R. Petta, S. Tarucha and L. M. K. Vandersypen, Rev. Mod. Phys. 79, 1217 (2007).


Sequential Decision Problems with Weak Feedback
Arun Verma

Doctor of Philosophy
Industrial Engineering and Operations Research
Prof. Manjesh K. Hanawal

Prof. N. Hemachandra

Dedicated to my beloved parents.

[pages=-, offset=75 -75]chapter0/thesis_approval.pdf

CHAPTER: ACKNOWLEDGMENT

CHAPTER: PUBLICATIONS

CHAPTER: INTRODUCTION

Many real-life systems involve actions being applied sequentially, with each action resulting in a certain reward. The goal is to apply an action from a set of available actions that results in maximum reward (or minimum cost) in each round. However, which action is the best in a given round, may not be known. In such systems, the `goodness' of action has to be inferred from the results of past actions. For example, in the medical diagnosis of an ailment, a doctor prescribes a treatment from a set of available treatments, and by observing their effectiveness, narrows down on the most appropriate treatment [Thompson, 1933]. Such a decision-making process is commonly known as online learning in literature, where the goal is to learn the best action as quickly as possible.

Online machine learning is a branch of machine learning where actions are made sequentially using past feedback, and associated rewards of actions are observed. As opposed to batch learning where all the data are available beforehand, in online machine learning, an instance gets revealed in each round, and after selecting an action, feedback is observed. The feedback could reveal how good the selected action is or may not provide any information at all. The goal is to identify the best action to play in each round. In the batch learning methods, the entire data are available in one go, and the best predictor is obtained using all the data and used for prediction on any new instance. In online learning methods, learning is incremental: as more and more feedback is available, the action to apply in the next instance is dynamically updated using all the past feedback.

Several variants of batch learning methods have been studied in the past: supervised, unsupervised, and active learning. In supervised learning methods, true labels for all data points are known, whereas, in unsupervised methods, they are not known [Friedman et al., 2001, Shalev-Shwartz and Ben-David, 2014]. In the active learning (a special case of semi-supervised learning where the algorithm has a small labeled set of training instances with a larger unlabeled set), the labels are not available by default but can be acquired if necessary [Settles, 2010]. For example, obtaining labels could be expensive, in which case labels for the most 'important' data points can be acquired on payment or with lots of human effort. All three variants in the batch setting are well-studied in literature.

Many variants of online machine learning are considered in the literature, depending on the type of feedback and the amount of information they reveal about the rewards. The multi-armed bandits and the expert setting [Auer et al., 2002, Bubeck et al., 2012] are well-studied problems where feedback provides direct information about the rewards. In the multi-armed bandit setting, feedback observed from an action reveals only the reward associated with that action. However, in the expert setting, the feedback observed from an action reveals reward associated with the action played as well as all the other actions. The settings that span in between these two extreme cases are also studied, namely, bandits with side-information [Mannor and Shamir, 2011, Alon et al., 2013, Alon et al., 2015, Wu et al., 2015].

The online learning methods studied in the literature work under the assumption that feedback from an action provides information about the reward of that action, though the amount of information revealed could vary from one action to another. It is also possible that the feedback from actions is indirectly tied to the rewards. This setting is referred as partial monitoring setting [Cesa-Bianchi et al., 2006, Bartók and Szepesvári, 2012, Bartók et al., 2014]. It includes all the previously described online learning setups as special cases. A summary of the various online learning setups discussed is shown in below Figure <ref>.

Classification of various online learning settings.

The finite partial monitoring setting [Bartók et al., 2014] is characterized by a pair of $K \times O$ matrices, the reward matrix $R$, and the feedback matrix $F$. In each round $t$, the learner selects an action $I_t \in \{1,2,\ldots, K\}$ and simultaneously, the environment selects an outcome $J_t \in \{1,2,\ldots, O\}$. Then, the learner observes entry $F_{I_t, J_t}$ as feedback and receives instantaneous reward $R_{I_t, J_t}$, which is not revealed to him. The feedback can be thought of as masked information about the outcome of $J_t$. Sometimes, $F_{I_t, J_t}$  may uniquely determine the outcome; otherwise, the feedback may give only partial or no information about the outcome. The learner's performance is measured in terms of regret, i.e., the total reward  $\sum_{t=1 }^T R_{I_t, J_t}$ of the learner is compared with the total reward of best-fixed action in hindsight. Generally, the regret grows with the number of rounds $T$. The learner must have sub-linear regret, which means that the learner's average per-round reward approaches the average per-round reward of the best action in hindsight.

The partial monitoring setting is the most general online machine learning setting, but most of the earlier work on partial monitoring is restricted to cases where feedback from the actions allows the learner to fully identify the rewards of the actions. However, in many areas like crowd-sourcing, medical diagnosis, security management, resource allocation, and many others, feedback from actions may not even be sufficient to fully identify their rewards. In this thesis, the algorithms that identify optimal actions without requiring explicit reward values are developed. Broadly, the aim is to identify the scenarios where the reward values cannot be identified and then develop algorithms to learn optimal action by exploiting the problem structure. We categorized such problem as sequential decision problems with weak feedback (SDWF).

Our main objective in this thesis is to identify scenarios when it is possible to act optimally in SDWF. Clearly, without any information about rewards, one cannot learn which action is good in general. So, the specific problem structure has to be exploited to learn good actions. We propose to identify a set of necessary and sufficient conditions that lead to the learnability of optimal actions. We then identify classes of problems that satisfy these conditions and prepare a common framework for such classes to study partial monitoring problems with weak feedback. For the class of SDWF problems that can be learned, we develop learning algorithms. The goal is to develop provably optimal algorithms that are also easy to implement in practice. The performance of the developed algorithms is evaluated on synthetic and real datasets drawn from various application fields and compared with natural benchmarks.

In this thesis, the following classes of SDWF problems are studied: Uncensored Sequential Selection, Censored Semi-Bandits, and Collaborative Learning in Multi-Player Multi-Armed Bandits. In subsequent, these classes of SDWF problems are briefly presented.

§.§ Unsupervised Sequential Selection

The problems involving the SDWF arise naturally in areas like security, medical diagnosis, and crowd-sourcing, where we may not benefit from knowing the ground-truth associated with the task. For example, in medical diagnosis, patients may not reveal the outcome of treatment due to privacy reasons. Hence, the treatment's effectiveness may not be known. In crowd-sourcing systems, crowd-sourced workers may not reveal their expertise levels, and hence their effectiveness is not known. We named this class of problems as `Unsupervised Sequential Selection' (USS).

The main difficulty in the study of the USS problem is the lack of reward information. Moreover, the techniques developed for the partial monitoring problem, where rewards are identifiable from feedback, are not applicable to the USS setup. However, one would still like to act optimally.  We develop algorithms that learn to select the optimal action when rewards are not fully identifiable from the feedback. Specifically, we exploit the problem structure to learn the optimal action rather than rely on the availability of the reward information.

§.§ Censored Semi-Bandits

Sequential allocation problems with a censored feedback structure have received significant interest in recent times [Abernethy et al., 2016]. Censoring occurs naturally in several applications such as police patrolling of opportunistic crimes, supplier selection, budget allocation, and several others. In all these applications, an allocation to a set of options (allocation of price to items, allocation of the patrol to locations, etc.) is made by a learner, and then the response to the allocation (purchase, crime, etc.) is observed. If a positive response is observed (crime occurs, a purchase is made), the learner receives information about the goodness of allocation. However, if a negative response (censored) is observed (no crime, no purchase, etc.), the learner cannot decide if it is because of a good allocation or because of an inherent low propensity of a response. The goal then is to make repeated allocations and learn the best possible allocation strategy. We model these problems as stochastic semi-bandits and refer to this class of problems as `Censored Semi-Bandits' (CSB).

In the CSB problem, the amount of reward information revealed to the learner via feedback depends on the resource allocation. Hence the reward is only partially observable from feedback. We develop algorithms that first estimate the threshold so that we fully observe the reward from feedback and then find the best resource allocation.

§.§ Distributed Learning in Multi-Player Multi-Armed Bandits

We consider a communication network consisting of multiple users and multiple channels where each user wishes to use one of the available channels. A user can transmit on any of the channels. To keep the model more general, users cannot communicate directly between them, and any information about the other users can be obtained only by collisions. In particular, if the user transmits on a channel, it gets a sample of the gain of the channel only if no other user transmits on that channel. Otherwise, it only gets to know that at least one other user is transmitted on that channel. We assume that the expected gain of each of the channels is distinct. Even though users cannot communicate and have to make decisions in a distributed fashion, we consider a non-strategic setting where users have a common goal to maximize the total gain in the network, i.e., to achieve an optimal allocation, which is realized when all the users occupy non-overlapping channels in the top channels. Here the top channels refer to the set of channels with the highest expected gain. As the goal is to achieve optimal network allocation,  we assume that each user is satisfied if she gets one of the top channels when no other better channel is free. Our aim in this work is to find a distributed strategy that minimizes the time by which the users reach an optimal allocation while keeping the regret and number of collisions as small as possible. In this thesis, this setup is modeled as `Multi-Player Multi-Armed Bandits' (MP-MAB).

The reward information is the same as feedback in the MP-MAB problem unless two players select the same arm. We propose distributed strategies that assign arms to players and then learn in a distributed fashion by using all players for collecting information about the arms with a minimal number of collisions.

§.§ Organization of Thesis

In <ref>, we introduce Unsupervised Sensor Selection problem. We provide an optimal algorithm for solving the USS problem under certain conditions. We study the contextual version of the USS problem in <ref>, where the optimal action could be task-dependent. In <ref>, we proposed a novel framework for resource allocation problems using a variant of semi-bandits named censored semi-bandits. We use censored semi-bandits for solving the stochastic network utility maximization problem. We showcase the application of the CSB setup for network utility maximization. In <ref>, we set up the channels selection problem in the communication network as a stochastic multi-player multi-armed bandits. We developed distributed strategies without centralized communication that achieve constant regret with a high probability. Since we studied the different sequential decision problems in this thesis, the related literature is given in the respective chapters.

CHAPTER: UNSUPERVISED SEQUENTIAL SELECTION

In this chapter, we study Unsupervised Sequential Selection (USS) problem. The USS problem is a variant of the stochastic multi-armed bandits problem, where the loss of an arm can not be inferred from the observed feedback. In the USS setup, arms are associated with fixed costs and are ordered, forming a cascade. The learner selects an arm in each round and observes the feedback from arms up to the selected arm. The learner's goal is to find the arm that minimizes the expected total loss. The total loss is the sum of the cost incurred for selecting the arm and the stochastic loss associated with the selected arm. The problem is challenging because, without knowing the mean loss, one cannot compute the total loss for the selected arm. Clearly, learning is feasible only if the optimal arm can be inferred from the problem structure. It is observed that this happens if the problem satisfies the `Weak Dominance' (WD) property.  We set up the USS problem as a stochastic partial monitoring problem and develop algorithms with sub-linear regret under the WD property. We argue that our algorithms are optimal and evaluate its performance on problem instances generated from synthetic and real-world datasets.

§ UNSUPERVISED SEQUENTIAL SELECTION (USS)

§ PROBLEM SETTING

§ CONDITIONS FOR LEARNING OPTIMAL ARM

§ UPPER CONFIDENCE BOUND BASED ALGORITHM FOR USS PROBLEM

§ THOMPSON SAMPLING BASED ALGORITHM FOR USS PROBLEM

§ EXPERIMENTS

§ APPENDIX

CHAPTER: CONTEXTUAL UNSUPERVISED SEQUENTIAL SELECTION

In this chapter, we study Contextual Unsupervised Sequential Selection (USS), a variant of the stochastic contextual bandits problem where the loss incurred by the arm cannot be inferred from the observed feedback. In our setup, arms are associated with fixed costs and are ordered, forming a cascade. In each round, a context is presented, and the learner selects the arms sequentially till some depth.  The total cost incurred by stopping at an arm is the sum of fixed costs of arms selected and the stochastic loss associated with the arm.  The learner's goal is to learn a decision rule that maps contexts to arms with the goal of minimizing the total expected loss. The problem is challenging as we are faced with an unsupervised setting as the total loss cannot be estimated. Clearly, learning is feasible only if the optimal arm can be inferred (explicitly or implicitly) from the problem structure. We observe that learning is still possible when the problem instance satisfies the so-called `Contextual Weak Dominance' $(\CWD)$ property. Under $\CWD$, we propose an algorithm for the contextual USS problem and demonstrate that it has sub-linear regret. Experiments on synthetic and real datasets validate our algorithm.

§ CONTEXTUAL USS

§ PROBLEM SETTING

§.§ Contextual Weak Dominance

§ PARAMETERIZATION OF PAIRWISE DISAGREEMENT  PROBABILITY

§ ALGORITHM FOR CONTEXTUAL USS: <REF>

§ EXPERIMENT

§ APPENDIX

§.§ Algorithm with $\lambda\mathrm{I}_	{d^\prime}$ Initialization

§.§ More Experiments

CHAPTER: CENSORED SEMI-BANDITS

In this chapter, we consider the problem of sequentially allocating resources in a censored semi-bandits setup, where the learner allocates resources at each step to the arms and observes loss. The loss depends on two hidden parameters, one specific to the arm but independent of the resource allocation, and the other depends on the allocated resource. More specifically, the loss equals zero for an arm if the resource allocated to it exceeds a constant (but unknown) arm dependent threshold. The goal is to learn a resource allocation that minimizes the expected loss. The problem is challenging because the loss distribution and threshold value of each arm are unknown. We study this setting by establishing its `equivalence' to Multiple-Play Multi-Armed Bandits (MP-MAB) and Combinatorial Semi-Bandits. Exploiting these equivalences, we derive optimal algorithms for our problem setting using known algorithms for MP-MAB and Combinatorial Semi-Bandits. The experiments on synthetically generated data validate the performance guarantees of the proposed algorithms.

§ CENSORED SEMI-BANDITS (CSB)

§ PROBLEM SETTING

§ HORIZON-DEPENDENT ALGORITHMS WITH KNOWN LOWER BOUND ON MEAN LOSSES

§ ANYTIME ALGORITHMS

§ EXPERIMENTS

§ CSB FOR STOCHASTIC NETWORK UTILITY MAXIMIZATION

§ APPENDIX

CHAPTER: DISTRIBUTED LEARNING IN MULTI-PLAYER MULTI-ARMED BANDITS

We consider a communication network consisting of multiple users and multiple channels where each user wishes to use one of the available channels. We propose distributed strategies that minimize the sample complexity and regret of acquiring the best subset of channels. Agents cannot directly communicate with each other, and no central coordination is possible. Each agent can transmit on one channel at a time. If multiple agents transmit on the same channel simultaneously, a collision occurs. When a collision occurs, no agent gets any information about the channel gain or how many other agents are transmitting on that channel.  If no collision occurs, the agent observes a reward (or gain) sample drawn from an underlying distribution associated with the channel.  We modeled this problem as a Multi-Player Multi-Armed bandits problem. One important property of our algorithms that distinguishes it from the prior work is that it requires no information about the difference of the means of the channel gains. Our approach results in fewer collisions with improved regret performance compared to the state-of-the-art algorithms.  We validate our theoretical guarantees with experiments.

§  MULTI-PLAYER MULTI-ARMED BANDITS

§  PROBLEM SETUP

§ ALGORITHM WITH NO INFORMATION EXCHANGE

§ ALGORITHM WITH LIMITED INFORMATION EXCHANGE

§ EXPERIMENT

§ APPENDIX

CHAPTER: SUMMARY AND FUTURE DIRECTIONS

In this chapter, we summarize the various contributions of this thesis, followed by some interesting directions for future research.

§ SUMMARY

In this thesis, we consider sequential decision problems with weak feedback, where the loss incurred by selecting an action may not be inferred from observed feedback. A major part of the thesis (<ref> and <ref>) focuses on the unsupervised sequential selection problem, where the loss incurred for selecting an action can not be inferred from observed feedback. In <ref>, we introduced the Censored Semi-Bandits problem, where the loss incurred for selecting an action can be observed under certain conditions. Finally, we study the channel selection in the communication networks via distributed learning in Multi-Player Multi-Armed Bandits in <ref>, where the reward for an action is only observed in the round when no other player selects that action to play in that round.
The major contributions of this thesis are summarized as follows:

* Unsupervised Sequential Selection.   We study the unsupervised sequential selection (USS) problem in <ref>, where both mean loss and cost of using arms are important. It is a variant of the stochastic partial monitoring problem, where the losses are not observed. However, one can still compare the feedback of two arms to see if they agree or disagree. We estimate the disagreement probability between each pair of the arms and develop the upper confidence bound and Thompson Sampling based algorithms (<ref> and <ref>) that give sub-linear regret under the `Weak Dominance $(\WD)$ property.' We show that both algorithms enjoy regret of order $\tilde{O}(T^{2/3})$ (hiding logarithmic terms) for the USS problem that satisfies $(\WD)$ property.  We also show that when the problem instance satisfies the more stringent Strong Dominance property, the regret bound improves to $\tilde{O}(T^{1/2})$. We demonstrate our algorithms' performance on the problem instances derived from synthetic and two real datasets and empirically show that any USS problem instance satisfying $\WD$ property has sub-linear regret.

* Contextual Unsupervised Sequential Selection.   We extend the study of the unsupervised sequential selection setup to the case in <ref>, where the contextual information about the task is available. The problem is a partial monitoring stochastic contextual bandit, where the loss of an arm can not be inferred from the observed feedback. Since the disagreement between arms can be observed, we model the disagreement probability between each pair of the arms as linearly parameterized and develop an algorithm named <ref> that achieves $O(\log T)$ regret with high probability. The analysis of <ref> is based on recent results on the sample normality confidence bound for the maximum likelihood estimator of generalized linear models.

* Censored Semi-Bandits.   We introduce a novel framework for resource allocation problems using a variant of semi-bandits and name it censored semi-bandits (CSB) in <ref>. In the CSB setup, the loss observed from an arm depends on the amount of resource allocated, and hence, it can be censored. We propose a threshold-based model where loss from an arm is observed when the allocated resource is below a threshold. The goal is to assign a given resource to arms such that total expected loss is minimized. We consider two variants of the problem, depending on whether or not the thresholds are the same across the arms. For the variant where thresholds are the same across the arms, we establish that it is equivalent to the Multiple-Play Multi-Armed Bandit problem. For the second variant where the threshold can depend on the arm, we establish that it is equivalent to a more general Combinatorial Semi-Bandit problem. Exploiting these equivalences, we develop algorithms that enjoy optimal performance guarantees.  We also showcase the application of CSB setup to stochastic network utilization maximization by extending the CSB setup to reward maximization setting.

* Distributed Learning in Multi-Player Bandits.   Motivated by channel selection problem in communication networks, we consider a multi-player bandits problem in <ref>, where multiple players access the same set of arms in the absence of any central coordination. The players aim to maximize total reward in a distributed fashion. The mean rewards of arms and the number of players present in the network are unknown. We set up the problem as a stochastic multi-player multi-armed bandits and develop completely distributed algorithms that achieve constant regret with a high probability. We also empirically show that our algorithms have constant regret.

§ FUTURE RESEARCH DIRECTIONS

There are various open questions arising out of this thesis. A few of them are as follows:

* Unsupervised Sequential Selection.   We ignore the inherent side observations due to the arms' cascade structure. By using these side observations, we can have better regret bounds. Another interesting future direction is to develop algorithms that relax the cascade structure assumption and selects the best subset of arms. The problem-dependent lower bound of regret for the USS problem is still an open question.

* Contextual Unsupervised Sequential Selection.   We exploit the contextual information for a better selection of arm but ignore the inherent side observations due to the arms' cascade structure. By using these side observations, we can tighten the regret bounds.  In the contextual USS setup, the optimal action for contexts that do not satisfy $\WD$ cannot be learned. However, these contexts are still useful to improve estimates of the parameters and it will be interesting to characterize how much they will affect the regret. It will also be interesting to develop algorithms that decide whether it needs to go further down in the cascade when more information about context is revealed along the cascade. Many open questions like regret lower bounds for the contextual USS problem, contexts with private information, and the problems with real-valued feedback can be explored in the future.

* Censored Semi-Bandits.   The CSB setting considered so far does not use any similarity metric between the arms. For example, in the case of the police patrol allocation, the nearby nodes may have similar parameters, and we may be able to make use of the spatial coherence. It will be interesting to integrate existing work on contextual bandits with the graph structure and the CSB setup. As we only consider one type of resource, it will be interesting to consider multi-type resources into the CSB setup. Another extension of the CSB setup is to relax the assumption that the lower bound on leftover resources after the optimal allocation is known. One can also extend CSB setup from threshold type loss to continuous type loss functions and from the stochastic environment to the adversarial environment. Settings such as these are left for future work.

* Distributed Learning in Multi-Player Bandits.   We consider that the quality of the arms is the same across all the players (symmetric), and the number of users was fixed throughout (static). It will be interesting to study the setting, where the quality of arms could potentially differ across players (asymmetric) and allow the number of users to vary with time (dynamic networks). There are many interesting extensions of the proposed setup, like anytime algorithm for regret minimization, problems without collision sensing, and learning with asynchronous players that can be explored in the future.

[Abbasi-Yadkori et al., 2011]
Yasin Abbasi-Yadkori, Dávid Pál, and Csaba Szepesvári.
Improved algorithms for linear stochastic bandits.
In Advances in Neural Information Processing Systems, pages
2312–2320, 2011.

[Abernethy et al., 2016]
Jacob D Abernethy, Kareem Amin, and Ruihao Zhu.
Threshold bandits, with and without censored feedback.
In Advances In Neural Information Processing Systems, pages
4889–4897, 2016.

[Adler et al., 2014]
Nicole Adler, Alfred Shalom Hakkert, Jonathan Kornbluth, Tal Raviv, and Mali
Location-allocation models for traffic police patrol vehicles on an
interurban network.
Annals of Operations Research, 2210 (1):0
9–31, 2014.

[Agarwal et al., 2014]
Alekh Agarwal, Daniel Hsu, Satyen Kale, John Langford, Lihong Li, and Robert E.
Taming the monster: A fast and simple algorithm for contextual
In International Conference on Machine Learning, 2014.

[Agrawal and Goyal, 2012]
Shipra Agrawal and Navin Goyal.
Analysis of thompson sampling for the multi-armed bandit problem.
In Conference on Learning Theory, pages 39–1, 2012.

[Agrawal and Goyal, 2013]
Shipra Agrawal and Navin Goyal.
Further optimal regret bounds for thompson sampling.
In Artificial intelligence and statistics, pages 99–107,

[Alon et al., 2013]
Noga Alon, Nicolo Cesa-Bianchi, Claudio Gentile, and Yishay Mansour.
From bandits to experts: A tale of domination and independence.
In Advances in Neural Information Processing Systems, pages
1610–1618, 2013.

[Alon et al., 2015]
Noga Alon, Nicolo Cesa-Bianchi, Ofer Dekel, and Tomer Koren.
Online learning with feedback graphs: Beyond bandits.
In JMLR WORKSHOP AND CONFERENCE PROCEEDINGS, volume 40.
Microtome Publishing, 2015.

[Amin et al., 2012]
Kareem Amin, Michael Kearns, Peter Key, and Anton Schwaighofer.
Budget optimization for sponsored search: censored learning in mdps.
In Proceedings of the Twenty-Eighth Conference on Uncertainty
in Artificial Intelligence, pages 54–63, 2012.

[Anandkumar et al., 2011]
Animashree Anandkumar, Nithin Michael, Ao Kevin Tang, and Ananthram Swami.
Distributed algorithms for learning and cognitive medium access with
logarithmic regret.
IEEE Journal on Selected Areas in Communications, 290
(4):0 731–745, 2011.

[Anantharam et al., 1987]
Venkatachalam Anantharam, Pravin Varaiya, and Jean Walrand.
Asymptotically Efficient Allocation Rules for the Multiarmed Bandit
Problem with Multiple Plays-Part I: I.I.D. Rewards.
IEEE Transactions on Automatic Control, 320
(11):0 968–976, 1987.

[Auer, 2002]
Peter Auer.
Using confidence bounds for exploitation-exploration trade-offs.
Journal of Machine Learning Research, pages 397–422, 2002.

[Auer et al., 2002]
Peter Auer, Nicolo Cesa-Bianchi, and Paul Fischer.
Finite-time analysis of the multiarmed bandit problem.
Machine learning, pages 235–256, 2002.

[Avner and Mannor, 2014]
Orly Avner and Shie Mannor.
Concurrent bandits and cognitive radio networks.
In Proceedings of the Machine Learning and Knowledge Discovery
in Databases, Stanford, CA, 2014. Springer.

[Avner and Mannor, 2016]
Orly Avner and Shie Mannor.
Multi-user lax communications: A multi-armed bandit approach.
In IEEE International Conference on Computer Communications
(INFOCOM), San Francisco, CA, USA, 2016.

[Badanidiyuru et al., 2018]
Ashwinkumar Badanidiyuru, Robert Kleinberg, and Aleksandrs Slivkins.
Bandits with knapsacks.
Journal of the ACM (JACM), 650 (3):0 13, 2018.

[Balseiro and Gur, 2019]
Santiago R Balseiro and Yonatan Gur.
Learning in repeated auctions with budgets: Regret minimization and
Management Science, 650 (9):0 3952–3968,

[Baltaoglu et al., 2017]
M Sevi Baltaoglu, Lang Tong, and Qing Zhao.
Online learning of optimal bidding strategy in repeated
multi-commodity auctions.
In Advances in Neural Information Processing Systems, pages
4507–4517, 2017.

[Bartók and Szepesvári, 2012]
Gábor Bartók and Csaba Szepesvári.
Partial monitoring with side information.
In International Conference on Algorithmic Learning Theory,
pages 305–319. Springer, 2012a.

[Bartók and Szepesvári, 2012]
Gábor Bartók and Csaba Szepesvári.
Partial monitoring with side information.
In International Conference on Algorithmic Learning Theory,
pages 305–319. Springer, 2012b.

[Bartók et al., 2014]
Gábor Bartók, Dean P Foster, Dávid Pál, Alexander Rakhlin, and
Csaba Szepesvári.
Partial monitoring—classification, regret bounds, and algorithms.
Mathematics of Operations Research, 390 (4):0
967–997, 2014.

[Bengs and Hüllermeier, 2020]
Viktor Bengs and Eyke Hüllermeier.
Multi-armed bandits with censored consumption of resources.
arXiv preprint arXiv:2011.00813, 2020.

[Besson and Kaufmann, 2018]
Lilian Besson and Emilie Kaufmann.
Multi-player bandits revisited.
In Algorithmic Learning Theory, pages 56–92, 2018.

[Beygelzimer et al., 2011]
Alina Beygelzimer, John Langford, Lihong Li, Lev Reyzin, and Robert E.
Contextual bandit algorithms with supervised learning guarantees.
In International Conference on Artificial Intelligence and
Statistics (AISTATS), 2011.

[Bonald and Combes, 2017]
Thomas Bonald and Richard Combes.
A minimax optimal algorithm for crowdsourcing.
In Advances in Neural Information Processing Systems, pages
4352–4360, 2017.

[Boursier and Perchet, 2019]
Etienne Boursier and Vianney Perchet.
Sic-mmab: synchronisation involves communication in multiplayer
multi-armed bandits.
In Advances in Neural Information Processing Systems, pages
12071–12080, 2019.

[Bubeck et al., 2012]
Sébastien Bubeck, Nicolo Cesa-Bianchi, et al.
Regret analysis of stochastic and nonstochastic multi-armed bandit
Foundations and Trends® in Machine Learning,
50 (1):0 1–122, 2012.

[Cesa-Bianchi and Lugosi, 2012]
Nicolo Cesa-Bianchi and Gábor Lugosi.
Combinatorial bandits.
Journal of Computer and System Sciences, 780
(5):0 1404–1422, 2012.

[Cesa-Bianchi et al., 2006]
Nicolo Cesa-Bianchi, Gábor Lugosi, and Gilles Stoltz.
Regret minimization under partial monitoring.
Mathematics of Operations Research, 310 (3):0
562–580, 2006.

[Chapelle and Li, 2011]
Olivier Chapelle and Lihong Li.
An empirical evaluation of thompson sampling.
In Advances in neural information processing systems, pages
2249–2257, 2011.

[Chen et al., 2012]
M. Chen, Z. Xu, K.Q. Weinberger, O. Chapelle, and D. Kedem.
Classifier cascade: Tradeoff between accuracy and feature evaluation
In International Conference on Artificial Intelligence and
Statistics, pages 235–242, 2012.

[Chen et al., 2012]
Minmin Chen, Zhixiang Xu, Kilian Weinberger, Olivier Chapelle, and Dor Kedem.
Classifier cascade for minimizing feature evaluation cost.
In Artificial Intelligence and Statistics, pages 218–226,

[Chen et al., 2013]
Wei Chen, Yajun Wang, and Yang Yuan.
Combinatorial multi-armed bandit: General framework and applications.
In International Conference on Machine Learning, pages
151–159, 2013.

[Chen et al., 2016]
Wei Chen, Wei Hu, Fu Li, Jian Li, Yu Liu, and Pinyan Lu.
Combinatorial multi-armed bandit with general reward functions.
In Advances in Neural Information Processing Systems, pages
1659–1667, 2016.

[Chu et al., 2011]
Wei Chu, Lihong Li, Lev Reyzin, and Robert E Schapire.
Contextual bandits with linear payoff functions.
In International Conference on Artificial Intelligence and
Statistics, pages 208–214, 2011.

[Combes et al., 2015]
Richard Combes, Mohammad Sadegh Talebi Mazraeh Shahi, Alexandre Proutiere,
et al.
Combinatorial bandits revisited.
In Advances in Neural Information Processing Systems, pages
2116–2124, 2015.

[Curtin et al., 2010]
Kevin M Curtin, Karen Hayslett-McCall, and Fang Qiu.
Determining optimal police patrol areas with maximal covering and
backup covering location models.
Networks and Spatial Economics, 100 (1):0
125–145, 2010.

[Dagan and Crammer, 2018]
Yuval Dagan and Koby Crammer.
A better resource allocation algorithm with semi-bandit feedback.
In Algorithmic Learning Theory, pages 268–320, 2018.

[Dani et al., 2008]
Varsha Dani, Thomas P Hayes, and Sham M Kakade.
Stochastic linear optimization under bandit feedback.
In COLT, pages 355–366, 2008.

[Detrano, 1998]
Robert Detrano.
V.A. Medical Center, Long Beach and Cleveland Clinic Foundation:
Robert Detrano, MD, Ph.D., Donor: David W. Aha, 1998.
URL <https://archive.ics.uci.edu/ml/datasets/Heart+Disease>.

[Dheeru and Karra Taniskidou, 2017]
Dua Dheeru and Efi Karra Taniskidou.
UCI machine learning repository, 2017.
URL <http://archive.ics.uci.edu/ml>.

[Eryilmaz and Koprulu, 2017]
A. Eryilmaz and Irem Koprulu.
Discounted-rate utility maximization (drum): A framework for
delay-sensitive fair resource allocation.
In IEEE WiOpt, 2017.

[et al, 2018]
A. Ali et al.
Channel clustering and qos level identification scheme for
multi-channel cognitive radio networks.
IEEE Communication Magazine, 560 (4):0
164–171, 2018.

[Filippi et al., 2010]
Sarah Filippi, Olivier Cappe, Aurélien Garivier, and Csaba Szepesvári.
Parametric bandits: The generalized linear case.
In Advances in Neural Information Processing Systems, pages
586–594, 2010.

[Fontaine et al., 2020]
Xavier Fontaine, Shie Mannor, and Vianney Perchet.
An adaptive stochastic optimization algorithm for resource
In Algorithmic Learning Theory, pages 319–363. PMLR, 2020.

[Friedman et al., 2001]
Jerome Friedman, Trevor Hastie, and Robert Tibshirani.
The elements of statistical learning, volume 1.
Springer series in statistics New York, 2001.

[Gai and Krishnamachari, 2014]
Yi Gai and Bhaskar Krishnamachari.
Distributed stochastic online learning policies for opportunistic
spectrum access.
IEEE Transactions on Signal Processing, 620 (23),

[Garivier and Cappé, 2011]
Aurélien Garivier and Olivier Cappé.
The kl-ucb algorithm for bounded stochastic bandits and beyond.
In Proceedings of the 24th annual Conference On Learning
Theory, pages 359–376, 2011.

[Gholami et al., 2018]
Shahrzad Gholami, Sara Mc Carthy, Bistra Dilkina, , et al.
Adversary models account for imperfect crime data: Forecasting and
planning against real-world poachers.
In Proceedings of the 17th International Conference on
Autonomous Agents and MultiAgent Systems, pages 823–831, 2018.

[Greiner et al., 2002]
Russell Greiner, Adam J Grove, and Dan Roth.
Learning cost-sensitive active classifiers.
Artificial Intelligence, 1390 (2):0 137–174,

[Gummadi et al., 2011]
R Gummadi, Peter B Key, and Alexandre Proutiere.
Optimal bidding strategies in dynamic auctions with budget
In 2011 49th Annual Allerton Conference on Communication,
Control, and Computing (Allerton), pages 588–588. IEEE, 2011.

[Hanawal et al., 2017]
Manjesh Hanawal, Csaba Szepesvari, and Venkatesh Saligrama.
Unsupervised sequential sensor acquisition.
In Artificial Intelligence and Statistics, pages 803–811,

[Hanawal and Darak, 2018]
Manjesh K Hanawal and Sumit J Darak.
Multi-player bandits: A trekking approach.
arXiv preprint arXiv:1809.06040, 2018.

[Hifi and Mhalla, 2013]
Mhand Hifi and Hedi Mhalla.
Sensitivity analysis to perturbations of the weight of a subset of
items: The knapsack case study.
Discrete Optimization, 100 (4):0 320–330,

[Hoeffding, 1963]
Wassily Hoeffding.
Probability inequalities for sums of bounded random variables.
Journal of the American statistical association, 580
(301):0 13–30, 1963.

[Jain and Jamieson, 2018]
Lalit Jain and Kevin Jamieson.
Firing bandits: Optimizing crowdfunding.
In International Conference on Machine Learning, pages
2211–2219, 2018.

[Jamieson et al., 2013]
Kevin Jamieson, Matthew Malloy, and Robert Nowak.
lil'ucb:an optimal exploration algorithm for multi-armed bandits.
In Conference on Learning Theory (COLT), 2013.

[Jun et al., 2017]
Kwang-Sung Jun, Aniruddha Bhargava, Robert Nowak, and Rebecca Willett.
Scalable generalized linear bandits: Online computation and hashing.
In Advances in Neural Information Processing Systems, pages
99–109, 2017.

[Kaggle, 2016]
UCI Machine Learning, Kaggle.
Pima Indians Diabetes Database, 2016.
URL <https://www.kaggle.com/uciml/pima-indians-diabetes-

[Kalathil et al., 2014]
Dileep Kalathil, Naumaan Nayyar, and Rahul Jain.
Decentralized learning for multiplayer multiarmed bandits.
IEEE Transactions on Information Theory, 600
(4):0 2331–2345, 2014.

[Kalyanakrishnan et al., 2012]
Shivaram Kalyanakrishnan, Ambuj Tewari, Peter Auer, and Peter Stone.
Pac subset selection in stochastic multi-armed bandits.
In ICML, volume 12, pages 655–662, 2012.

[Kaufmann and Kalyanakrishnan, 2013]
Emilie Kaufmann and Shivaram Kalyanakrishnan.
Information complexity in bandit subset selection.
In Conference on Learning Theory, pages 228–251, 2013.

[Kaufmann et al., 2012]
Emilie Kaufmann, Nathaniel Korda, and Rémi Munos.
Thompson sampling: An asymptotically optimal finite-time analysis.
In International Conference on Algorithmic Learning Theory,
pages 199–213. Springer, 2012.

[Kelly, 1997]
F. P. Kelly.
Charging and rate control for elastic traffic.
European Transactions on Telecommunications, 80
(1):0 33–37, 1997.

[Kleindessner and Awasthi, 2018]
Matthäus Kleindessner and Pranjal Awasthi.
Crowdsourcing with arbitrary adversaries.
In International Conference on Machine Learning, pages
2713–2722, 2018.

[Komiyama et al., 2015]
Junpei Komiyama, Junya Honda, and Hiroshi Nakagawa.
Optimal regret analysis of thompson sampling in stochastic
multi-armed bandit problem with multiple plays.
In International Conference on Machine Learning, pages
1152–1161, 2015.

[Kumar et al., 2018]
R. Kumar, A. Yadav, S. J. Darak, and M. K. Hanawal.
Trekking based distributed algorithm for opportunistic spectrum
access in infrastructureless network.
In 16th International Symposium on Modeling and Optimization in
Mobile, Ad Hoc, and Wireless Networks (WiOpt), 2018.

[Lai et al., 2011]
Lifeng Lai, Hesham El Gamal, Hai Jiang, and H. Vincent Poor.
Cognitive medium access: Exploration, exploitation, and competition.
IEEE Transaction on Mobile Computing, 100 (2), 2011.

[Langford and Zhang, 2008]
John Langford and Tong Zhang.
The epoch-greedy algorithm for multi-armed bandits with side
In Advances in neural information processing systems, pages
817–824, 2008.

[Lattimore and Szepesvári, 2020]
Tor Lattimore and Csaba Szepesvári.
Bandit algorithms, 2020a.

[Lattimore and Szepesvári, 2020]
Tor Lattimore and Csaba Szepesvári.
Bandit algorithms.
Cambridge University Press, 2020b.

[Lattimore et al., 2014]
Tor Lattimore, Koby Crammer, and Csaba Szepesvári.
Optimal resource allocation with semi-bandit feedback.
In Proceedings of the Thirtieth Conference on Uncertainty in
Artificial Intelligence, pages 477–486. AUAI Press, 2014.

[Lattimore et al., 2015]
Tor Lattimore, Koby Crammer, and Csaba Szepesvári.
Linear multi-resource allocation with semi-bandit feedback.
In Advances in Neural Information Processing Systems, pages
964–972, 2015.

[Li et al., 2010]
Lihong Li, Wei Chu, John Langford, and Robert E Schapire.
A contextual-bandit approach to personalized news article
In Proceedings of the 19th international conference on World
wide web, pages 661–670. ACM, 2010.

[Li et al., 2017]
Lihong Li, Yu Lu, and Dengyong Zhou.
Provably optimal algorithms for generalized linear contextual
In International Conference on Machine Learning, pages
2071–2080, 2017.

[Liu and Zhao, 2010]
Keqin Liu and Qing Zhao.
Distributed learning in multi-armed bandit with multiple players.
IEEE Transactions on Signal Processing, 580 (11),

[Mannor and Shamir, 2011]
Shie Mannor and Ohad Shamir.
From bandits to experts: On the value of side-observations.
In Advances in Neural Information Processing Systems, pages
684–692, 2011.

[Mohri and Medina, 2014]
Mehryar Mohri and Andres Munoz Medina.
Learning theory and algorithms for revenue optimization in second
price auctions with reserve.
In International Conference on Machine Learning, pages
262–270. PMLR, 2014.

[Neely, 2010]
M. J. Neely.
Delay based network utility maximization.
In IEEE INFOCOM, 2010.

[Nguyen et al., 2016]
Thanh H Nguyen, Arunesh Sinha, Shahrzad Gholami, et al.
Capture: A new predictive anti-poaching tool for wildlife protection.
In Proceedings of the 2016 International Conference on
Autonomous Agents & Multiagent Systems, pages 767–775, 2016.

[Ozger et al., 2018]
Mustafa Ozger, Fatih Alagoz, and Ozgur B. Akan.
Clustering in multi-channel cognitive radio ad hoc and sensor
IEEE Communication Magazine, 560 (4):0
156–162, 2018.

[Palomar and Chiang, 2006]
D.P. Palomar and M. Chiang.
A tutorial on decomposition methods for network utility maximization.
IEEE Journal on Selected Areas in Communications, 240
(8):0 1439–1451, 2006.

[Palomar and Chiang, 2007]
D.P. Palomar and M. Chiang.
Alternative distributed algorithms for network utility maximization:
Framework and applications.
IEEE Transaction on Automatic Control, 520
(12):0 2254–2269, 2007.

[Parkvall et al., 2017]
Stefan Parkvall, Erik Dahlman, Anders Furuskár, and Mattas Frenne.
Nr: The new 5g radio access technology.
IEEE Communication Standards Magazine, 10
(4):0 24–30, 2017.

[Perrault et al., 2020]
Pierre Perrault, Etienne Boursier, Vianney Perchet, and Michal Valko.
Statistical efficiency of thompson sampling for combinatorial
In Advances in Neural Information Processing Systems, 2020.

[Platanios et al., 2017]
Emmanouil Platanios, Hoifung Poon, Tom M Mitchell, and Eric J Horvitz.
Estimating accuracy from unlabeled data: A probabilistic logic
In Advances in Neural Information Processing Systems, pages
4361–4370, 2017.

[Platanios et al., 2014]
Emmanouil Antonios Platanios, Avrim Blum, and Tom M Mitchell.
Estimating accuracy from unlabeled data.
In UAI, pages 682–691, 2014.

[Platanios et al., 2016]
Emmanouil Antonios Platanios, Avinava Dubey, and Tom Mitchell.
Estimating accuracy from unlabeled data: A bayesian approach.
In International Conference on Machine Learning, pages
1416–1425, 2016.

[Póczos et al., 2009]
Barnabás Póczos, Yasin Abbasi-Yadkori, Csaba Szepesvári, Russell
Greiner, and Nathan Sturtevant.
Learning when to stop thinking and do something!
In Proceedings of the 26th Annual International Conference on
Machine Learning, pages 825–832. ACM, 2009.

[Rajkumar and Agarwal, 2014]
Arun Rajkumar and Shivani Agarwal.
Online decision-making in general combinatorial spaces.
In Advances in Neural Information Processing Systems, pages
3482–3490, 2014.

[Rosenfeld and Kraus, 2017]
Ariel Rosenfeld and Sarit Kraus.
When security games hit traffic: Optimal traffic enforcement under
one sided uncertainty.
In IJCAI, pages 3814–3822, 2017.

[Rosenski et al., 2016]
Jonathan Rosenski, Ohad Shami, and Liran Szlak.
Multi-player bandits – a musical chairs approach.
In Proceedings of International Conference on Machine Learning
(ICML), New York, USA, 2016.

[Rusmevichientong and Tsitsiklis, 2010]
Paat Rusmevichientong and John N Tsitsiklis.
Linearly parameterized bandits.
Mathematics of Operations Research, 350 (2):0
395–411, 2010.

[Seldin et al., 2014]
Yevgeny Seldin, Peter L Bartlett, Koby Crammer, and Yasin Abbasi-Yadkori.
Prediction with limited advice and multiarmed bandits with paid
In ICML, pages 280–287, 2014.

[Settles, 2010]
Burr Settles.
Active learning literature survey. 2010.
Computer Sciences Technical Report, 1648, 2010.

[Shalev-Shwartz and Ben-David, 2014]
Shai Shalev-Shwartz and Shai Ben-David.
Understanding machine learning: From theory to algorithms.
Cambridge university press, 2014.

[Sinha and Modiano, 2018]
A. Sinha and E. Modiano.
Network utility maximization with heterogeneous traffic flows.
In IEEE WiOpt, 2018.

[Sinha et al., 2018]
Arunesh Sinha, Fei Fang, Bo An, Christopher Kiekintveld, and Milind Tambe.
Stackelberg security games: Looking beyond a decade of success.
In IJCAI, pages 5494–5501, 2018.

[Tewari and Murphy, 2017]
Ambuj Tewari and Susan A Murphy.
From ads to interventions: Contextual bandits in mobile health.
In Mobile Health, pages 495–517. Springer, 2017.

[Thompson, 1933]
William R Thompson.
On the likelihood that one unknown probability exceeds another in
view of the evidence of two samples.
Biometrika, 250 (3/4):0 285–294, 1933.

[Tibrewal et al., 2019]
Harshvardhan Tibrewal, Sravan Patchala, Manjesh K Hanawal, and Sumit J Darak.
Distributed learning and optimal assignment in multiplayer
heterogeneous networks.
In (to appear) IEEE International Conference on Computer
Communications (INFOCOM), 2019.

[Trapeznikov and Saligrama, 2013]
Kirill Trapeznikov and Venkatesh Saligrama.
Supervised sequential classification under budget constraints.
In Artificial Intelligence and Statistics, pages 581–589,

[Verma and Hanawal, 2018]
Arun Verma and Manjesh K Hanawal.
Unsupervised cost sensitive predictions with side information.
In Proceedings of the ACM India Joint International Conference
on Data Science and Management of Data, pages 364–367, 2018.

[Verma and Hanawal, 2020]
Arun Verma and Manjesh K Hanawal.
Stochastic network utility maximization with unknown utilities:
Multi-armed bandits approach.
In IEEE INFOCOM 2020-IEEE Conference on Computer
Communications, pages 189–198. IEEE, 2020.

[Verma et al., 2019]
Arun Verma, Manjesh Hanawal, Arun Rajkumar, and Raman Sankaran.
Censored semi-bandits: A framework for resource allocation with
censored feedback.
In Advances in Neural Information Processing Systems, pages
14499–14509, 2019a.

[Verma et al., 2019]
Arun Verma, Manjesh Hanawal, Csaba Szepesvari, and Venkatesh Saligrama.
Online algorithm for unsupervised sensor selection.
In Artificial Intelligence and Statistics, pages 3168–3176,

[Verma et al., 2019]
Arun Verma, Manjesh K Hanawal, and Rahul Vaze.
Distributed algorithms for efficient learning and coordination in ad
hoc networks.
In 2019 International Symposium on Modeling and Optimization in
Mobile, Ad Hoc, and Wireless Networks (WiOPT), pages 1–8. IEEE,

[Verma et al., 2020]
Arun Verma, Manjesh K Hanawal, and Nandyala Hemachandra.
Thompson sampling for unsupervised sequential selection.
In Asian Conference on Machine Learning, pages 545–560. PMLR,

[Verma et al., 2020]
Arun Verma, Manjesh K Hanawal, and Nandyala Hemachandra.
Unsupervised online feature selection for cost-sensitive medical
In 2020 International Conference on COMmunication Systems &
NETworkS (COMSNETS), pages 1–6. IEEE, 2020b.

[Verma et al., 2020]
Arun Verma, Manjesh Kumar Hanawal, Csaba Szepesvari, and Venkatesh Saligrama.
Online algorithm for unsupervised sequential selection with
contextual information.
In Advances in Neural Information Processing Systems,
volume 33, pages 778–788. Curran Associates, Inc., 2020c.

[Verma et al., 2021]
Arun Verma, Manjesh K Hanawal, Arun Rajkumar, and Raman Sankaran.
Censored semi-bandits for resource allocation.
arXiv preprint arXiv:2104.05781, 2021.

[Vershynin, 2012]
Roman Vershynin.
Introduction to the non-asymptotic analysis of random
matrices, page 210–268.
Cambridge University Press, 2012.

[Wang et al., 2015]
Joseph Wang, Kirill Trapeznikov, and Venkatesh Saligrama.
Efficient learning by directed acyclic graph for resource constrained
In Advances in Neural Information Processing Systems 28, pages
2152–2160, 2015.

[Wang and Chen, 2018]
Siwei Wang and Wei Chen.
Thompson sampling for combinatorial semi-bandits.
In International Conference on Machine Learning, pages
5101–5109, 2018.

[Weed et al., 2016]
Jonathan Weed, Vianney Perchet, and Philippe Rigollet.
Online learning in repeated auctions.
In Conference on Learning Theory, pages 1562–1583, 2016.

[Woodroofe, 1979]
Michael Woodroofe.
A one-armed bandit problem with a concomitant variable.
Journal of the American Statistical Association, 740
(368):0 799–806, 1979.

[Wu et al., 2015]
Yifan Wu, András György, and Csaba Szepesvári.
Online learning with gaussian payoffs and side observations.
In Advances in Neural Information Processing Systems, pages
1360–1368, 2015.

[Yi and Chiang, 2008]
Y. Yi and M. Chiang.
Stochastic network utility maximisation—a tribute to kelly's paper
published in this journal a decade ago.
European Transactions on Telecommunications, 190
(4):0 421–442, 2008.

[Zandi et al., 2016]
Marjan Zandi, Min Dong, and Ali Grami.
Distributed stochastic learning and adaptation to primary traffic for
dynamic spectrum access.
IEEE Transactions on Wireless Communications, 150 (3),

[Zhang et al., 2015]
Chao Zhang, Arunesh Sinha, and Milind Tambe.
Keeping pace with criminals: Designing patrol allocation against
adaptive opportunistic criminals.
In Proceedings of the 2015 international conference on
Autonomous agents and multiagent systems, pages 1351–1359, 2015.

[Zhang et al., 2016]
Chao Zhang, Victor Bucarey, Ayan Mukhopadhyay, Arunesh Sinha, Yundi Qian,
Yevgeniy Vorobeychik, and Milind Tambe.
Using abstractions to solve opportunistic crime security games at
In Proceedings of the 2016 International Conference on
Autonomous Agents & Multiagent Systems, pages 196–204, 2016a.

[Zhang et al., 2016]
Lijun Zhang, Tianbao Yang, Rong Jin, Yichi Xiao, and Zhi-hua Zhou.
Online stochastic linear optimization under one-bit feedback.
In International Conference on Machine Learning, pages
392–401, 2016b.

[Zolghadr et al., 2013]
Navid Zolghadr, Gábor Bartók, Russell Greiner, András György,
and Csaba Szepesvári.
Online learning with costly features and labels.
In Advances in Neural Information Processing Systems, pages
1241–1249, 2013.