_{\tau\rightarrow\infty}\frac{1}{\tau}\sum_{t=0}^{\tau-1}\sum_{i=1}^I\textbf{E}\{c(t)D_i^d(t)\} \\
&\mbox{s.t.}\,\,\text{stability of}~~ Q_{ij}(t)~~(\forall i\in \textbf{I},\forall j\in \textbf{J}_i), (1), (2), (3), (4), (5), (6) \notag\end{aligned}$$where the expectation above is with respect to the potential randomness of the control policy. The stability of these queues are defined in Definition 1.

\[def\_1\] A discrete queue evolving like $Q(t+1)=[Q(t)-S(t)]^+ +D(t)$ is *strongly stable* if: $$\lim_{t\rightarrow \infty}\sup \frac{1}{t}\sum_{\tau =0}^{t-1}\textbf{E}\{Q(\tau)\}<\infty.$$

If demand queues including the battery dynamics iteration (4) are *strongly stable*, the system is stable. According to Neely’s monograph [@Neely2], a *strongly stable* queue must be *rate stable* for finite service $S(t)$ and stochastic arrival $D(t)$. Furthermore, if and only if $d\leq s$ holds, $Q(t)$ is *rate stable*, where $d=\lim_{t\rightarrow \infty }\frac{1}{t}\sum_{\tau =0}^{t-1}\textbf{E}\{D(\tau)\}$ and $s=\lim_{t\rightarrow \infty }\frac{1}{t}\sum_{\tau =0}^{t-1}\textbf{E}\{S(\tau)\}$.

There is time coupling difficulty in solving P1 due to (4), that is to say the current charging and discharging decisions will affect the energy level in battery in the future. To tackle this difficulty, a relaxed problem will be studied: $$\begin{aligned}
\widetilde\textbf{P1}\textbf{:}~&\min~~
\lim_{\tau\rightarrow\infty}\frac{1}{\tau}\sum_{t=0}^{\tau-1}\!\sum_{i=1}^I\!\textbf{E}\{c(t)D_i^d(t)\} \\
\mbox{s.t.}\,\,&\text{stability of}~Q_{ij}(t)~(\forall i\in \textbf{I},\forall j\in \textbf{J}_i), (1), (2), (3), (6),\\
&\lim_{\tau \rightarrow \infty }\frac{1}{\tau }\sum\limits_{t=0}^{\tau -1}\!\mathbf{E}\!\left\{ \eta ^{-}\!\left( \!\left(\!\sum\limits_{j=1}^{J_{i}}r_{ij}\left( t\right)\!-\!U_{i}\left( t\right)\!\right)^{+}\!-\!D_{i}^{d}\left( t\right)\!\right)\!\right\}\!\geq \!\lim_{\tau \rightarrow\infty}
\frac{1}{\tau}\!\sum\limits_{t=0}^{\tau-1}\!\mathbf{E}\!\left\{ \eta^{+}R_{i}\left(t\right)\!\right\}.\end{aligned}$$

Denote the optimal value of **P1** as $P^*$ and $\widetilde \textbf{P1}$ as $P_{re}^*$. Notice 