gradient of $J$ in  with respect to $\mtvec\left(\mtbfH\right)$ to zero gives the closed-form solution for the projection step as follows $$\mtvec\left(\mhbfH\right)=\left[\mtbfF_{1}+\mtbfL_{\otimes}\left(\mu_{1},\zeta_{1}\right)\right]^{-1}\left[\mtbfF_{2}\mtvec\left(\mtbfR\right)+\gamma\mtbfF_{3}\mtvec\left(\mtbfW\right)\right],\label{eq:closedFormSolution_4_projectionStep}$$ where $\mtbfL_{\otimes}\left(\mu_{1},\zeta_{1}\right)=\left(\mu_{1}\mtbfL^{\intercal}+\zeta_{1}\mtbfI_{N}\right)\otimes\mtbfI_{u}\in\mtbbR^{uN\times uN}$.

Updating Rules for the Fixed Point step 
----------------------------------------

Considering the 1st term in the fixed point step  gives

$$\begin{aligned}
O_{1} & =\sum_{n}\left\Vert \Phi_{n}\mtbfw_{n}-\Phi_{n}\mhbfh_{n}\right\Vert _{2}^{2}=\left\Vert \Phi_{\otimes}\mtvec\left(\mtbfW\right)-\Phi_{\otimes}\mtvec\left(\mhbfH\right)\right\Vert _{2}^{2}\end{aligned}$$

where $\Phi_{\otimes}=\left(\sum_{n=1}^{N}\mtbfE_{n}\otimes\Phi_{n}\right)$. To facilitate the optimization, we design $\left\{ \Phi_{n}\right\} _{n=1}^{N}$ to let $\Phi_{\otimes}=\mtbfF_{1}+\mtbfL_{\otimes}\left(\mu_{1},\zeta_{1}\right)$, which leads to $$\begin{aligned}
O_{1} & =\left\Vert \Phi_{\otimes}\mtvec\left(\mtbfW\right)-\Phi_{\otimes}\mtvec\left(\mhbfH\right)\right\Vert _{2}^{2}\\
 & =\left\Vert \Phi_{\otimes}\mtvec\left(\mtbfW\right)-\left[\mtbfF_{2}\mtvec\left(\mtbfR\right)+\gamma\mtbfF_{3}\mtvec\left(\mtbfW\right)\right]\right\Vert \\
 & =\left\Vert \left(\Phi_{\otimes}-\gamma\mtbfF_{3}\right)\mtvec\left(\mtbfW\right)-\mtbfF_{2}\mtvec\left(\mtbfR\right)\right\Vert _{2}^{2},\end{aligned}$$ and finally results in an easy solution for the critic updating  (cf. Theorem \[thm:B\_sysmmetirc\_positive\_definite\]). Letting $\mtbfP=\Phi_{\otimes}-\gamma\mtbfF_{3}$, we have $$O_{1}=\left\Vert \mtbfP\mtvec\left(\mtbfW\right)-\mtbfF_{2}\mtvec\left(\mtbfR\right)\right\Vert _{2}^{2}.$$ The partial derivative of $O_{1}$ with respect to $\mtvec\left(\mtbfW\right)$ is $$\frac{\partial O_{1}}{\partial\mtvec\left(\mtbfW\right)}=2\mtbfP^{\intercal}\mtbfP\mtvec\left(\mtbfW\right)-2\mtbfP^{\intercal}\mtbfF_{2}\mtvec\left(\mtbfR\right).$$ Considering the partial derivative of the cohesion constraint and the Frobenius norm based smooth constraint with respect to $\mtvec\left(\mtbfH\right)$, and setting the overll partial derivative to zero, i.e. $\partial O/\partial\mtvec\left(\mtbfW\right)=\mtbfzero$, we can obtain the following closed-form solution $$\mtvec\left(\mtbfW^{*}\right)=\left[\mtbfP^{\intercal}\mtbfP+\mtbfL_{\otimes}\left(\mu_{2},\zeta_{2}\right)\right]^{-1}\mtbfP^{\intercal}\mtbfF_{2}\mtvec\left(\mtbfR\right),\label{eq:CriticUpdatingRule_=00003D0000231_GraphRL}$$ where $\mtbfL_{\otimes}\left(\mu_{2},\zeta_{2}\right)=\left(\mu_{2}\mtbfL^{\intercal}+\zeta_{2}\mtbfI_{N}\right)\otimes\mtbfI_{u}$.

\[thm:B\_sysmmetirc\_positive\_definite\] $\mtbfB=\mtbfP^{\intercal}\mtbfP+\mtbfL_{\otimes}\left(\mu_{2},\zeta_{2}\right)$ is a symmetric and positive definite matrix, which leads to an easy critic updating rule in .

\[lem:KroneckerProduct\_eigenvalues\] Suppose that $\mtbfA\in\mtbbR^{n\times n}$ and $\mtbfB\in\mtbbR^{m\times m}$ are square matrices. Let $\lambda_{1},\cdots,\lambda_{n}$ be the eigenvalues of $\mtbfA$ and $\nu_{1},\cdots,\nu_{m}$ be those of $\mtbfB$. Then the eigenvalues 