$T_{\eta}\in\mathbb{N}$ such that $\left|\eta^{t}D_{i}^{t}\right|\leq\eta$ for
every $t\geq T_{\eta}$ and for every $i\in\\{H,L\\}$. Therefore, it holds that

$\left|p_{i}^{t+1}-p_{i}^{t}\right|=\left|\operatorname{Proj}_{\mathcal{P}}\left(p_{i}^{t}+\eta^{t}D_{i}^{t}\right)-p_{i}^{t}\right|\\\
\leq\left|\eta^{t}D_{i}^{t}\right|\leq\eta,\quad\forall t\geq T_{\eta},$ (77)

where the first inequality is due to the property of the projection operator.
Then, by the reference price update, we have for $t\geq T_{\eta}$ and for
$i\in\\{H,L\\}$ that

$\displaystyle\left|r_{i}^{t+1}-p_{i}^{t+1}\right|$
$\displaystyle=\left|\alpha r_{i}^{t}+(1-\alpha)p_{i}^{t}-p_{i}^{t+1}\right|$
(78)
$\displaystyle=\left|\alpha\left(r_{i}^{t}-p_{i}^{t}\right)+\left(p_{i}^{t+1}-p_{i}^{t}\right)\right|$
$\displaystyle\leq\alpha\left|r_{i}^{t}-p_{i}^{t}\right|+\left|p_{i}^{t+1}-p_{i}^{t}\right|$
$\displaystyle\leq\alpha\left|r_{i}^{t}-p_{i}^{t}\right|+\eta,$

where the last line follows from the upper bound in Eq. (77). Applying Eq.
(78) recursively from $t$ to $T_{\eta}$, we further derive that

$\displaystyle\Big{|}r_{i}^{t+1}-p_{i}^{t+1}\Big{|}$
$\displaystyle\leq\alpha^{t+1-T_{\eta}}\cdot\Big{|}r_{i}^{T_{\eta}}-p_{i}^{T_{\eta}}\Big{|}+\eta\sum_{\tau=T_{\eta}}^{t}\alpha^{\tau-
T_{\eta}}$ (79)
$\displaystyle\leq\alpha^{t+1-T_{\eta}}\cdot(\overline{p}-\underline{p})+\dfrac{\eta}{1-\alpha},\quad\forall
i\in\\{H,L\\}.$

Since $\eta$ can be arbitrarily close to $0$, we have that
$\left|r_{i}^{t}-p_{i}^{t}\right|\rightarrow 0$ as $t\rightarrow\infty$, which
completes the proof of the convergence. $\Box$

###### Lemma E.2

Define the function $\mathcal{G}(\mathbf{p})$ as

$\mathcal{G}(\mathbf{p}):=\operatorname{sign}(p_{H}^{\star\star}-p_{H})\cdot
G_{H}(\mathbf{p},\mathbf{p})+\operatorname{sign}(p_{L}^{\star\star}-p_{L})\cdot
G_{L}(\mathbf{p},\mathbf{p}),$ (80)

where $G_{i}(\mathbf{p},\mathbf{r})$ is the scaled partial derivative
introduced in Eq. (25). Then, it always holds that
$\mathcal{G}(\mathbf{p})>0,\forall\mathbf{p}\in\mathcal{P}^{2}\backslash\\{\mathbf{p}^{\star\star}\\}$,
where
$\mathbf{p}^{\star\star}=\left(p_{H}^{\star\star},p_{L}^{\star\star}\right)$
denotes the unique SNE, and the function $\operatorname{sign}(\cdot)$ is
defined in Eq. (27).

Furthermore, for every $\epsilon>0$, there exists $M_{\epsilon}>0$ such that
$\mathcal{G}(\mathbf{p})\geq M_{\epsilon}$ if
$\varepsilon(\mathbf{p})\geq\epsilon$, where $\varepsilon(\mathbf{p})$ is the
weighted $\ell_{1}$-distance function defined in Eq. (24).

Proof. Firstly, by the first-order condition at the SNE (see Eq. (16)), we
have
$G_{H}(\mathbf{p}^{\star\star},\mathbf{p}^{\star\star})=G_{L}(\mathbf{p}^{\star\star},\mathbf{p}^{\star\star})=0$,
which also implies $\mathcal{G}(\mathbf{p}^{\star\star})=0$. Then, we recall
the definition of four regions $N_{1}$, $N_{2}$, $N_{3}$, and $N_{4}$ in Eq.
(28). We show that $\mathcal{G}(\mathbf{p})>0$ when $\mathbf{p}$ belongs to
either one of the four regions.

  1. 1.

When $\mathbf{p}\in N_{1}$, i.e., $\ p_{H}>p^{\star\star}_{H}$ and $p_{L}\geq
p_{L}^{\star\star}$, the function $\mathcal{G}(\mathbf{p})$ becomes

$\displaystyle\mathcal{G}(\mathbf{p})$
$\displaystyle=-G_{H}(\mathbf{p},\mathbf{p})-G_{L}(\mathbf{p},\mathbf{p})$
(81)
$\displaystyle=-\dfrac{1}{(b_{H}+c_{H})p_{H}}-\dfrac{1}{(b_{L}+c_{L})p_{L}}-\big{(}d_{H}(\mathbf{p},\mathbf{p})+d_{L}(\mathbf{p},\mathbf{p})\big{)}+2$
$\displaystyle=-\dfrac{1}{(b_{H}+c_{H})p_{H}}-\dfrac{1}{(b_{L}+c_{L})p_{L}}+d_{0}(\mathbf{p},\mathbf{p})+1$
$\displaystyle=-\dfrac{1}{(b_{H}+c_{H})p_{H}}-\dfrac{1}{(b_{L}+c_{L})p_{L}}+\dfrac{1}{1+\exp(a_{H}-b_{H}p_{H})+\exp(a_{L}-b_{L}p_{L})}+1,$

where
$d_{0}(\mathbf{p},\mathbf{r})=1-d_{H}(\mathbf{p},\mathbf{r})-d_{L}(\mathbf{p},\mathbf{r})$
denotes the no purchase probability. We observe from Eq. (81) that
$\mathcal{G}(\mathbf{p})$ is strictly increasing in $p_{H}$ and $p_{L}$.
Together with the fact that $p_{H}$ and $p_{L}$ are lowered bounded by
$p_{H}^{\star\star}$ and $p_{L}^{\star\star}$, respectively, and
$\mathcal{G}(\mathbf{p}^{\star\star})=0$, we verify that
$\mathcal{G}(\mathbf{p})>0$ when $\mathbf{p}\in N_{1}$. With the similar
approach, we show that when $\mathbf{p}\in N_{3}$, i.e.,
$p_{H}<p^{\star\star}_{H}$ and $p_{L}\leq p_{L}^{\star\star}$, it also follows
that $\mathcal{G}(\mathbf{p})>0$.

  2. 2.

When $\mathbf{p}\in N_{2}$, i.e., $p_{H}\leq p_{H}^{\star\star}$ and
$p_{L}>p_{L}^{\star\star}$, the function $\mathcal{G}(\mathbf{p})$ becomes

$\displaystyle\mathcal{G}(\mathbf{p})$
$\displaystyle=G_{H}(\mathbf{p},\mathbf{p})-G_{L}(\mathbf{p},\mathbf{p})$ (82)
$\displaystyle=\dfrac{1}{(b_{H}+c_{H})p_{H}}-\dfrac{1}{(b_{L}+c_{L})p_{L}}+d_{H}(\mathbf{p},\mathbf{p})-d_{L}(\mathbf{p},\mathbf{p})$
$\displaystyle=\dfrac{1}{(b_{H}+c_{H})p_{H}}-\dfrac{1}{(b_{L}+c_{L})p_{L}}+\dfrac{\exp(a_{H}-b_{H}p_{H})-\exp(a_{L}-b_{L}p_{L})}{1+\exp(a_{H}-b_{H}p_{H})+\exp(a_{L}-b_{L}p_{L})}.$

By Eq. (82), we notice that $\mathcal{G}(\mathbf{p})$ under region $N_{2}$ is
strictly decreasing in $p_{H}$ and strictly increasing in $p_{L}$. Meanwhile,
since $p_{H}\leq p_{H}^{\star\star}$, $p_{L}>p_{L}^{\star\star}$, and
$\mathcal{G}(\mathbf{p}^{\star\star})=0$, it implies that
$\mathcal{G}(\mathbf{p})>0$ when $\mathbf{p}\in N_{2}$. Moreover, by similar
reasoning, we show that when $\mathbf{p}\in N_{4}$, i.e., $p_{H}\geq
p_{H}^{\star\star}$ and $p_{L}<p_{L}^{\star\star}$, the inequality
$\mathcal{G}(\mathbf{p})>0$ also holds.

Finally, we are left to establish the existence of $M_{\epsilon}$. It suffices
to show that

$\min_{\varepsilon(\mathbf{p})=\epsilon_{1}}\mathcal{G}(\mathbf{p})>\min_{\varepsilon(\mathbf{p})=\epsilon_{2}}\mathcal{G}(\mathbf{p}).$
(83)

for every $\epsilon_{1}>\epsilon_{2}>0$. Suppose
$\mathbf{p}^{\epsilon_{1}}:=\mathop{\rm
arg\,min}_{\varepsilon(\mathbf{p})=\epsilon_{1}}\mathcal{G}(\mathbf{p})$.
Define
$\mathbf{p}^{\epsilon_{2}}:=(\epsilon_{2}/\epsilon_{1})\mathbf{p}^{\epsilon_{1}}+(1-\epsilon_{2}/\epsilon_{1})\mathbf{p}^{\star\star}$,
which satisfies that $\varepsilon(\mathbf{p}^{\epsilon_{2}})=\epsilon_{2}$.
Then, we have that

$\displaystyle\mathcal{G}(\mathbf{p}^{\epsilon_{2}})$ $\displaystyle\ \
=\operatorname{sign}(p_{H}^{\star\star}-p_{H}^{\epsilon_{2}})\cdot
G_{H}(\mathbf{p}^{\epsilon_{2}},\mathbf{p}^{\epsilon_{2}})+\operatorname{sign}(p_{L}^{\star\star}-p_{L}^{\epsilon_{2}})\cdot
G_{L}(\mathbf{p}^{\epsilon_{2}},\mathbf{p}^{\epsilon_{2}})$ (84)
$\displaystyle\overset{(\Delta_{1})}{=}\operatorname{sign}\Big{(}\dfrac{\epsilon_{2}}{\epsilon_{1}}(p_{H}^{\star\star}-p_{H}^{\epsilon_{1}})\Big{)}\cdot
G_{H}(\mathbf{p}^{\epsilon_{2}},\mathbf{p}^{\epsilon_{2}})+\operatorname{sign}\Big{(}\dfrac{\epsilon_{2}}{\epsilon_{1}}(p_{L}^{\star\star}-p_{L}^{\epsilon_{1}})\Big{)}\cdot
G_{L}(\mathbf{p}^{\epsilon_{2}},\mathbf{p}^{\epsilon_{2}})$ $\displaystyle\ \
=\operatorname{sign}(p_{H}^{\star\star}-p_{H}^{\epsilon_{1}})\cdot
G_{H}(\mathbf{p}^{\epsilon_{2}},\mathbf{p}^{\epsilon_{2}})+\operatorname{sign}(p_{L}^{\star\star}-p_{L}^{\epsilon_{1}})\cdot
G_{L}(\mathbf{p}^{\epsilon_{2}},\mathbf{p}^{\epsilon_{2}})$
$\displaystyle\overset{(\Delta_{2})}{\leq}\operatorname{sign}(p_{H}^{\star\star}-p_{H}^{\epsilon_{1}})\cdot
G_{H}(\mathbf{p}^{\epsilon_{1}},\mathbf{p}^{\epsilon_{1}})+\operatorname{sign}(p_{L}^{\star\star}-p_{L}^{\epsilon_{1}})\cdot
G_{L}(\mathbf{p}^{\epsilon_{1}},\mathbf{p}^{\epsilon_{1}})=\mathcal{G}(\mathbf{p}^{\epsilon_{1}}),$

where $(\Delta_{1})$ follows from substituting $p_{i}^{\epsilon_{2}}$ in
$\operatorname{sign}(p_{i}^{\star\star}-p_{i}^{\epsilon_{2}})$ with
$p_{i}^{\epsilon_{2}}=(\epsilon_{2}/\epsilon_{1})p_{i}^{\epsilon_{1}}+(1-\epsilon_{2}/\epsilon_{1})p_{i}^{\star\star}$.
To see why $(\Delta_{2})$ holds, recall that we have shown in Eq. (81) and Eq.
(82) that when two prices are in the same region (see four regions defined in
Eq. (28)), the price closer to $\mathbf{p}^{\star\star}$ in terms of the
metric $\varepsilon(\cdot)$ has a greater value of $\mathcal{G}(\mathbf{p})$.
Since $\mathbf{p}^{\epsilon_{1}}$ and $\mathbf{p}^{\epsilon_{2}}$ are from the
same region and $\epsilon_{1}>\epsilon_{2}$, we conclude that
$\min_{\varepsilon(\mathbf{p})=\epsilon_{1}}\mathcal{G}(\mathbf{p})=\mathcal{G}(\mathbf{p}^{\epsilon_{1}})>\mathcal{G}(\mathbf{p}^{\epsilon_{2}})\geq\min_{\varepsilon(\mathbf{p})=\epsilon_{2}}\mathcal{G}(\mathbf{p})$.
$\Box$

###### Lemma E.3

Define function $\mathcal{H}(\mathbf{p})$ as follows

$\mathcal{H}(\mathbf{p}):=(b_{H}+c_{H})\cdot
G_{H}(\mathbf{p},\mathbf{p})\cdot(p_{H}^{\star\star}-p_{H})+(b_{L}+c_{L})\cdot
G_{L}(\mathbf{p},\mathbf{p})\cdot(p_{L}^{\star\star}-p_{L})$ (85)

Then, there exist $\gamma>0$ and a open set
$U_{\gamma}\ni\mathbf{p}^{\star\star}$ such that

$\mathcal{H}(\mathbf{p})\geq\gamma\cdot\|\mathbf{p}-\mathbf{p}^{\star\star}\|_{2}^{2},\quad\forall\mathbf{p}\in
U_{\gamma}.$ (86)

Proof. According to the partial derivatives in Eq. (92a) and Eq. (92b) from
Lemma E.5, we have

$\displaystyle\dfrac{\partial G_{i}(\mathbf{p},\mathbf{p})}{\partial p_{i}}$
$\displaystyle=-\dfrac{1}{(b_{i}+c_{i})p_{i}^{2}}-b_{i}\cdot
d_{i}(\mathbf{p},\mathbf{p})\cdot\big{(}1-d_{i}(\mathbf{p},\mathbf{p})\big{)};$
(87) $\displaystyle\dfrac{\partial G_{i}(\mathbf{p},\mathbf{p})}{\partial
p_{-i}}$ $\displaystyle=b_{-i}\cdot d_{i}(\mathbf{p},\mathbf{p})\cdot
d_{-i}(\mathbf{p},\mathbf{p}).$

Then, to compute the gradient
$\nabla\mathcal{H}(\mathbf{p})=\left[\partial\mathcal{H}(\mathbf{p})/\partial
p_{H},\partial\mathcal{H}(\mathbf{p})/\partial p_{L}\right]$, we utilize
partial derivatives of $G_{i}(\mathbf{p},\mathbf{p})$ in Eq. (87) and obtain
the partial derivatives of $\mathcal{H}(\mathbf{p})$ for $i\in\\{H,L\\}$:

$\displaystyle\dfrac{\partial\mathcal{H}(\mathbf{p})}{\partial p_{i}}=$
$\displaystyle(b_{i}+c_{i})\left[\dfrac{\partial
G_{i}(\mathbf{p},\mathbf{p})}{\partial
p_{i}}(p_{i}^{\star\star}-p_{i})-G_{i}(\mathbf{p},\mathbf{p})\right]+(b_{-i}+c_{-i})(p_{-i}^{\star\star}-p_{-i})\dfrac{\partial
G_{-i}(\mathbf{p},\mathbf{p})}{\partial p_{i}}$ (88) $\displaystyle=$
$\displaystyle-\dfrac{p_{i}^{\star\star}}{p_{i}^{2}}-(b_{i}+c_{i})\big{(}d_{i}(\mathbf{p},\mathbf{p})-1\big{)}-b_{i}(b_{i}+c_{i})\cdot
d_{i}(\mathbf{p},\mathbf{p})\cdot\big{(}1-d_{i}(\mathbf{p},\mathbf{p})\big{)}(p_{i}^{\star\star}-p_{i})$
$\displaystyle+b_{i}(b_{-i}+c_{-i})\cdot d_{i}(\mathbf{p},\mathbf{p})\cdot
d_{-i}(\mathbf{p},\mathbf{p})\cdot(p_{-i}^{\star\star}-p_{-i}).$

From the definition of $G_{i}(\cdot,\cdot)$ in Eq. (31) and the system
equations of first-order condition in Eq. (16), we have
$G_{i}(\mathbf{p}^{\star\star},\mathbf{p}^{\star\star})=1/\big{[}(b_{i}+c_{i})p_{i}^{\star\star}\big{]}+(d_{i}^{\star\star}-1)=0$,
where
$d_{i}^{\star\star}:=d_{i}(\mathbf{p}^{\star\star},\mathbf{p}^{\star\star})$
denotes the market share of product $i$ at the SNE. Thereby, it follows that
$\nabla\mathcal{H}(\mathbf{p}^{\star\star})=0$.

Next, the Hessian matrix $\nabla^{2}\mathcal{H}(\mathbf{p})$ evaluated at
$\mathbf{p}^{\star\star}$ can be computed as

$\displaystyle\quad\ \ \nabla^{2}\mathcal{H}(\mathbf{p}^{\star\star})$
$\displaystyle=\left[\begin{array}[]{ll}\dfrac{\partial^{2}\mathcal{H}(\mathbf{p}^{\star\star})}{{\partial{p_{H}}^{2}}},&\dfrac{\partial^{2}\mathcal{H}(\mathbf{p}^{\star\star})}{{\partial{p_{H}}\partial
p_{L}}}\vspace{8pt}\\\
\dfrac{\partial^{2}\mathcal{H}(\mathbf{p}^{\star\star})}{\partial
p_{L}\partial
p_{H}},&\dfrac{\partial^{2}\mathcal{H}(\mathbf{p}^{\star\star})}{\partial{p_{L}}^{2}}\end{array}\right]$
$\displaystyle=\left[\begin{array}[]{ll}\dfrac{2p_{H}^{\star\star}}{(p_{H}^{\star\star})^{3}}+2b_{H}(b_{H}+c_{H})\cdot
d_{H}^{\star\star}\big{(}1-d_{H}^{\star\star}\big{)},&-\big{[}b_{H}(b_{L}+c_{L})+b_{L}(b_{H}+c_{H})\big{]}\cdot
d_{H}^{\star\star}d_{L}^{\star\star}\vspace{8pt}\\\
-\big{[}b_{H}(b_{L}+c_{L})+b_{L}(b_{H}+c_{H})\big{]}\cdot
d_{H}^{\star\star}d_{L}^{\star\star},&\dfrac{2p_{L}^{\star\star}}{(p_{L}^{\star\star})^{3}}+2b_{L}(b_{L}+c_{L})\cdot
d_{L}^{\star\star}\big{(}1-d_{L}^{\star\star}\big{)}\end{array}\right]$
$\displaystyle=\left[\begin{array}[]{ll}2(b_{H}+c_{H})\cdot\big{(}1-d_{H}^{\star\star}\big{)}\cdot\big{[}(b_{H}+c_{H})-c_{H}d_{H}^{\star\star}\big{]},&-\big{[}b_{H}(b_{L}+c_{L})+b_{L}(b_{H}+c_{H})\big{]}\cdot
d_{H}^{\star\star}d_{L}^{\star\star}\vspace{8pt}\\\
-\big{[}b_{H}(b_{L}+c_{L})+b_{L}(b_{H}+c_{H})\big{]}\cdot
d_{H}^{\star\star}d_{L}^{\star\star},&2(b_{L}+c_{L})\cdot\big{(}1-d_{L}^{\star\star}\big{)}\cdot\big{[}(b_{L}+c_{L})-c_{L}d_{L}^{\star\star}\big{]}\end{array}\right].$

Note that the last equality results again from substituting in the identity
$G_{i}(\mathbf{p}^{\star\star},\mathbf{p}^{\star\star})=1/\big{[}(b_{i}+c_{i})\cdot
p_{i}^{\star\star}\big{]}+(d_{i}^{\star\star}-1)=0$.

The diagonal entries of $\nabla^{2}\mathcal{H}(\mathbf{p}^{\star\star})$ are
clearly positive. For $i\in\\{H,L\\}$, define
$k_{i}:=b_{i}/(b_{i}+c_{i})\in[0,1]$. Then, the determinant of
$\nabla^{2}\mathcal{H}(\mathbf{p}^{\star\star})$ can be computed as follows:

$\displaystyle\operatorname{det}\big{(}\nabla^{2}\mathcal{H}(\mathbf{p}^{\star\star})\big{)}$
(89) $\displaystyle=$ $\displaystyle
4(b_{H}+c_{H})(b_{L}+c_{L})\cdot\big{(}1-d_{H}^{\star\star}\big{)}\big{(}1-d_{L}^{\star\star}\big{)}\cdot\big{[}(b_{H}+c_{H})-c_{H}d_{H}^{\star\star}\big{]}\big{[}(b_{L}+c_{L})-c_{L}d_{L}^{\star\star}\big{]}$
$\displaystyle-\Big{(}\big{[}b_{H}(b_{L}+c_{L})+b_{L}(b_{H}+c_{H})\big{]}\cdot
d_{H}^{\star\star}d_{L}^{\star\star}\Big{)}^{2}$ $\displaystyle=$
$\displaystyle(b_{H}+c_{H})^{2}(b_{L}+c_{L})^{2}\cdot\Big{(}4\big{(}1-d_{H}^{\star\star}\big{)}\big{(}1-d_{L}^{\star\star}\big{)}\cdot\big{[}1-(1-k_{H})d_{H}^{\star\star}\big{]}\big{[}1-(1-k_{L})d_{L}^{\star\star}\big{]}$
$\displaystyle\qquad\qquad\qquad\qquad\qquad\
-(k_{H}+k_{L})^{2}\big{(}d_{H}^{\star\star}d_{L}^{\star\star}\big{)}^{2}\Big{)}$
$\displaystyle\overset{(\Delta_{1})}{\geq}$ $\displaystyle
4(b_{H}+c_{H})^{2}(b_{L}+c_{L})^{2}\cdot\Big{(}\big{(}1-d_{H}^{\star\star}\big{)}\big{(}1-d_{L}^{\star\star}\big{)}\cdot\big{[}1-(1-k_{H})d_{H}^{\star\star}\big{]}\big{[}1-(1-k_{L})d_{L}^{\star\star}\big{]}-\big{(}d_{H}^{\star\star}d_{L}^{\star\star}\big{)}^{2}\Big{)}$
$\displaystyle\overset{(\Delta_{2})}{\geq}$ $\displaystyle
4(b_{H}+c_{H})^{2}(b_{L}+c_{L})^{2}\cdot
d_{H}^{\star\star}d_{L}^{\star\star}\Big{(}\big{[}1-(1-k_{H})d_{H}^{\star\star}\big{]}\big{[}1-(1-k_{L})\cdot
d_{L}^{\star\star}\big{]}-d_{H}^{\star\star}d_{L}^{\star\star}\Big{)}$
$\displaystyle\overset{(\Delta_{3})}{\geq}$ $\displaystyle
4(b_{H}+c_{H})^{2}(b_{L}+c_{L})^{2}\cdot
d_{H}^{\star\star}d_{L}^{\star\star}\Big{[}\big{(}1-d_{H}^{\star\star}\big{)}\big{(}1-d_{L}^{\star\star}\big{)}-d_{H}^{\star\star}d_{L}^{\star\star}\Big{]}$
$\displaystyle=$ $\displaystyle 4(b_{H}+c_{H})^{2}(b_{L}+c_{L})^{2}\cdot
d_{H}^{\star\star}d_{L}^{\star\star}\big{(}1-d_{H}^{\star\star}-d_{L}^{\star\star}\big{)}$
$\displaystyle\overset{(\Delta_{4})}{>}$ $\displaystyle 0,$

where inequalities respectively result from the following facts
$(\Delta_{1})$: $k_{H}+k_{L}\leq 2$; $(\Delta_{2})$:
$1-d_{i}^{\star\star}>d_{-i}^{\star\star}$ for $i\in\\{H,L\\}$;
$(\Delta_{3})$: $1-k_{i}\leq 1$ for $i\in\\{H,L\\}$; $(\Delta_{4})$:
$d_{H}^{\star\star}+d_{L}^{\star\star}<1$. Thus, we conclude that
$\nabla^{2}\mathcal{H}(\mathbf{p}^{\star\star})$ is positive definite.

By the continuity of $\nabla^{2}\mathcal{H}(\mathbf{p})$, there exists some
constant $\gamma>0$ and a open set $U_{\gamma}\ni\mathbf{p}^{\star\star}$ such
that $\nabla^{2}\mathcal{H}(\mathbf{p})\succeq 2\gamma I_{2}$,
$\forall\mathbf{p}\in U_{\gamma}$, where $I_{2}$ is the $2\times 2$ identity
matrix. Using the second-order Taylor expansion at $\mathbf{p}^{\star\star}$,
for all $\mathbf{p}\in U_{\gamma}$, there exists $\widetilde{\mathbf{p}}\in
U_{\gamma}$ such that

$\displaystyle\mathcal{H}(\mathbf{p})$
$\displaystyle=\mathcal{H}(\mathbf{p}^{\star\star})+\nabla\mathcal{H}(\mathbf{p}^{\star\star})\cdot\big{(}\mathbf{p}-\mathbf{p}^{\star\star}\big{)}+\dfrac{1}{2}\big{(}\mathbf{p}-\mathbf{p}^{\star\star}\big{)}^{\top}\cdot\nabla^{2}\mathcal{H}(\widetilde{\mathbf{p}})\cdot\big{(}\mathbf{p}-\mathbf{p}^{\star\star}\big{)}$
(90)
$\displaystyle=\dfrac{1}{2}\big{(}\mathbf{p}-\mathbf{p}^{\star\star}\big{)}^{\top}\cdot\nabla^{2}\mathcal{H}(\widetilde{\mathbf{p}})\cdot\big{(}\mathbf{p}-\mathbf{p}^{\star\star}\big{)}$
$\displaystyle\geq\dfrac{1}{2}\big{(}\mathbf{p}-\mathbf{p}^{\star\star}\big{)}^{\top}\cdot
2\gamma I_{2}\cdot\big{(}\mathbf{p}-\mathbf{p}^{\star\star}\big{)}$
$\displaystyle=\gamma\|\mathbf{p}-\mathbf{p}^{\star\star}\|_{2}^{2},$

where the second equality arises from that
$\mathcal{H}(\mathbf{p}^{\star\star})=0$ and
$\nabla\mathcal{H}(\mathbf{p}^{\star\star})=0$. $\Box$

###### Lemma E.4

For any product $i\in\\{H,L\\}$, let

$\widehat{G}_{i}(\mathbf{p}):=\operatorname{sign}\big{(}p_{i}^{\star\star}-p_{i}\big{)}\cdot
G_{i}\big{(}\mathbf{p},\mathbf{p}\big{)},$ (91)

where $G_{i}(\mathbf{p},\mathbf{r})$ is the scaled partial derivative defined
in Eq. (25). Then, $\widehat{G}_{i}(\mathbf{p})$ is always increasing as
$|p_{i}^{\star\star}-p_{i}|$ increases, and

  1. 1.

when $\mathbf{p}\in{N_{1}}\cup N_{3}$ (see the definition in Eq. (28)),
$\widehat{G}_{i}(\mathbf{p})$ is decreasing as $|p_{-i}^{\star\star}-p_{-i}|$
increases;

  2. 2.

when $\mathbf{p}\in{N_{2}}\cup N_{4}$, $\widehat{G}_{i}(\mathbf{p})$ is
increasing as $|p_{-i}^{\star\star}-p_{-i}|$ increases.

Proof. Without loss of generality, consider the case that product $i=H$ and
product $-i=L$.

Apparently, we have
$\operatorname{sign}\big{(}p_{H}^{\star\star}-p_{H}\big{)}\leq 0$ in
$N_{1}\cup N_{4}$, and
$\operatorname{sign}\big{(}p_{H}^{\star\star}-p_{H}\big{)}\geq 0$ in
$N_{2}\cup N_{3}$ by definitions (see Eq. (28)). On the other hand, by Eq.
(92b) in Lemma E.5, it holds that $\partial
G_{H}\big{(}\mathbf{p},\mathbf{p}\big{)}/\partial p_{H}<0$ and $\partial
G_{H}\big{(}\mathbf{p},\mathbf{p}\big{)}/\partial p_{L}>0$,
$\forall\mathbf{p}\in\mathcal{P}^{2}$. Thus,

  1. 1.

in $N_{1}\cup N_{4}$, $G_{H}\big{(}\mathbf{p},\mathbf{p}\big{)}$ is decreasing
as $|p_{H}^{\star\star}-p_{H}|$ increases; in $N_{2}\cup N_{3}$,
$G_{H}\big{(}\mathbf{p},\mathbf{p}\big{)}$ is increasing as
$|p_{H}^{\star\star}-p_{H}|$ increases;

  2. 2.

in $N_{1}\cup N_{2}$, $G_{H}\big{(}\mathbf{p},\mathbf{p}\big{)}$ is increasing
as $|p_{L}^{\star\star}-p_{L}|$ increases; conversely,
$G_{H}\big{(}\mathbf{p},\mathbf{p}\big{)}$ is decreasing as
$|p_{L}^{\star\star}-p_{L}|$ increases in $N_{3}\cup N_{4}$.

The final results directly follows by combining the above pieces together.
$\Box$

###### Lemma E.5

Let $G_{i}(\mathbf{p},\mathbf{r})$ be the scaled partial derivative defined in
Eq. (25), then partial derivatives of $G_{i}(\mathbf{p},\mathbf{r})$ with
respect to $\mathbf{p}$ and $\mathbf{r}$ are given as

$\displaystyle\dfrac{\partial G_{i}(\mathbf{p},\mathbf{r})}{\partial
p_{i}}=-\dfrac{1}{(b_{i}+c_{i})p_{i}^{2}}-(b_{i}+c_{i})\cdot
d_{i}(\mathbf{p},\mathbf{r})\cdot\big{(}1-d_{i}(\mathbf{p},\mathbf{r})\big{)};$
(92a) $\displaystyle\dfrac{\partial G_{i}(\mathbf{p},\mathbf{r})}{\partial
p_{-i}}=(b_{-i}+c_{-i})\cdot d_{i}(\mathbf{p},\mathbf{r})\cdot
d_{-i}(\mathbf{p},\mathbf{r});$ (92b) $\displaystyle\dfrac{\partial
G_{i}(\mathbf{p},\mathbf{r})}{\partial r_{i}}=c_{i}\cdot
d_{i}(\mathbf{p},\mathbf{r})\cdot\big{(}1-d_{i}(\mathbf{p},\mathbf{r})\big{)};$
(92c) $\displaystyle\dfrac{\partial G_{i}(\mathbf{p},\mathbf{r})}{\partial
r_{-i}}=-c_{-i}\cdot d_{i}(\mathbf{p},\mathbf{r})\cdot
d_{-i}(\mathbf{p},\mathbf{r}).$ (92d)

Meanwhile, $G_{i}(\mathbf{p},\mathbf{r})$ and its gradient are bounded as
follows

$\big{|}G_{i}(\mathbf{p},\mathbf{r})\big{|}\leq
M_{G},\quad\big{\|}\nabla_{\mathbf{r}}G_{i}(\mathbf{p},\mathbf{r})\big{\|}_{2}\leq
l_{r},\quad\forall\mathbf{p},\mathbf{r}\in\mathcal{P}^{2}\text{ and }\forall
i\in\\{H,L\\},$ (93)

where the upper bound constant $M_{G}$ and the Lipschitz constant $l_{r}$ are
defined as

$\displaystyle
M_{G}:=\max\left\\{\dfrac{1}{(b_{H}+c_{L})\underline{p}},\dfrac{1}{(b_{L}+c_{L})\underline{p}}\right\\}+1,\quad
l_{r}:=\dfrac{1}{4}\sqrt{c_{H}^{2}+c_{L}^{2}}.$ (94)

Proof. We first verify the partial derivatives from Eq. (92a) to Eq. (92d):

$\displaystyle\dfrac{\partial G_{i}(\mathbf{p},\mathbf{r})}{\partial p_{i}}$
$\displaystyle=\dfrac{1}{(b_{i}+c_{i})p_{i}^{2}}+\dfrac{\partial
d_{i}(\mathbf{p},\mathbf{r})}{\partial p_{i}}$ (95)
$\displaystyle=\dfrac{1}{(b_{i}+c_{i})p_{i}^{2}}-\dfrac{(b_{i}+c_{i})\cdot\exp\big{(}u_{i}(p_{i},r_{i})\big{)}\cdot\Big{(}1+\exp\big{(}u_{-i}(p_{-i},r_{-i})\big{)}\Big{)}}{\Big{(}1+\exp\big{(}u_{i}(p_{i},r_{i})\big{)}+\exp\big{(}u_{-i}(p_{-i},r_{-i})\big{)}\Big{)}^{2}}$
$\displaystyle=\dfrac{1}{(b_{i}+c_{i})p_{i}^{2}}-(b_{i}+c_{i})\cdot
d_{i}(\mathbf{p},\mathbf{r})\cdot\big{(}1-d_{i}(\mathbf{p},\mathbf{r})\big{)}.$
$\displaystyle\dfrac{\partial G_{i}(\mathbf{p},\mathbf{r})}{\partial p_{-i}}$
$\displaystyle=\dfrac{\partial d_{i}(\mathbf{p},\mathbf{r})}{\partial p_{-i}}$
$\displaystyle=\dfrac{(b_{-i}+c_{-i})\cdot\exp\big{(}u_{i}(p_{i},r_{i})\big{)}\cdot\exp\big{(}u_{-i}(p_{-i},r_{-i})\big{)}}{\Big{(}1+\exp\big{(}u_{i}(p_{i},r_{i})\big{)}+\exp\big{(}u_{-i}(p_{-i},r_{-i})\big{)}\Big{)}^{2}}$
$\displaystyle=(b_{-i}+c_{-i})\cdot d_{i}(\mathbf{p},\mathbf{r})\cdot
d_{-i}(\mathbf{p},\mathbf{r}).$

Then, the partial derivatives with respect to $\mathbf{r}$, as shown in Eq.
(92c) and Eq. (92d), can be similarly computed.

In the next part, we show that $G_{i}(\mathbf{p},\mathbf{r})$ is bounded for
$\mathbf{p},\mathbf{r}\in\mathcal{P}^{2}$:

$\displaystyle\big{|}G_{i}(\mathbf{p},\mathbf{r})\big{|}$
$\displaystyle=\left|\dfrac{1}{(b_{i}+c_{i})p_{i}}+d_{i}(\mathbf{p},\mathbf{r})-1\right|$
(96)
$\displaystyle\leq\left|\dfrac{1}{(b_{i}+c_{i})p_{i}}\right|+\big{|}d_{i}(\mathbf{p},\mathbf{r})-1\big{|}$
$\displaystyle\leq\dfrac{1}{(b_{i}+c_{i})\underline{p}}+1.$

Hence, it follows that $\big{|}G_{i}(\mathbf{p},\mathbf{r})\big{|}$ for every
$i\in\\{H,L\\}$ is upper bounded by

$\big{|}G_{i}(\mathbf{p},\mathbf{r})\big{|}\leq\max\left\\{\dfrac{1}{(b_{H}+c_{L})\underline{p}},\dfrac{1}{(b_{L}+c_{L})\underline{p}}\right\\}+1=:M_{G},\quad\forall\mathbf{p},\mathbf{r}\in\mathcal{P}^{2},\forall
i\in\\{H,L\\}.$ (97)

Finally, we demonstrate that
$\left\|\nabla_{\mathbf{r}}G_{i}(\mathbf{p},\mathbf{r})\right\|$ is also
bounded $\forall\mathbf{p},\mathbf{r}\in\mathcal{P}^{2}$ and $\forall
i\in\\{H,L\\}$. From Eq. (92c) and Eq. (92d), we have

$\displaystyle\big{\|}\nabla_{\mathbf{r}}G_{i}(\mathbf{p},\mathbf{r})\big{\|}_{2}^{2}$
$\displaystyle=\Big{(}c_{i}\cdot
d_{i}(\mathbf{p},\mathbf{r})\cdot\big{(}1-d_{i}(\mathbf{p},\mathbf{r})\big{)}\Big{)}^{2}+\Big{(}-c_{-i}\cdot
d_{i}(\mathbf{p},\mathbf{r})\cdot d_{-i}(\mathbf{p},\mathbf{r})\Big{)}^{2}$
(98)
$\displaystyle=c_{i}^{2}\cdot\Big{(}d_{i}(\mathbf{p},\mathbf{r})\cdot\big{(}1-d_{i}(\mathbf{p},\mathbf{r})\big{)}\Big{)}^{2}+c_{-i}^{2}\cdot\Big{(}d_{i}(\mathbf{p},\mathbf{r})\cdot
d_{-i}(\mathbf{p},\mathbf{r})\Big{)}^{2}$
$\displaystyle\leq\dfrac{1}{16}\left(c_{i}^{2}+c_{-i}^{2}\right),$

where the inequality follows from the fact that $x\cdot y\leq 1/4$ for any two
numbers such that $x,y>0$ and $x+y\leq 1$. Thus, it follows that
$\big{\|}\nabla_{\mathbf{r}}G_{i}(\mathbf{p},\mathbf{r})\big{\|}_{2}\leq(1/4)\sqrt{c_{H}^{2}+c_{L}^{2}}:=l_{r}$,
$\forall\mathbf{p},\mathbf{r}\in\mathcal{P}^{2}$ and $\forall i\in\\{H,L\\}$.
$\Box$

## Limitaions

This is a theoretical work that concerns with algorithm design in a
competitive market, with the primary goal of learning a stable equilibrium.
The consumer demand follows the multinomial logit model, and we assume that
their price/reference price sensitivities are positive, i.e., $b_{i},c_{i}>0$.
The feasible price range $\mathcal{P}=[\underline{p},\overline{p}]$ is assumed
to contain the unique SNE, with the lower bound $\underline{p}>0$. We have
justified both assumptions in the main body of the paper (see the discussion
below Eq. (3) and Proposition 3.1).

The convergence results in the paper assume the firms take diminishing step-
sizes $\\{\eta^{t}\\}_{t\geq 0}$ with $\sum_{t=0}^{\infty}\eta^{t}=\infty$,
which is common in the literature of online games (see, e.g., [15, 14, 40]).
We also discuss the extension to constant step-sizes in Remark 5.3.


# Roles of Scaling and Instruction Tuning in Language Perception:
Model vs. Human Attention

Changjiang Gao1 Shujian Huang1 Jixing Li2 Jiajun Chen1
1National Key Laboratory for Novel Software Technology, Nanjing University
2Department of Linguistics and Translation, City University of Hong Kong
<EMAIL_ADDRESS>{huangsj<EMAIL_ADDRESS>
<EMAIL_ADDRESS>Corresponding author

###### Abstract

Recent large language models (LLMs) have revealed strong abilities to
understand natural language. Since most of them share the same basic
structure, i.e. the transformer block, possible contributors to their success
in the training process are scaling and instruction tuning. However, how these
factors affect the models’ language perception is unclear. This work compares
the self-attention of several existing LLMs (LLaMA, Alpaca and Vicuna) in
different sizes (7B, 13B, 30B, 65B), together with eye saccade, an aspect of
human reading attention, to assess the effect of scaling and instruction
tuning on language perception. Results show that scaling enhances the human
resemblance and improves the effective attention by reducing the trivial
pattern reliance, while instruction tuning does not. However, instruction
tuning significantly enhances the models’ sensitivity to instructions. We also
find that current LLMs are consistently closer to non-native than native
speakers in attention, suggesting a sub-optimal language perception of all
models. Our code and data used in the analysis is available on GitHub.

##  1 Introduction

Large language models (LLMs), e.g., GPT-4, ChatGPT and Vicuna, have shown
nearly human-level understanding of text inputs, indicating better human-like
language perception compared with their predecessors such as BERT (Devlin et
al., 2019) and GPT-2 (Radford et al., 2019). However, the mechanism behind
such improvement is largely unknown. One way to interpret such mechanism is
comparing model computation processes to human data. For example, prior work
compared language models (mainly GPT-2) with human neuroimaging data during
language comprehension, suggesting that models with higher human resemblance
also perform better in NLP tasks. (Hasson et al., 2020; Schrimpf et al.,
2021). However, given recent breakthroughs in LLMs , it remains to be tested
whether the newest LLMs still align with human perception data. Based on the
training pipeline of current LLMs, there are two possible sources for the
potential change: the scaled model/data size and the aligning process after
pretraining, such as instruction tuning (Ouyang et al., 2022; Wei et al.,
2022; Sanh et al., 2022). This paper aims to provide a further understanding
of LLMs’ mechanism by investigating the roles of these two factors in language
understanding.

In this paper, we analyze self-attention as a channel to the language
perception ablility of LLMs, because it is the key mechanism of transformer
language models, and it naturally resembles the human attention mechanism in
language perceiving (Merkx and Frank, 2021). The attention patterns under
investigation includes those from trending open-sourced LLMs with different
sizes, i.e., 7B, 13B, 30B, 65B, and different training stages, i.e.,
pretrained (LLaMA) and instruction-tuned (Alpaca and Vicuna) models. Given
these models are all trained mainly with English data, we also compare the
attention patterns of LLMs with human saccade (a form of eye-movement) data,
including both native speakers (L1) and non-native learners (L2) of English
(Li et al., 2019) to see if the models correlates more with L1 subjects.

Our analysis is three-fold. First, we compare the general attention
distributions of LLMs to measure the impact of these two varying factors on
the model (§3.1). Second, we perform linear regressions (LRs) between the
model self-attention and human saccade matrices to see how human-like they
are, and how the resemblance is affected by the two factors (§3.2). Third, we
analyze the relation between the given attention and some trivial patterns,
revealing the difference between different levels of language perception
abilities (§3.3). The main findings are:

  * •

Scaling significantly affects the general attention distribution on plain
text, while instruction tuning has limited effect. However, instruction tuning
enhances the model’s sensitivity to instructions (§5.1).

  * •

Higher human resemblance is significantly correlated to better language
modeling. Scaling improves the resemblance by scaling law (Henighan et al.,
2020), while instruction tuning reduces it. All models have higher resemblance
to L2 rather than to L1, suggesting further room for the improvement in
language perception (§5.2).

  * •

L2 saccade has higher dependency on trivial attention patterns than L1,
indicating a more rigid way of perception. Scaling significantly lowers LLMs’
reliance on trivial attention patterns, while instruction tuning does not
(§5.3).

##  2 Related Work

##### Self-attention analysis

is a common way to interpret transformer models. Several studies has shown
that it is correlated to linguistic information, such as syntactic or semantic
structures (Mareček and Rosa, 2018; Raganato and Tiedemann, 2018; Voita et
al., 2019; Clark et al., 2019). There are also some attempts and debates to
explain the models’ prediction with attention (Jain and Wallace, 2019;
Wiegreffe and Pinter, 2019; Serrano and Smith, 2019; Vashishth et al., 2019).
Different from them, we use self-attention patterns as a measurement of the
language perception abilities of LLMs. Also, some new ways to interpret model
self-attention have been proposed, such as Attention Flow (Abnar and Zuidema,
2020), vector norm (Kobayashi et al., 2020, 2021), ALTI (Ferrando et al.,
2022), Value Zeroing (Mohebbi et al., 2023), and logit analysis (Ferrando et
al., 2023). These methods can raise the interpretability of model attention
w.r.t. linguistic structural information, but are not tested with human eye-
tracking data. Also, some of them suffer from huge computational cost.
Instead, we choose to use the raw attention scores, as it is produced in the
models’ original computational processes, and they already correlate the human
data, demonstrating their interpretability.

##### Instruction tuned LLMs

achieve better performances in performing tasks (Ouyang et al., 2022; Wei et
al., 2022; Sanh et al., 2022). It seems that the LLMs are better at
understanding human instructions. However, there is little discussion on how
the instruction tuning process affects the language perception.

##### Eye-movement in the human reading process

has drawn attention in the computational linguistic field. It is proposed that
scanpaths, i.e. tracks of eye-movement, reveals characteristics of the parsing
processes and relates to working memory during reading (von der Malsburg and
Vasishth, 2013), and the irregularity of scanpaths is affected by sentence
structures and age (von der Malsburg et al., 2015). Such work provides
theoretical ground for our analysis on saccade data.

##### Joint study of language models and human eye-movement

is also receiving growing interest. Prior studies have shown the correlation
between language processing of human and artificial neural networks (Kell et
al., 2018), especially pretrained language models such as GPT-2 (Caucheteux et
al., 2021, 2022; Schrimpf et al., 2021; Lamarre et al., 2022). Especially, Oh
and Schuler (2023a, b) explored the fit of model surprisal to human reading
times, noting the effect of models’ sequencial memorization and dataset size.
Hollenstein and Beinborn (2021) studied the correlation between relative word
importance derived from models and humans, and Morger et al. (2022) extended
that research to a multilingual setting, as well as analysis on model
attention. Some studies also use human attention as supervision to assist
specific tasks (Sood et al., 2020; Bansal et al., 2023), or use sequencial
models to predict eye-tracking data (Chersoni et al., 2022). Different from
them, we use human attention as references for the language perception
abilities.

##  3 Methods

To show the effect of scaling and instruction tuning on different models, we
compare the self-attention scores of different LLMs given the same input, by
viewing model attention as probability distributions and calculating the
general attention divergence based on Jensen-Shannon divergence.

To analyze model self-attention, we take the human saccade as a reference, and
design a human resemblance metric based on linear regression scores.

In addition, we select several trivial, context-free attention patterns, and
design a trivial pattern reliance metric to help demonstrate the difference
between models and human subject groups.

###  3.1 General Attention Divergence

We assesses the mean Jensen-Shannon (J-S) divergence between attention from
different models on all sentences. For two attention matrices $A,\
B\in\mathbb{R}^{n_{\mathrm{word}}\times n_{\mathrm{word}}}$, the J-S
divergence is:

$\displaystyle D_{\mathrm{JS}}(A,B)=$
$\displaystyle\frac{1}{2}\sum_{i=1}^{n_{\mathrm{word}}}[D_{\mathrm{KL}}(A_{i}\|B_{i})$
$\displaystyle+D_{\mathrm{KL}}(B_{i}\|A_{i})]$

where $D_{\mathrm{KL}}$ is the Kullback–Leibler (K-L) divergence, $A_{i}$ and
$B_{i}$ are the $i$-th rows in the two matrices. Note the attention matrices
are re-normalized to keep the sum of every row to 1.0. We calculate the J-S
divergence between heads-averaged attention scores per sentence, and
demonstrate the mean values. For models with the same number of layers, the
divergence is calculated layerwise; For models with different numbers of
layers, we divide their layers into four quarters and calculate the divergence
quarter-wise.

To identify high and low divergence, a reference value is required. We propose
to use the attention divergence between two models that share the same
structure, size, and training data, but have small difference in training
strategies as the reference. A divergence value higher than the reference
suggests relatively large change attention, while one lower than that
indicates small (but not necessarily negligible) change.

##### Divergence by Scaling and Instruction Tuning

Models with different scales or training stages are compared with the above
divergence to show the effect of different factors.

##### Sensitivity to Instructions

To evaluate the effect of instruction tuning, we measure a given model’s
sensitivity to instructions by the same general attention divergence metric,
but on different input, i.e., plain input and input with instructions. If the
divergence between them is significantly higher than reference, we say the
model has high sensitivity to instructions.

To construct instructed text, we attach two prefixes to plain text sentences:
"Please translate this sentence into German:", and "Please paraphrase this
sentence:". They are within the reported capacity of most LLMs, and require no
extra information to follow. Then, we collect model attention on the two
corpora within the original sentence spans (re-normalized) to represent non-
instruction and instruction scenarios. Also, as a control group, we attached a
noise prefix made of 5 randomly sampled English words "Cigarette first steel
convenience champion.", and calculate the divergence between prefixed and
plain text to account for the effect of a meaningless prefix.

###  3.2 Human Resemblance

####  3.2.1 Human Resemblance Calculation

Given the model self-attention and the human saccade data in matrix form for
each sentence, we first extract the lower-triangle parts of the matrices
(marking right-to-left attendance), because prior studies have found that such
eye-movements in reading are related to working memory efficiency (Walczyk and
Taylor, 1996) and parsing strategies (von der Malsburg and Vasishth, 2013),
and the auto-regressive LLMs only have right-to-left self-attention. Then, we
flatten the extracted parts of each sentence attention matrix, and concatenate
them to get attention vectors $\\{v_{\mathrm{human}}^{i}\\}$ for each subject
$i$, and $\\{v_{\mathrm{model}}^{j,k}\\}$ for head $k$ in model layer $j$. We
then stack $\\{v_{\mathrm{model}}^{j,k}\\}$ along the attention heads and get
a matrix $\\{V_{\mathrm{model}}^{j}\\}$ for the $j$-th layer. The LR for model
layer $j$ and subject $i$ is defined as:

$\arg\min_{w,b}\left\|V_{\mathrm{model}}^{j\top}w+b-v_{\mathrm{human}}^{i}\right\|_{2}^{2}$

This means we take the linear combination of attention scores from all heads
in a layer to predict the human saccade. Note that the model attention here
are not re-normalized, in order to preserve the relative weight w.r.t. the <s>
token. The regression scores $R_{i,j}^{2}$ (ranging from 0 to 1, larger value
means better regression) are averaged over subjects to represent the human
resemblance of layer $j$, and the highest layer score is considered as the
regression score of the whole model, $R_{\mathrm{model}}^{2}$.

Because the human saccade pattern varies across individuals, there exists a
natural upper bound for human resemblance of models, i.e., the human-human
resemblance, or the inter-subject correlation. To calculate this correlation
within the two language groups, L1 and L2, we take the group mean to regress
over each individual data independently, and use the average regression score
$R_{\mathrm{inter}}^{2}$ as the inter-subject correlation for that group. The
final human resemblance score is
$R_{\mathrm{model}}^{2}/R_{\mathrm{inter}}^{2}$, ranging from 0 to 100%.

####  3.2.2 Human Resemblance vs. Next-Token Prediction

A previous study finds a positive correlation between the next-token
predication (NTP) performance and the "Brain Score" of neural networks
(Schrimpf et al., 2021), which supports the intuition that the more human-like
models are, the better generation they output. To test this finding in our
setting, we calculated the per-token prediction loss (the negative log
likelihood loss normalized by sequence lengths) of the models employed in this
study on the Reading Brain dataset, and calculate the Pearson’s Correlation
between their prediction loss and max layerwise human resemblance scores.

###  3.3 Trivial Pattern Reliance

####  3.3.1 Trivial Pattern Reliance of Models

We select three representative trivial patterns of transformer attention:
attending to the first word in the sentence, attending to the previous word
(Vig and Belinkov, 2019), and self-attending (Clark et al., 2019), and analyze
the effect of scaling and instruction tuning on models’ reliance on them.

First, we represent the trivial patterns as binary adjacent matrices, where
attending relations are marked with 1, others with 0, and use their flattened
lower-triangle parts as the independent variables. The model attention are
similarly processed to be the dependent variables. Then, an LR is fit between
the two, and the regression scores are collected to represent the trivial
pattern reliance of each model layer.

####  3.3.2 Trivial Pattern Reliance of Humans

This research compares the models’ resemblance to L1 and L2 humans because
intuitively, L1 is better than L2 in language understanding. To test this
intuition, we compare their reliance on the three trivial patterns to see
whether their attention mode differs. Similar to model attention, the
flattened human saccade vectors are used as the dependent variable to fit the
LRs with the trivial patterns.

(a) Individual

(b) Group Mean

Figure 1: Examples of individual and group mean saccade of the sentence "Could
humans live on Mars some day?".

##  4 Experiment Settings

###  4.1 Data

We use the Reading Brain dataset (Li et al., 2019) for both the text input and
the human data. The dataset includes 5 English STEM articles. Each article has
29.6±0.68 sentences of 10.33±0.15 words. It also includes human eye-tracking
and fMRI data recorded synchronously during self-paced reading on the
articles. The human subjects are 52 native English speakers (L1) and 56 non-
native learners of English (L2). We choose this dataset for the following
reasons: (1) The text is presented to the subjects sentence by sentence
instead of page by page, making in-sentence reading patterns clearer. (2) It
has synchronous eye-tracking and fMRI data, which allows both behavioral- and
neural-level analysis. In this article we mainly focus on the behavioral part,
but we also have some preliminary results on the fMRI analysis (See Appendix
D).

We take the saccade number, i.e. times of eye-movements from a word to
another, instead of saccade duration, to represent the human reading
attention, because this reduces the effect of other factors such as word
length. The saccade data of a human subject for a given sentence is a
$n_{\mathrm{word}}\times n_{\mathrm{word}}$ matrix. Figure 1 gives examples of
individual and group mean saccades.

###  4.2 LLMs

We employ LLaMA (Touvron et al., 2023) and its instruction-tuned versions,
Alpaca (Taori et al., 2023) and Vicuna (Chiang et al., 2023) in this research.
We use them because: (1) The model sizes and different instruction-tuning
methods covers the majority of current open-sourced LLMs. (2) These models are
being widely used and customized globally. Thus, analysis on them is
representative and meaningful.

##### LLaMA

is a series of pretrained causal language models with parameter sizes 7B, 13B,
30B and 65B, trained on over 1T publicly available text tokens, and reaches
state-of-the-art on most LLM benchmarks (Touvron et al., 2023). The training
corpus is mainly English. We use all 4 sizes of LLaMA, which covers the sizes
of most current open-sourced LLMs. We also use the 774M GPT-2 Large (Radford
et al., 2019) to represent smaller pretrained models in the scaling analysis.

##### Alpaca

is fine-tuned from the 7B LLaMA model on 52K English instruction-following
demonstrations generated by GPT-3 (Brown et al., 2020) using self-instruct
(Wang et al., 2023). To analyze the effect of scaling, we also trained a 13B
version of Alpaca using the official data and training strategy of Alpaca. Our
13B Alpaca model scores 43.9 and 46.0 on the MMLU dataset in zero-shot and
one-shot setting respectively, proving the soundness of our fine-tuning. (The
corresponding scores are 40.9 and 39.2 for the official Alpaca 7B
model.111Based on this GitHub repository for LLM evaluation.)

##### Vicuna

models are fine-tuned from the 7B and 13B LLaMA models on 70K user-shared
conversations with ChatGPT. The data contains instruction and in-context
learning samples in multiple languages, thus the Vicuna models can also be
viewed as instruction-tuned.

###  4.3 LLMs Attention Collection

To obtain model attention, text is provided to the models sentence by sentence
to perform next token predict, and we collect the attention tensors in each
layer. This resembles the human self-paced task-free reading process, where
only one sentence is visible at a time to the subjects, and the natural human
behavior is to predict the next word (Kell et al., 2018; Schrimpf et al.,
2021).

Because models use the SentencePiece tokenization (Kudo and Richardson, 2018),
but the human saccade data is recorded in words, the attention matrices need
to be reshaped. Following Clark et al. (2019) and Manning et al. (2020), we
take the sum over the "to" tokens, and the average over the "from" tokens in a
split word. Also, the <s> token is dropped as it is not presented to human
subjects.

##### Reference of Divergence

We use the attention divergence of Vicuna v0 and Vicuna v1.1 13B as the
reference values. The two versions share the same training data and strategy,
but have minor difference in separators and loss computation222According to
notes in the Vicuna GitHub repository.. We use the divergence calculated on
13B Vicunas across all divergence analyses, because it has lower mean and
variance than the 7B one, making our comparison fairer and stricter.

##  5 Results

Figure 2: Mean J-S divergence between 7B and 13B model attention in each layer
quarter, measured on non-instruction sentences.

###  5.1 General Attention Divergence

####  5.1.1 Scaling Significantly Changes the General Attention

The general attention divergence between models in different sizes is shown in
Figure 2. The results shows that LLaMA, Alpaca and Vicuna all reveal
divergence significantly higher than the reference values when scaled from 7B
to 13B. This means scaling causes large change in the overall attention
distribution.

####  5.1.2 Instruction Tuning Has Limited Effect on General Attention

The general attention divergence between pretrained and instruction tuned
models is shown in Figure 3. It can be observed that only Vicuna 13B brings
divergence above the reference values, while all the others don’t. The same
result can also be observed on instruction-prefixed sentences. This suggests
that instruction tuning can only bring limited change to the models’ general
attention distribution. This can also be verified by other results in the
following parts.

Figure 3: General attention divergence between pretrained and instruction-
tuned models in 7B and 13B, measured on non-instruction sentences.

####  5.1.3 Instruction Tuning Enhances Sensitivity to Instructions

The divergence between the models’ attention obtained on the original and
instruction-attached sentences are shown in Figure 4. There are two major
observations. Firstly, LLaMA, Alpaca and Vicuna all show divergence
significantly above the reference values between the two types of sentences,
especially in the shallower layers, which means they all have sensitivity to
instructions. Secondly, while the divergence of LLaMA drops in the deeper
layers, the divergence of Alpaca and Vicuna does not go down, but rather up,
suggesting higher sensitivity to instructions in the deeper layers. This
effect is not observed in the noise-prefixed scenario (See Appendix A).

From these two observations, we can conclude that the pretrained LLaMA has
already obtained some sensitivity to instructions, i.e. to change its
attention mode when it meets instructions, and this sensitivity is further
strengthened by instruction tuning, which makes this sensitivity higher in the
deeper layers. Because in causal language models like LLaMA, the last few
layers are related to generation, this effect of instruction tuning may bring
more suitable generation for a certain instruction.

Figure 4: Attention divergence on the original and instruction-attached text
of 7B and 13B models.

###  5.2 Human Resemblance

####  5.2.1 Higher Human Resemblance, Lower Prediction Loss

The per-token NTP loss and the maximum layerwise human resemblance scores are
compared in Table 1 and Figure 5. One can tell from it that, within the
considered range of parameter sizes, the human resemblance is negatively
correlated with the NTP loss, where the resemblance goes higher nearly
linearly with the loss going lower. The Pearson’s correlation is $-0.875$ for
L1 ($p<0.002$) and $-0.917$ for L2 ($p<0.0005$), supporting a significant and
strong negative linear correlation.

This result shows that, the human resemblance we defined is positively related
to the language modeling performance of the LLMs considered in this research,
giving a practical meaning to our human resemblance analysis. Also, based on
this result, we know the pursuit for higher human resemblance is consistent
with the target of training better LLMs. Similarly, factors negatively
impacting the human resemblance could also harm the models’ language modeling
performance, such as instruction tuning brings lower human resemblance as well
as higher NTP loss.

Size | Name | Loss | L1(%) | L2(%)
---|---|---|---|---
774M | GPT-2 | 0.3264 | 34.99 | 40.03
7B | LLaMA | 0.2408 | 53.04 | 62.44
Alpaca | 0.2646 | 52.51 | 61.71
Vicuna | 0.2593 | 51.90 | 61.19
13B | LLaMA | 0.2406 | 55.66 | 64.20
Alpaca | 0.2634 | 55.05 | 64.46
Vicuna | 0.2847 | 54.26 | 61.31
30B | LLaMA | 0.2372 | 63.16 | 69.40
65B | LLaMA | 0.2375 | 64.05 | 70.07
Table 1: Comparison between NTP loss and the human resemblance scores Figure
5: The correlation between the per-token negative log likelihood loss and the
max layerwise human resemblance of LLMs.

####  5.2.2 Scaling Enhances Human Resemblance

The layerwise human resemblance of pretrained models in different scales are
shown in Figure 6, where the LLaMAs are very different from GPT-2 Large in two
ways. First, the layerwise human resemblance of the LLaMAs are much higher
than GPT-2, but the gaps among the LLaMAs are small; Second, human resemblance
of GPT-2 drop quickly in the deeper layers, while it remains high across all
layers in the LLaMAs. one can refer from these results that LLaMA has
undergone a phase change in terms of human-like attention from the level of
GPT-2. However, the scaling from 7B to 30B does not cause another phase
change.

Table 2 and Figure 7 show the max layerwise human resemblance increases
linearly while the model parameter size increases exponentially, where the
Pearson’s correlation is 0.989 for L1 ($p<0.002$) and 0.964 for L2 ($p<0.01$).
This agrees with the scaling law of LLMs (Henighan et al., 2020).

This result shows that, within the scope of this study, scaling significantly
enhances the human resemblance. If the scaling law continues to take effect,
the human resemblance is expected to reach 98.80% for L2 and 88.82% for L1 at
the scale of 100B. However, because the law is expected to fail after reaching
a threshold (Hestness et al., 2017), the relation between human resemblance
and model scale is expected to change before reaching that scale.

Figure 6: Layerwise L1 and L2 human resemblance for different sized pretrained models. Model | Resemblance (%)
---|---
Name | Size | L1 | L2
GPT-2 | 774M | 34.99 | 40.03
LLaMA | 7B | 53.04 | 62.44
13B | 55.56 | 64.20
30B | 63.16 | 69.40
65B | 64.05 | 70.07
Table 2: Max layerwise human resemblance of models ranging from 774M to 65B.
Figure 7: The change of max layerwise human resemblance caused by the scaling
of model sizes, where the x-axis is log scaled.

####  5.2.3 Instruction Tuning Harms Human Resemblance

Table 3 shows the max layerwise human resemblance of the pretrained and
instruction tuned models. It shows that the instruction tuned Alpaca and
Vicuna models have lower max layerwise human resemblance to both L1 and L2
than LLaMA in both sizes, which suggests that instruction tuning causes
decline in the human resemblance of attention. However, the decline is minor
compared with the increase brought by scaling, which agrees with the limited
effect of instruction tuning.

We also conduct relative t-tests to detect significant difference in layerwise
human resemblance ($p=0.05/n_{\mathrm{test}}$). The numbers of significant
layers are shown in Table 4, and the complete lists of layers are in Table 6,
7 in Appendix C). The tables show that instruction tuning enhances or reduces
human resemblance on about the equal numbers of layers on the 7B models, but
reduces human resemblance to much more layers on the 13B models. Also, the
layers with significantly lower resemblance are distributed across all layers.
This result means that, while the total effect of instruction tuning is small,
the drop in human resemblance caused by it is significant and widespread.

To sum up, instruction tuning does small but significant harm to the human
resemblance of LLMs, which in turn may bring potential damage to their
language perception. This challenges the common assumption that instruction
tuning can activate the model’s ability to perceive language with human-
aligned behaviors. However, this negative effect could be compensated by
scaling, for scaling increases the human resemblance in a larger degree. This
provides a novel reason for further scaling.

Model | Resemblance(%)
---|---
Size | Name | L1 | L2
7B | LLaMA | 53.04 | 62.44
Alpaca | 52.51 | 61.71
Vicuna | 51.90 | 61.19
13B | LLaMA | 55.66 | 64.20
Alpaca | 55.05 | 63.46
Vicuna | 54.26 | 61.31
Table 3: Max layerwise human resemblance scores of pretrained and instruction tuned models. This shows instruction tuning causes small decline in the max layerwise human resemblance. Model | Higher | Lower
---|---|---
Size | Name | L1 | L2 | L1 | L2
7B | Alpaca | 3 | 8 | 4 | 7
Vicuna | 9 | 10 | 8 | 9
13B | Alpaca | 4 | 7 | 16 | 25
Vicuna | 5 | 8 | 14 | 16
Table 4: Numbers of Alpaca and Vicuna layers with significantly higher or
lower human resemblance compared with LLaMA in the same sizes.

####  5.2.4 All Models Show Higher L2 Resemblance

Comparing the L1 and L2 human resemblance of all models (LLaMA, Alpaca and
Vicuna, 7B and 13B) in Table 2 and Table 3, an advantage of L2 over L1 can be
observed steadily. Independent t-test also shows that, all models considered
in this research show significantly higher resemblance to L2 than to L1
($p=0.05/n_{\mathrm{test}}$ to avoid false positives). This means the models
are closer to non-native English learners than to native English speakers in
attention, though they are all trained mainly on English data. Furthermore,
this trend is not reversed by the scaling or instruction tuning within the
scope of this research, which suggests that it is a innate feature of LLMs. We
will look further into this phenomenon in the next section.

###  5.3 Trivial Pattern Reliance

####  5.3.1 L2 Relies More on Trivial Patterns

The results of LR scores between trivial patterns and L1 and L2 human saccade
is shown in Table 5, where L2’s score is higher than L1’s in minimum, maximum
and mean values. Independent t-test also supports a significant advantage of
L2 over L1 on the regression scores ($p<5\times 10^{-8}$). This means one
significant difference between L1 and L2 English speakers is that L2 people
show more trivial and fixed attention mode than L1 people while reading, which
suggests a weaker language understanding ability as those patterns contain no
linguistic or factual information.

This finding can help us interpret the difference between the models’ L1 and
L2 human resemblance. Note that all models considered in this research are
trained mainly on English, but show higher attention resemblance to L2
subjects, this result suggests that their language perception is not ideal.

Group | Min | Max | Mean | SE
---|---|---|---|---
L1 | 0.0048 | 0.1525 | 0.0473 | 0.0037
L2 | 0.0134 | 0.2165 | 0.0894 | 0.0060
Table 5: Regression scores of human saccade on the trivial patterns, across L1
and L2 subjects. SE stands for standard errors.

####  5.3.2 Scaling Reduces Trivial Pattern Reliance

The relative difference between 13B and 7B models in trivial pattern reliance
is shown in Figure 8, where the decline in the deeper layer quarters is
substantial across all models. There is also a consistent increase in the
second quarter, but the amplitude is small, so the total trivial pattern
reliance is reduced. This phenomenon is also observed between LLaMA 13B and
30B (Figure 11 in Appendix B). This means scaling can effectively reduce the
trivial pattern reliance, especially in the deeper layers, indicating a more
contextualized and adaptive language perception.

Figure 8: The difference between trivial pattern reliance of 7B and 13B model
attention.

####  5.3.3 Instruction Tuning Increases Trivial Pattern Reliance

Figure 9 shows the relative difference between pretrained and instruction
tuned models in trivial pattern reliance, where the tuned ones show reliance
gain in the deeper layers. There is also a small drop in the second quarter,
but the total trivial pattern reliance is increasing. Also, total increase
here is also smaller in amplitude than the reduction caused by scaling, again
supporting the limit effect of instruction tuning.

Also, one can tell from Figure 8 and Figure 9 that scaling and instruction
tuning changes the trivial pattern reliance in opposite directions. However,
because the effect of instruction tuning is also smaller than that of scaling,
as long as the model size continues to grow, this gain in trivial pattern
reliance brought by instruction tuning could be compensated or even covered,
offereing another reason for further scaling.

Figure 9: The difference in trivial pattern reliance of attentions in
pretrained and instruction-tuned models.

##  6 Conclusion

This research evaluates the effects of scaling and instruction tuning on LLMs’
attention. We find that scaling effectively changes the general attention
distribution, enhances the human resemblance, and reduces the trivial pattern
reliance, while instruction tuning does the opposite, but also increases the
models’ sensitivity to instructions. Furthermore, we find current open-sourced
LLMs closer to non-native English speakers in language perception, which
contains more trivial attention patterns. To the best of our knowledge, we are
the first to analyze the effect of instruction tuning on the human resemblance
of model attention. We hope this can inspire future work on analyzing LLMs.

## Limitations

One key limitation of this study is that we cannot apply our current method to
closed-sourced LLMs such as ChatGPT, for their layerwise attention scores are
unavailable. With the rapid development of open LLMs, it is hopeful that we
can examine these findings on the largest and most advanced LLMs in the near
future.

Besides, though we demonstrate the difference in LLMs’ resemblance to L1 and
L2 people, and partially explained the effect by trivial pattern reliance, it
is not enough to explain the underlying cause or consequences of this
phenomenon. We do have some observations, such as the inter-subject
correlation in the L1 group ($0.1560\pm 0.0073$) is lower than L2 ($0.2463\pm
0.0090$), and L1 subjects tend to look back to pervious words less frequently
than L2 subjects. However, the underlying mechanism is still unclear. We hope
to dive deeper into this non-native resemblance in our following study.

Another limitation of this work comes from the way we obtain model attention.
We used the attention scores when predicting the next single token, which
makes sense. However, the attention patterns could be different in the
following decoding steps. Also, it is possible that combining attention scores
across multiple layers can improve the analysis. We will investigate this in
our future work.

In addition, the Reading Brain dataset we used in this research is
monolingual, which does not reflect the language perception process in the
cross-lingual setting, which is also important for the application of LLMs. We
also hope to examine our findings in the cross-lingual scenario in the future.

## Ethics Statement

The authors declare no competing interests. The human behavioral data we use
is publicly available and does not contain personal information of the
subjects.

## Acknowledgements

We would like to thank the anonymous reviewers for their insightful comments.
Shujian Huang is the corresponding author. We thank Yunzhe Lv and Wenhao Zhu
for finetuning the Alpaca 13B model. This work is supported by National
Science Foundation of China (No. 62376116, 62176120), the Liaoning Provincial
Research Foundation for Basic Research (No. 2022-KF-26-02).

## References

  * Abnar and Zuidema (2020) Samira Abnar and Willem Zuidema. 2020.  Quantifying Attention Flow in Transformers.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 4190–4197, Online. Association for Computational Linguistics.
  * Bansal et al. (2023) Aakash Bansal, Bonita Sharif, and Collin McMillan. 2023.  Towards modeling human attention from eye movements for neural source code summarization.  _Proceedings of the ACM on Human-Computer Interaction_ , 7(ETRA):1–19.
  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020.  Language models are few-shot learners.  _Advances in neural information processing systems_ , 33:1877–1901.
  * Caucheteux et al. (2021) Charlotte Caucheteux, Alexandre Gramfort, and Jean-Rémi King. 2021.  Model-based analysis of brain activity reveals the hierarchy of language in 305 subjects.  _arXiv preprint arXiv:2110.06078_.
  * Caucheteux et al. (2022) Charlotte Caucheteux, Alexandre Gramfort, and Jean-Rémi King. 2022.  Deep language algorithms predict semantic comprehension from brain activity.  _Scientific Reports_ , 12(1):16327.
  * Chersoni et al. (2022) Emmanuele Chersoni, Nora Hollenstein, Cassandra Jacobs, Yohei Oseki, Laurent Prévot, and Enrico Santus, editors. 2022.  _Proceedings of the Workshop on Cognitive Modeling and Computational Linguistics_. Association for Computational Linguistics, Dublin, Ireland.
  * Chiang et al. (2023) Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. 2023.  Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality.
  * Clark et al. (2019) Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher D. Manning. 2019.  What does BERT look at? an analysis of BERT’s attention.  In _Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP_ , pages 276–286, Florence, Italy. Association for Computational Linguistics.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Ferrando et al. (2022) Javier Ferrando, Gerard I. Gállego, and Marta R. Costa-jussà. 2022.  Measuring the Mixing of Contextual Information in the Transformer.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 8698–8714, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * Ferrando et al. (2023) Javier Ferrando, Gerard I. Gállego, Ioannis Tsiamas, and Marta R. Costa-jussà. 2023.  Explaining How Transformers Use Context to Build Predictions.  In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 5486–5513, Toronto, Canada. Association for Computational Linguistics.
  * Hasson et al. (2020) Uri Hasson, Samuel A. Nastase, and Ariel Goldstein. 2020.  Direct fit to nature: An evolutionary perspective on biological and artificial neural networks.  _Neuron_ , 105(3):416–434.
  * Henighan et al. (2020) Tom Henighan, Jared Kaplan, Mor Katz, Mark Chen, Christopher Hesse, Jacob Jackson, Heewoo Jun, Tom B. Brown, Prafulla Dhariwal, Scott Gray, Chris Hallacy, Benjamin Mann, Alec Radford, Aditya Ramesh, Nick Ryder, Daniel M. Ziegler, John Schulman, Dario Amodei, and Sam McCandlish. 2020.  Scaling Laws for Autoregressive Generative Modeling.  ArXiv:2010.14701 [cs].
  * Hestness et al. (2017) Joel Hestness, Sharan Narang, Newsha Ardalani, Gregory Diamos, Heewoo Jun, Hassan Kianinejad, Md Mostofa Ali Patwary, Yang Yang, and Yanqi Zhou. 2017.  Deep learning scaling is predictable, empirically.  _arXiv preprint arXiv:1712.00409_.
  * Hollenstein and Beinborn (2021) Nora Hollenstein and Lisa Beinborn. 2021.  Relative Importance in Sentence Processing.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers)_ , pages 141–150, Online. Association for Computational Linguistics.
  * Jain and Wallace (2019) Sarthak Jain and Byron C. Wallace. 2019.  Attention is not Explanation.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 3543–3556, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Kell et al. (2018) Alexander JE Kell, Daniel LK Yamins, Erica N Shook, Sam V Norman-Haignere, and Josh H McDermott. 2018.  A task-optimized neural network replicates human auditory behavior, predicts brain responses, and reveals a cortical processing hierarchy.  _Neuron_ , 98(3):630–644.
  * Kobayashi et al. (2020) Goro Kobayashi, Tatsuki Kuribayashi, Sho Yokoi, and Kentaro Inui. 2020.  Attention is Not Only a Weight: Analyzing Transformers with Vector Norms.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 7057–7075, Online. Association for Computational Linguistics.
  * Kobayashi et al. (2021) Goro Kobayashi, Tatsuki Kuribayashi, Sho Yokoi, and Kentaro Inui. 2021.  Incorporating Residual and Normalization Layers into Analysis of Masked Language Models.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 4547–4568, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Kudo and Richardson (2018) Taku Kudo and John Richardson. 2018.  SentencePiece: A simple and language independent subword tokenizer and detokenizer for neural text processing.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_ , pages 66–71, Brussels, Belgium. Association for Computational Linguistics.
  * Lamarre et al. (2022) Mathis Lamarre, Catherine Chen, and Fatma Deniz. 2022.  Attention weights accurately predict language representations in the brain.  In _Findings of the Association for Computational Linguistics: EMNLP 2022_ , pages 4513–4529, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * Li et al. (2019) Ping Li, Anya Yu, Benjamin Schloss, Chun-Ting Hsu, Lindsey Ma, Chih-Ting Chang, and Marissa Scotto. 2019.  The reading brain project: An open science data-sharing initiative.  In _26th Annual Meeting of the Cognitive Neuroscience Society_.
  * Manning et al. (2020) Christopher D Manning, Kevin Clark, John Hewitt, Urvashi Khandelwal, and Omer Levy. 2020.  Emergent linguistic structure in artificial neural networks trained by self-supervision.  _Proceedings of the National Academy of Sciences_ , 117(48):30046–30054.
  * Mareček and Rosa (2018) David Mareček and Rudolf Rosa. 2018.  Extracting syntactic trees from transformer encoder self-attentions.  In _Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP_ , pages 347–349, Brussels, Belgium. Association for Computational Linguistics.
  * Merkx and Frank (2021) Danny Merkx and Stefan L. Frank. 2021.  Human sentence processing: Recurrence or attention? In _Proceedings of the Workshop on Cognitive Modeling and Computational Linguistics_ , pages 12–22, Online. Association for Computational Linguistics.
  * Mohebbi et al. (2023) Hosein Mohebbi, Willem Zuidema, Grzegorz Chrupa\la, and Afra Alishahi. 2023.  Quantifying Context Mixing in Transformers.  In _Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics_ , pages 3378–3400, Dubrovnik, Croatia. Association for Computational Linguistics.
  * Morger et al. (2022) Felix Morger, Stephanie Brandl, Lisa Beinborn, and Nora Hollenstein. 2022.  A Cross-lingual Comparison of Human and Model Relative Word Importance.  In _Proceedings of the 2022 CLASP Conference on (Dis)embodiment_ , pages 11–23, Gothenburg, Sweden. Association for Computational Linguistics.
  * Oh and Schuler (2023a) Byung-Doh Oh and William Schuler. 2023a.  Transformer-Based LM Surprisal Predicts Human Reading Times Best with About Two Billion Training Tokens.
  * Oh and Schuler (2023b) Byung-Doh Oh and William Schuler. 2023b.  Why Does Surprisal From Larger Transformer-Based Language Models Provide a Poorer Fit to Human Reading Times? _Transactions of the Association for Computational Linguistics_ , 11:336–350.
  * Ouyang et al. (2022) Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. 2022.  Training language models to follow instructions with human feedback.  ArXiv:2203.02155 [cs].
  * Radford et al. (2019) Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019.  Language Models are Unsupervised Multitask Learners.
  * Raganato and Tiedemann (2018) Alessandro Raganato and Jörg Tiedemann. 2018.  An analysis of encoder representations in transformer-based machine translation.  In _Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP_ , pages 287–297, Brussels, Belgium. Association for Computational Linguistics.
  * Sanh et al. (2022) Victor Sanh, Albert Webson, Colin Raffel, Stephen H. Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Teven Le Scao, Arun Raja, Manan Dey, M. Saiful Bari, Canwen Xu, Urmish Thakker, Shanya Sharma Sharma, Eliza Szczechla, Taewoon Kim, Gunjan Chhablani, Nihal Nayak, Debajyoti Datta, Jonathan Chang, Mike Tian-Jian Jiang, Han Wang, Matteo Manica, Sheng Shen, Zheng Xin Yong, Harshit Pandey, Rachel Bawden, Thomas Wang, Trishala Neeraj, Jos Rozen, Abheesht Sharma, Andrea Santilli, Thibault Fevry, Jason Alan Fries, Ryan Teehan, Tali Bers, Stella Biderman, Leo Gao, Thomas Wolf, and Alexander M. Rush. 2022.  Multitask Prompted Training Enables Zero-Shot Task Generalization.  ArXiv:2110.08207 [cs].
  * Schrimpf et al. (2021) Martin Schrimpf, Idan Asher Blank, Greta Tuckute, Carina Kauf, Eghbal A Hosseini, Nancy Kanwisher, Joshua B Tenenbaum, and Evelina Fedorenko. 2021.  The neural architecture of language: Integrative modeling converges on predictive processing.  _Proceedings of the National Academy of Sciences_ , 118(45):e2105646118.
  * Serrano and Smith (2019) Sofia Serrano and Noah A. Smith. 2019.  Is attention interpretable? In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 2931–2951, Florence, Italy. Association for Computational Linguistics.
  * Sood et al. (2020) Ekta Sood, Simon Tannert, Philipp Müller, and Andreas Bulling. 2020.  Improving natural language processing tasks with human gaze-guided neural attention.  _Advances in Neural Information Processing Systems_ , 33:6327–6341.
  * Taori et al. (2023) Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023.  Stanford alpaca: An instruction-following llama model.  https://github.com/tatsu-lab/stanford_alpaca.
  * Touvron et al. (2023) Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023.  Llama: Open and efficient foundation language models.
  * Vashishth et al. (2019) Shikhar Vashishth, Shyam Upadhyay, Gaurav Singh Tomar, and Manaal Faruqui. 2019\.  Attention interpretability across nlp tasks.  _arXiv preprint arXiv:1909.11218_.
  * Vig and Belinkov (2019) Jesse Vig and Yonatan Belinkov. 2019.  Analyzing the structure of attention in a transformer language model.  In _Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP_ , pages 63–76, Florence, Italy. Association for Computational Linguistics.
  * Voita et al. (2019) Elena Voita, David Talbot, Fedor Moiseev, Rico Sennrich, and Ivan Titov. 2019.  Analyzing multi-head self-attention: Specialized heads do the heavy lifting, the rest can be pruned.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 5797–5808, Florence, Italy. Association for Computational Linguistics.
  * von der Malsburg et al. (2015) Titus von der Malsburg, Reinhold Kliegl, and Shravan Vasishth. 2015.  Determinants of Scanpath Regularity in Reading.  _Cognitive Science_ , 39(7):1675–1703.
  * von der Malsburg and Vasishth (2013) Titus von der Malsburg and Shravan Vasishth. 2013.  Scanpaths reveal syntactic underspecification and reanalysis strategies.  _Language and Cognitive Processes_ , 28(10):1545–1578.
  * Walczyk and Taylor (1996) Jeffrey J. Walczyk and Robert W. Taylor. 1996.  How do the efficiencies of reading subcomponents relate to looking back in text? _Journal of Educational Psychology_ , 88(3):537–545.
  * Wang et al. (2023) Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. 2023.  Self-Instruct: Aligning Language Models with Self-Generated Instructions.  ArXiv:2212.10560 [cs].
  * Wei et al. (2022) Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le. 2022.  Finetuned language models are zero-shot learners.  In _International Conference on Learning Representations_.
  * Wiegreffe and Pinter (2019) Sarah Wiegreffe and Yuval Pinter. 2019.  Attention is not not explanation.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 11–20, Hong Kong, China. Association for Computational Linguistics.

##  Appendix A Attention divergence on plain and noise-prefixed text

As shown in Figure 10, the divergence between plain and noise-prefixed text of
Alpaca and Vicuna is not significantly higher than LLaMA in most of the
layers, which further supports our findings on the higher instruction
sensitivity of instruction-tuned models.

Figure 10: Attention divergence on the original and noise-attached text of 7B
and 13B models.

##  Appendix B Layerwise trivial pattern reliance

Figure 11 shows the layerwise trivial pattern reliance of 7B and 13B models,
as well as the results for LLaMA 30B. The reliance keeps decreasing as the
model size scales up, especially in the deeper layers.

Figure 11: Lyerwise trivial pattern reliance of 7B and 13B models. "RB" stands
for "Reading Brain".

##  Appendix C Layers with significant changes in human resemblance after
instruction tuning

Table 6 shows the list of layers in 7B Alpaca and Vicuna with significant
change in human resemblance after instruction tuning compared with LLaMA, and
Table 7 shows the list in 13B.

Relation | Group | Alpaca | Vicuna
---|---|---|---
Higher | L1 | 2, 5, 7 | 12, 13, 17,
| 18, 21, 26,
| 28, 29, 30
L2 | 2, 5, 6, | 3, 13, 15,
8, 12, 15, | 18, 21, 23,
21, 27 | 26, 28, 29,
| 30
Lower | L1 | 4, 10, 23, | 0, 1, 4,
31 | 5, 9, 11,
| 20, 31
L2 | 0, 10, 14, | 0, 1, 4,
19, 23, | 5, 9, 11,
28, 31 | 14, 20, 31
Table 6: Layers of Alpaca and Vicuna with significantly higher or lower human resemblance compared with LLaMA in 7B. Relation | Group | Alpaca | Vicuna
---|---|---|---
Higher | L1 | 16, 32, 33, | 4, 16, 27,
36 | 33, 36
L2 | 1, 16, 19, | 4, 16, 19,
27, 32, 33, | 23, 27, 33,
36 | 36, 37
Lower | L1 | 0, 2, 3, | 0, 1, 3,
5, 6, 7, | 5, 6, 9,
8, 9, 14, | 10, 14, 17,
18, 24, 25, | 18, 24, 30,
28, 30, 35, | 34, 39
39 |
L2 | 0, 3, 4, | 0, 2, 3,
5, 6, 7, | 5, 6, 8,
8, 9, 10, | 9, 10, 11,
11, 12, 14, | 12, 14, 17,
15, 17, 18, | 18, 20, 21
20, 24, 25, | 30
28, 29, 30, |
31, 35, 38, |
39 |
Table 7: Layers of Alpaca and Vicuna with significantly higher or lower human
resemblance compared with LLaMA in 13B.

##  Appendix D Preliminary fMRI results

Some preliminary analysis on the fMRI data in the Reading Brain dataset is
done, and the results comply with our findings on saccade data. We investigate
several linguistics-related brain regions, and find that the highest
correlation of fMRI signals to model attention are from the anterior left
temporal gyrus (Figure 12). The fMRI data from this region is pre-processed
into matrix forms as saccade data. Figure 14 gives examples of individual and
group mean fMRI matrices.

The findings are: (1) The models’ NTP loss significantly ($p<0.02$ for both L1
and L2) and strongly negatively ($r=-0.777$ for L1, $r=-0.796$ for L2)
correlates with fMRI resemblance (Figure 13); (2) The models’ highest
layerwise fMRI resemblance significantly ($r>0.99,\ p<0.001$ for both L1 and
L2) correlates with the model scale in logarithm (Figure 15, 16); (3) While
scaling enhances human fMRI resemblance, instruction tuning does not (Table
8); (4) Resemblance to L2 fMRI is consistantly higher than that to L1.

Figure 12: An illustration of the anterior left temporal gyrus in the human
brain. Figure 13: The correlation between per-token negative log likelihood
loss and max layerwise fMRI resemblance of the LLMs.

(a) Individual

(b) Group Mean

Figure 14: Examples of individual and group mean fMRI signals of the sentence
"Could humans live on Mars some day?".

Figure 15: Layerwise L1 and L2 human fMRI resemblance for different sized pretrained models. Figure 16: The change of max layerwise human fMRI resemblance caused by scaling. The x-axis is log scaled. Model | Resemblance(%)
---|---
Size | Name | L1 | L2
7B | LLaMA | 11.02 | 15.45
Alpaca | 10.99 | 15.61
Vicuna | 11.35 | 15.44
13B | LLaMA | 14.69 | 20.19
Alpaca | 14.67 | 19.44
Vicuna | 14.02 | 18.57
Table 8: Max layerwise human fMRI resemblance scores of pretrained and
instruction tuned models. This shows instruction tuning causes small decline
in the max layerwise human resemblance.