Question:
You are a highly experienced, conscientious, and fair academic reviewer, please help me review this paper. The review should be organized into nine sections:
1. Summary: A summary of the paper in 100-150 words.
2. Strengths/Weaknesses/Questions: The Strengths/Weaknesses/Questions of paper, which should be listed in bullet points, with each point supported by specific examples from the article where possible.
3. Soundness/Contribution/Presentation: Rate the paper's Soundness/Contribution/Presentation, and match this score to the corresponding description from the list below and provide the result. The possible scores and their descriptions are:
    1 poor
    2 fair
    3 good
    4 excellent
4. Rating: Give this paper an appropriate rating, match this rating to the corresponding description from the list below and provide the result. The possible Ratings and their descriptions are:
    1 strong reject
    2 reject, significant issues present
    3 reject, not good enough
    4 possibly reject, but has redeeming facets
    5 marginally below the acceptance threshold
    6 marginally above the acceptance threshold
    7 accept, but needs minor improvements
    8 accept, good paper
    9 strong accept, excellent work
    10 strong accept, should be highlighted at the conference
5. Paper Decision: It must include the Decision itself(Accept or Reject) and the reasons for this decision, based on the criteria of originality, methodological soundness, significance of results, and clarity and logic of presentation.

Here is the template for a review format, you must follow this format to output your review result:
**Summary:**
Summary content

**Strengths:**
- Strength 1
- Strength 2
- ...

**Weaknesses:**
- Weakness 1
- Weakness 2
- ...

**Questions:**
- Question 1
- Question 2
- ...

**Soundness:**
Soundness result

**Presentation:**
Presentation result

**Contribution:**
Contribution result

**Rating:**
Rating result

**Paper Decision:**
- Decision: Accept/Reject
- Reasons: reasons content

Please ensure your feedback is objective and constructive. The paper is as follows:

On the Last-iterate Convergence in Time-varying Zero-sum Games: Extra Gradient Succeeds where Optimism Fails

 Yi Feng

512021310186@live.sufe.edu.cn

&Hu Fu

512021310186@live.sufe.edu.cn

&Qun Hu

512021310186@live.sufe.edu.cn

&Ping Li

512021310186@live.sufe.edu.cn

&Ioannis Panageas

512021310186@live.sufe.edu.cn

&Qun Hu

512021310186@live.sufe.edu.cn

&Qun Hu

512021310186@live.sufe.edu.cn

&Sureg

512021310186@live.sufe.edu.cn

&Qun Hu

512021310186@live.sufe.edu.cn

&Sureg

512021310186@live.sufe.edu.cn

&Qun Hu

512021310186@live.sufe.edu.cn

&Sureg

512021310186@live.sufe.edu.cn

&Qun Hu

512021310186@live.sufe.edu.cn

&Sureg

512021310186@live.sufe.edu.cn

&Qun Hu

512021310186@live.sufe.edu.cn

&Sureg

512021310186@live.sufe.edu.cn

&Qun Hu

512021310186@live.sufe.edu.cn

&Sureg

512021310186@live.sufe.edu.cn

&Qun Hu

512021310186@live.sufe.edu.cn

&Sureg

512021310186@live.sufe.edu.cn

&Qun Hu

512021310186@live.sufe.edu.cn

&Sureg

512021310186@live.sufe.edu.cn

&Qun Hu

512021310186@live.sufe.edu.cn

&Sureg

512021310186@live.sufe.edu.cn

&Qun Hu

512021310186@live.sufe.edu.cn

&Sureg

512021310186@live.sufe.edu.cn

&Qun Hu

512021310186@live.sufe.edu.cn

&Sureg

512021310186@live.sufe.edu.cn

&Qun Hu

512021310186@live.sufe.edu.cn

&Sureg

512021310186@live.sufe.edu.cn

&Qun Hu

512021310186@live.sufe.edu.cn

&Sureg

512021310186@live.sufe.edu.cn

&Qun Hu

512021310186@live.sufe.edu.cn

&Sureg

512021310186@live.sufe.edu.cn

&Qun Hu

512021310186@live.sufe.edu.cn

&Sureg

512021310186@live.sufe.edu.cn

&Qun Hu

512021310186@live.sufe.edu.cn

&Sureg

512021310186@live.sufe.edu.cn

&Qun Hu

512021310186@live.sufe.edu.cn

&Sureg

512021310186@live.sufe.edu.cn

&Qun Hu

512021310186@live.sufe.edu.cn

&Sureg

512021310186@live.sufe.edu.cn

&Qun Hu

512021310186@live.sufe.edu.cn

&Sureg

512021310186@live.sufe.edu.cn

&Qun Hu

512021310186@live.sufe.edu.cn

&Sureg

512021310186@live.sufe.edu.cn

&Qun Hu

512021310186@live.sufe.edu.cn

&Sureg

512021310186@live.sufe.edu.cn

&Qun Hu

512021310186@live.sufe.edu.cn

&Sureg

512021310186@live.sufe.edu.cn

&Qun Hu

512021310186@live.sufe.edu.cn

&Sureg

512021310186@live.sufe.edu.cn

&Qun Hu

512021310186@live.sufe.edu.cn

&Sureg

512021310186@live.sufe.edu.cn

&Qun Hu

512021310186@live.sufe.edu.cn

&Sureg

512021310186@live.sufe.edu.cn

&Qun Hu

512021310186@live.sufe.edu.cn

&Sureg

512021310186@live.sufe.edu.cn

&Qun Hu

512021310186@live.sufe.edu.cn

&Sureg

512021310186@live.sufe.edu.cn

&Qun Hu

512021310186@live.sufe.edu.cn

&Sureg

512021310186@live.sufe.edu.cn

&Qun Hu

512021310186@live.sufe.edu.cn

&Sureg

512021310186@live.sufe.edu.cn

&Qun Hu

512021310186@live.sufe.edu.cn

&Sureg

512021310186@live.sufe.edu.cn

&Qun Hu

512021310186@live.sufe.edu.cn

&Qun Hu

512021310186@live.sufe.edu.cn

&Sureg

512021310186@live.sufe.edu.cn

&Qun Hu

512021310186@live.sufe.edu.cn

&Sureg

512021310186@live.sufe.edu.cn

&Qun Hu

512021310186@live.sufe.edu.cn

&Sureg

512021310186@live.sufe.edu.cn

&Qun Hu

512021310186@live.sufe.edu.cn

&Qun Hu

512021310186@live.sufe.edu.cn

&Qun Hu

512021310186@live.sufe.edu.cn

&Sureg

512021310186@live.sufe.edu.cn

&Qun Hu

512021310186@live.sufe.edu.cn

&Qun Hu

512021310186@live.sufe.edu.cn

&Sureg

512021310186@live.sufe.edu.cn

&Qun Hu

512021310186@live.sufe.edu.cn

&Qun Hu

512021310186@live.sufe.edu.cn

&Qun Hu

512021310186@live.sufe.edu.cn

&Qun Hu

512021310186@live.sufe.edu.cn

&Qun Hu

512021310186@live.sufe.edu.cn

&Qun Hu

512021310186@live.sufe.edu.cn

&Qun Hu

512021310186@live.sufe.edu.cn

&Qun Hu

512021310186@live.sufe.edu.cn

&Qun Hu

512021310186@live.sufe.edu.cn

&Qun Hu

512021310186@live.sufe.edu.cn

&Qun Hu

512021310186@live.sufe.edu.cn

&Qun Hu

512021310186@live.sufe.edu.cn

&Qun Hu

512021310186@live.sufe.edu.cn

&Qun Hu

512021310186@live.sufe.edu.cn

&Qun Hu

512021310186@live.sufe.edu.cn

&Qun Hu

512021310186@live.sufe.edu.cn

&Qun Hu

512021310186@live.sufe.edu.cn

&Qun Hu

512021310186@live.sufe.edu.cn

&Qun Hu

512021310186@live.sufe.edu.cn

&Qun Hu

512021310186@live.sufe.edu.cn

&Qun Hu

512021310186@live.sufe.edu.cn

&Qun Hu

512021310186@live.sufe.edu.cn

&Qun Hu

512021310186@live.sufe.edu.cn

&Qun Hu

512021310186@live.sufe.edu.cn

&Qun Hu

512021310186@live.sufe.edu.cn

&Qun Hu

512021310186@live.sufe.edu.cn

&Qun Hu

512021310186@live.sufe.edu.cn

&Qun Hu

512021310186@live.sufe.edu.cn

&Qun Hu

512021310186@live.sufe.edu.cn

&Qun Hu

512021310186@live.sufe.edu.cn

&Qun Hu

512021310186@live.sufe.edu.cn

&Qun Hu

512021310186@live.sufe.edu.cn

&Qun Hu

512021310186@live.sufe.edu.cn

&Qun Hu

512021310186@live.sufe.edu.cn

&Qun Hu

512021310186@live.sufe.edu.cn

&Qun Hu

512021310186@live.sufe.edu.cn

&Qun Hu

512021310186@live.sufe.edu.cn

&Qun Hu

512021310186@live.sufe.edu.cn

&Qun Hu

512021310186@live.sufe.edu.cn

&Qun Hu

512021310186@live.sufe.edu.cn

&Qun Hu

512021310186@live.sufe.edu.cn

&Qun Hu

512021310186@live.sufe.edu.cn

&Qun Hu

512021310186@live.sufe.edu.cn

&Qun Hu

512021310186@live.sufe.edu.cn

&Qun Hu

512021310186@live.sufe.edu.cn

&Qun Hu

512021310186@live.sufe.edu.cn

&Qun Hu

512021310186@live.sufe.edu.cn

&Qun Hu

512021310186@live.sufe.edu.cn

&Qun Hu

512021310186@live.sufe.edu.cn

&Qun Hu

512021310186@live.sufe.edu.cn

&Qun Hu

512021310186@live.sufe.edu.cn

&Qun Hu

512021310186@live.sufe.edu.cn

&Qun Hu

512021310186@live.sufe.edu.cn

&Qun Hu

512021310186@live.sufe.edu.cn

&Qun Hu

512021310186@live.sufe.edu.cn

&Qun Hu

512021310186@live.sufe.edu.cn

&Qun Hu

512021310186@live.

where \(\mathcal{X}\subset\mathbb{R}^{n}\) and \(\mathcal{Y}\subset\mathbb{R}^{m}\) are convex sets, and \(A\) is a \(n\times m\) payoff matrix. The above captures two-player zero-sum games in which \(x^{\top}Ay\) is interpreted as the payment of the "min player" \(x\) to the "max player" \(y\). If \(\mathcal{X}=\mathbb{R}^{n}\) and \(\mathcal{Y}=\mathbb{R}^{m}\) the setting is called _unconstrained_, otherwise it is _constrained_. Soon after the minimax theorem of Von Neumann was established (for compact \(\mathcal{X},\mathcal{Y}\)), learning dynamics such as fictitious play (Brown [1951]) were proposed for solving min-max optimization problems. Blackwell's approachability theorem (Blackwell [1956]) further propelled the field of online learning, which lead to the discovery of several learning algorithms; such learning methods include multiplicative-weights-update method, online gradient descent/ascent and their optimistic variants and extra-gradient methods.

Last Iterate Convergence.There have been a vast literature on whether or not the aforementioned dynamics converge in an average sense or exhibit last-iterate convergence when applied to zero-sum games. Dating back to Nesterov [2005], there have been quite a few results showing that online learning algorithms have last-iterate convergence to Nash equilibria in zero-sum games. Examples include optimistic multiplicative weights update (Daskalakis and Panageas [2019], Wei et al. [2021]), optimistic gradient descent ascent (OGDA) (Daskalakis et al. [2018], Liang and Stokes [2019a]) (applied even to GANs) for unconstrained zero-sum games, OGDA for constrained zero-sum games (Wei et al. [2021], Cai et al. [2022], Gorbunov et al. [2022b]) and extra-gradient methods (Mertikopoulos et al. [2019], Cai et al. [2022], Gorbunov et al. [2022a]) using various techniques, including sum of squares.

Nevertheless, all aforementioned results assume that the underlying repeated zero-sum game remains invariant throughout the learning process. In many learning environments that assumption is unrealistic, see Duvocelle et al. [2022], Mai et al. [2018], Cardoso et al. [2019]) and references therein. One more realistic learning setting is where the underlying game is actually changing; this game is called time-varying. There have been quite a few works that deal with time-varying games, where they aim at analyzing the duality gap or dynamic regret (Zhang et al. [2022]) and references therein for OGDA and variants. However, in all these prior works, last-iterate convergence has not been investigated; the main purpose of this paper is to fill in this gap. We aim at addressing the following question:

_Will learning algorithms such as optimistic gradient descent ascent or extra-gradient exhibit last-iterate converge in time-varying zero-sum games?_

Our contributions.We consider unconstrained two-player zero-sum games with a time-varying payoff matrix (that is the payoff matrix \(A_{t}\) depends on time \(t\))

\[\min_{x\in\mathbb{R}^{n}}\max_{y\in\mathbb{R}^{m}}x^{\top}A_{t}y,\] (Time-varying zero sum game)

in which the payoff matrix \(A_{t}\) varies with time in the following two ways:

* **Periodic games:**\(A_{t}\) is a periodic function with period \(T\), i.e., \(A_{t+T}=A_{t}\).
* **Convergent perturbed games:**\(A_{t}=A+B_{t}\), \(\lim_{t\rightarrow\infty}B_{t}=0\).

In a repeated time-varying zero-sum game, players choose their learning algorithms and repeatedly play the zero-sum game. In the \(t\)-th round, when the players use strategy \((x_{t},y_{t})\), they receive their payoff \(-A_{t}^{\top}x_{t}\) and \(A_{t}y_{t}\).

In this paper we show the following results:

* For **periodic games:** We prove that when two players use extra-gradient, their strategies will converge to the common Nash equilibrium of the games within a period with an exponential rate, see Theorem 3.1 for details. Additionally, we provide an example where optimistic gradient descent ascent and negative momentum method diverge from the equilibrium with an exponential rate, see Theorem 3.2

To the best of our knowledge, this is the first result that provides a clear separation between the behavior of extra-gradient methods and optimistic gradient descent ascent.
* For **convergent perturbed games:** Assuming \(\sum_{t=1}^{\infty}\|B_{t}\|_{2}\) is bounded, we prove that the extra-gradient, optimistic gradient descent ascent, and negative momentum method all converge to the Nash equilibrium of the game defined by payoff matrix \(A\) with a rate determined by \(\{B_{t}\}_{t}\) and singular values of \(A\), see Theorem 3.3 Furthermore, we prove that extra-gradient will asymptotically converge to equilibrium without any additional assumptions on perturbations besides \(\lim_{t\rightarrow\infty}B_{t}=0\), see Theorem 3.4Related work on time-varying games.The closest work to ours that argues about stabilization of mirror descent type dynamics on convergent strongly monotone games is [20]. Most of the literature has been focused on proving either recurrence/oscillating behavior of learning dynamics in time-varying periodic games [17, 18, 19] and references therein or performing regret analysis [13, 14, 15]. In particular, the latter work extends results on RVU [16] bounds to argue about dynamic regret [21].

Technical Comparison.We investigate the last iterate behaviors of learning dynamics through their formulation of linear difference systems. This approach has also been used to establish last iterate convergence results for learning dynamics in time-independent games [19, 18, 17]. One common key point of these works is to prove that a certain matrix has no eigenvalues with modulus larger than \(1\), then the last iterate convergence can be guaranteed by the general results of autonomous difference systems. However, this method cannot be generalized to the time-varying games where the corresponding difference systems are non-autonomous. In particular, the dynamical behavior of a non-autonomous system is not determined by the eigenvalues of a single matrix. In fact, it is difficult to establish convergence/divergence results even for non-autonomous system with special structures, such as periodic or perturbed systems. In this paper, to get such results, we employ both general results in linear difference systems, such as Floquet theorem and Gronwall inequality, and special structures of the difference systems associated to learning dynamics.

Organization.In Section2 we present the necessary background for this work. The main results are stated in Section3. In Section4 we present numerical experiments and in Section5 we conclude with a discussion and propose some future research problems.

## 2 Preliminaries

### Definitions

Zero-sum bilinear game.An unconstrained two players zero-sum game consists of two agents \(\mathcal{N}=\{1,2\}\), and losses of both players are determined via payoff matrix \(A\in\mathbb{R}^{n\times m}\). Given that player \(1\) selects strategy \(x\in\mathbb{R}^{n}\) and player \(2\) selects strategy \(y\in\mathbb{R}^{m}\), player \(1\) receives loss \(u_{1}(x,y)=\langle x,Ay\rangle\), and player \(2\) receives loss \(u_{2}(x,y)=-\langle y,A^{\top}x\rangle\). Naturally, players want to minimize their loss resulting the following min-max problem:

\[\min_{x\in\mathbb{R}^{n}}\max_{y\in\mathbb{R}^{m}}x^{\top}Ay\] (Zero-Sum Game)

Note that the set \(\{(x^{*},y^{*})|A^{\top}x^{*}=0,Ay^{*}=0\}\) represents the set of equilibrium of the game.

Time-varying zero-sum bilinear game.In this paper, we study games in which the payoff matrices vary over time and we define two kinds of such time-varying games.

**Definition 2.1** (Periodic games).: _A periodic game with period \(T\) is an infinite sequence of zero-sum bilinear games \(\{A_{t}\}_{t=0}^{\infty}\subset\mathbb{R}^{n\times m}\), and \(A_{t+T}=A_{t}\) for all \(t\geq 0\)._

Note that the periodic game defined here is the same as Definition 1 in [18] except for the fact that we are considering a discrete time setting. Therefore, we do not make the assumption that payoff entries are smoothly dependent on \(t\).

**Definition 2.2** (Convergent perturbed games).: _A convergent perturbed game is an infinite sequence of zero-sum bilinear games \(\{A_{t}\}_{t=0}^{\infty}\subset\mathbb{R}^{n\times m}\), and \(\lim_{t\rightarrow\infty}A_{t}=A\) for some \(A\in\mathbb{R}^{n\times m}\). Equivalently, write \(A_{t}=A+B_{t}\), then \(\lim_{t\rightarrow\infty}B_{t}=0\). We will refer to the zero-sum bilinear game defined by \(A\) as **stable game**._

Learning dynamics in games.In this paper, we consider three kinds of learning dynamics : optimistic gradient descent ascent (OGDA), extra-gradient (EG), and negative momentum method. All these methods possess the last-iterate convergence property in repeated game with a time-independent payoff matrix, as demonstrated in previous literature. However, here we state their forms within a time-varying context.

**Optimistic gradient descent-ascent.** We study the optimistic descent ascent method (OGDA) defined as follows:

\[x_{t+1} =x_{t}-2\eta A_{t}y_{t}+\eta A_{t-1}y_{t-1},\] (OGDA) \[y_{t+1} =y_{t}+2\eta A_{t}^{\top}x_{t}-\eta A_{t-1}^{\top}x_{t-1}.\]

Optimistic gradient descent ascent method was proposed in (Popov[1980]), and here we choose the same parameters as (Daskalakis et al.[2018]). The last iterate convergence property of OGDA in unconstrained bilinear game with a time-independent payoff was proved in (Daskalakis et al.[2018]). Recently, there are also works analyzing the regret behaviors of OGDA under a time varying setting (Zhang et al.[2022], (Anagnostes et al.[2023])).

**Extra gradient.** We study the extra gradient descent ascent method (EG) defined as follows:

\[x_{t+\frac{1}{2}} =x_{t}-\gamma A_{t}y_{t},\;\;y_{t+\frac{1}{2}}=y_{t}+\gamma A_{t} ^{\top}x_{t},\] (EG) \[x_{t+1} =x_{t}-\alpha A_{t}y_{t+\frac{1}{2}},\;\;y_{t+1}=y_{t}+\alpha A_{ t}^{\top}x_{t+\frac{1}{2}}.\]

Note that the extra-gradient method first calculates an intermediate state before proceeding to the next state. Extra-gradient was firstly proposed in (Korpelevich[1976]) with the restriction that \(\alpha=\gamma\). Here we choose the parameters same as in (Liang and Stokes[2019]), where the linear convergence rate of extra-gradient in the bilinear zero-sum game with time-independent was also proven. Convergence of extra-gradient on convex-concave game was analyzed in (Nemirovski[2004], Monteiro and Svaited[2010]), and convergence guarantees for special non-convex-non-concave game was provided in (Mertikopoulos et al.[2019]).

**Negative momentum method.** We study the alternating negative momentum method (NM), defined as follows:

\[x_{t+1} =x_{t}-\eta A_{t}y_{t}+\beta_{1}(x_{t}-x_{t-1}),\] (NM) \[y_{t+1} =y_{t}+\eta A_{t+1}^{\top}x_{t+1}+\beta_{2}(y_{t}-y_{t-1}),\]

where \(\beta_{1},\beta_{2}\leq 0\) are the momentum parameters.

Applications of negative momentum method in game optimization was firstly proposed in (Cidel et al.[2019]). Note that the algorithm has an alternating implementation: the update rule of \(y_{t+1}\) uses the payoff \(A_{t+1}^{\top}x_{t+1}\), thus in each round, the second player chooses his strategy after the first player has chosen his. It was shown in (Cidel et al.[2019]) that both the negative momentum parameters and alternating implementations are crucial for convergence in bilinear zero-sum games with time-independent payoff matrices. Analysis of the convergence rate of negative momentum method in strongly-convex strongly-concave games was provided in (Zhang and Wang[2021]).

### Difference systems

The analysis of the last-iterate behavior of learning algorithms can be reduced to analyzing the dynamical behavior of the associated linear difference systems (Zhang and Yu[2020], Daskalakis and Panageas[2018]). When the payoff matrix is time-independent, the associated difference systems is autonomous. However, as we are studying games with time-varying payoff matrices, we have to deal with non-autonomous difference systems. In general, the convergence behavior of non-autonomous difference systems is much more complicated than that of autonomous ones (Colonius and Kliemann[2014]).

**Linear difference system.** Given a sequence of iterate matrices \(\{\mathcal{A}_{t}\}_{t=1}^{\infty}\subset\mathbb{R}^{n\times n}\) and initial condition \(X_{0}\in\mathbb{R}^{n}\), a linear difference system has form

\[X_{t+1}=\mathcal{A}_{t}X_{t}.\] (Linear difference system)

If \(\mathcal{A}_{t}\equiv\mathcal{A}\) is a matrix independent of time, the system is called an **autonomous system**, otherwise, it is called a **non-autonomous system**. We care about the asymptotic behavior of \(X_{t}\), that is, what can we say about \(X_{t}\) as \(t\to\infty\).

**Definition 2.3**.: _A point \(X\) is called **asymptotically stable** under the linear difference system if_

\[\exists\ \delta>0,\forall\ Y,\ s.t.\ \|Y-X\|_{2}\leq\delta\Rightarrow\lim_{t\to \infty}\|(\prod_{t=1}^{\infty}\mathcal{A}_{t})Y-X\|_{2}=0.\]

_Moreover, \(X\) is called **exponentially asymptotically stable** if the above limit has an exponentially convergence rate, i.e., \(\exists\ \alpha\in(0,1)\), such that_

\[\|(\prod_{t=1}^{s}\mathcal{A}_{t})Y-X\|_{2}\leq\alpha^{s}\|Y-X\|_{2}.\]

It is well known that for autonomous linear systems, being asymptotically stable is equivalent to being exponentially asymptotically stable, as shown in Thm 1.5.11 in [Colonius and KliemannColonius and Kliemann, 2014]. However, this equivalence does not hold for non-autonomous systems.

Linear difference system has a formal solution \(X_{T+1}=\prod_{t=0}^{T}\mathcal{A}_{t}X_{0}\). However, such a representation does not yield much information about the asymptotic behavior of solution as \(t\to\infty\), except in the case of autonomous system. In this paper, we mainly consider two classes of non-autonomous linear difference systems: if the iterate matrix \(\mathcal{A}_{t}\) is a periodic function of \(t\), the system is called a periodic system; and if \(\mathcal{A}_{t}\) has a limit as \(t\) tends to infinity, the system is called a perturbed system.

**Periodic linear system.** If the iterate matrix \(\mathcal{A}_{t}\) in a linear difference system satisfies \(\mathcal{A}_{t}=\mathcal{A}_{t+T}\), \(\forall t\in\mathbb{Z}\), then the system is called a periodic linear system. Denote \(\widetilde{\mathcal{A}}=\prod_{j=1}^{T}\mathcal{A}_{T-j}\). For a \(T\)-periodic equation, the eigenvalues \(\alpha\in\mathbb{C}\) of \(\widetilde{\mathcal{A}}\) is called the Floquet multipliers, and the Floquet exponents are defined by \(\lambda_{j}=\frac{1}{T}\ln(|\alpha_{j}|)\). The following Floquet theorem characterizes the stability of a periodic linear difference equation.

**Proposition 2.4** (Floquet theorem, [Colonius and KliemannColonius and Kliemann, 2014]).: _The zero solution of a periodic linear difference equation is asymptotically stable if and only if all Floquet exponents are negative._

Although Floquet theorem provides a method to determine whether a periodic system converges in general, considerable further work may be necessary in order to obtain explicit convergence criteria for specific equation. That is because even if we know the modulus of the largest eigenvalue of each \(\mathcal{A}_{t}\), it is usually difficult to compute the the modulus of the largest eigenvalue of each \(\prod_{t=0}^{T}\mathcal{A}_{t}\) due to the complex behavior of eigenvalues under matrix multiplication.

**Perturbed linear system.** If the iterative matrix \(\mathcal{A}_{t}\) in a linear difference system satisfies \(\mathcal{A}_{t}=\mathcal{A}+\mathcal{B}_{t}\) and \(\lim_{t\to\infty}\mathcal{B}_{t}=0\), then the system is called a perturbed linear system. The convergence behavior of a perturbed linear system is not clear in the literature. A general result in this direction is the following Perron's theorem:

**Theorem 2.5** (Perron's theorem, [PitukPituk2002]).: _If \(X_{n}\) is a solution of a perturbed linear system, then either \(X_{n}=0\) for all sufficient large \(n\) or \(\rho=\lim_{n\to\infty}\sqrt[n]{\|X_{n}\|_{2}}\) exists and is equal to the modulus of one of the eigenvalues of matrix \(\mathcal{A}\)._

This result can only guarantee the convergence of \(X_{n}\) when all eigenvalues of \(\mathcal{A}\) have modulus smaller than 1. In this case, \(\lim_{n\to\infty}X_{n}=0\). However, for analyzing the non-autonomous linear systems associated to learning dynamics, it is not sufficient as we will show that the stablized matrix of these systems generally has eigenvalues equal to \(1\).

## 3 Main results

In this section, we present our main results. We present the relationship between learning dynamics and linear difference equation in Section5.1 investigate the last-iterate behaviors of learning dynamics in a periodic game in Section5.2 and investigate the last-iterate behaviors of learning dynamics in a convergent perturbed game in Section5.3 Proofs are deferred to the appendix.

### Learning dynamics as linear difference systems

Formalizing learning dynamics as linear difference systems is useful for studying their dynamical behaviors. In the following, we present the formulation of optimistic gradient descent ascent, extra-gradient, and negative momentum method as linear difference systems.

**Proposition 3.1**.: _Optimistic gradient descent ascent can be written as the following linear difference system:_

\[\begin{bmatrix}x_{t+1}\\ y_{t+1}\\ x_{t}\\ y_{t}\end{bmatrix}=\begin{bmatrix}I&-2\eta A_{t}&0&\eta A_{t-1}\\ 2\eta A_{t}^{\top}&I&-\eta A_{t-1}^{\top}&0\\ I&0&0&0\\ 0&I&0&0\end{bmatrix}\begin{bmatrix}x_{t}\\ y_{t}\\ x_{t-1}\\ y_{t-1}\end{bmatrix}. \tag{2}\]

_Extra-gradient can be written as the following linear difference system :_

\[\begin{bmatrix}x_{t+1}\\ y_{t+1}\end{bmatrix}=\begin{bmatrix}I-\alpha\gamma A_{t}A_{t}^{\top}&-\alpha A_ {t}\\ \\ \alpha A_{t}^{\top}&I-\gamma\alpha A_{t}^{\top}A_{t}\end{bmatrix}\begin{bmatrix} x_{t}\\ y_{t}\end{bmatrix}. \tag{3}\]

_Negative momentum method can be written as the following linear difference system:_

\[\begin{bmatrix}x_{t+1}\\ y_{t+1}\\ x_{t}\\ y_{t}\end{bmatrix}=\begin{bmatrix}(1+\beta_{1})I&-\eta A_{t}&-\beta_{1}I&0\\ \\ \eta(1+\beta_{1})A_{t+1}^{\top}&(1+\beta_{2})I-\eta^{2}A_{t+1}^{\top}A_{t}&- \eta\beta_{1}A_{t+1}^{\top}&-\beta_{2}I\\ \\ I&0&0&0\\ \\ 0&I&0&0\end{bmatrix}\begin{bmatrix}x_{t}\\ y_{t}\\ x_{t-1}\\ y_{t-1}\end{bmatrix}. \tag{4}\]

It is easy to verify that these linear difference systems are equivalent to their corresponding learning dynamics by directly writing down the matrix-vector product.

We will also refer to the iterative matrix of these linear difference systems as the **iterative matrix of their corresponding learning dynamics**. In the following, we will study the convergence/divergence behaviors of these linear difference systems under the condition that \(\{A_{t}\}_{t}\) is a periodic game or convergent perturbed game. Note that although an intermediate step \((x_{t+\frac{1}{2}},y_{t+\frac{1}{2}})\) is required in extra-gradient, it is eliminated in 3.

### Periodic games

Recall that in a periodic game with period \(\mathcal{T}\), the payoff matrices \(\{A_{s}\}_{s=1}^{\infty}\) satisfy \(A_{s+\mathcal{T}}=A_{s}\) for any \(s>0\). Define

\[\Delta_{i,t}=\|A_{i}^{\top}x_{t}\|_{2}+\|A_{iy}t\|_{2} \tag{5}\]

for \(i\in[\mathcal{T}]\). As in [4], we use \(\Delta_{i,t}\) as a measurement of the distance between the current strategy \((x_{t},y_{t})\) and a Nash equilibrium of the zero-sum bilinear game defined by the kernel space of payoff matrix \(A_{i}\). Note that if \((x^{*},y^{*})\) is a Nash equilibrium of the game defined by \(A_{i}\), then \((A_{i}^{\top}x^{*},A_{i}y^{*})=(0,0)\), and

\[\Delta_{i,t}=\|A_{i}^{\top}(x_{t}-x^{*})\|_{2}+\|A_{i}(y_{t}-y^{*})\|_{2}\]

thus when strategy is close to equilibrium, \(\Delta_{i,t}\) will be small. Moreover, \(\Delta_{i,t}=0\) if and only if \((x_{t},y_{t})\) is an equilibrium. In this section, we will consider the convergence/growth rate of \(\Delta_{i,t}\) at a function of \(t\).

We firstly consider Extra-gradient method. Denote the iterative matrix in the linear difference form of Extra-gradient 3 as \(\mathcal{A}_{t}\). In the following theorem, we prove that if two players use the Extra-gradient method, their strategies will converge to the common Nash equilibrium of games in the period with an exponential rate.

**Theorem 3.1**.: _When two players use extra-gradient in a periodic games with period \(T\), with step size \(\alpha=\gamma<\frac{1}{\sigma}\) where \(\sigma=\max\{\sigma^{\prime}|\sigma^{\prime}\) is a singular value of \(A_{i}\) for some \(i\in[\mathcal{T}]\}\). Then_

\[\Delta_{i,t}\in\mathcal{O}\left((\lambda_{*})^{t/\mathcal{T}}\cdot\text{Poly} (t)\right),\ \forall i\in[\mathcal{T}]\]

_where \(\lambda_{*}=\max\{\ |\lambda|\ |\ \lambda\ \text{is an eigenvalue of}\ \ \left(\prod_{t=1}^{\mathcal{T}}\mathcal{A}_{t}\right),\lambda\neq 1\}\), and \(\lambda_{*}<1\)._

Note that in a periodic game, the iterative matrices of learning dynamics are also periodic, which means that the learning difference systems of these learning dynamics are periodic systems. According to Floquet theorem, see proposition 2.4 the study of dynamical behaviors of a periodic system can be reduced to the autonomous system whose asymptotic behavior is determined by \(\prod_{t=1}^{\mathcal{T}}\mathcal{A}_{t}\).

The key point on the proof of Theorem 3.1 is an observation that the iterative matrix of extra-gradient is a normal matrix, which makes it possible to calculate the Jordan normal form of \(\prod_{t=1}^{\mathcal{T}}\mathcal{A}_{t}\) for arbitrary large \(\mathcal{T}\). The details of proof are left to Appendix B

In the following theorem, we provide an example demonstrating that when two players use the optimistic gradient descent ascent or negative momentum method, their strategy will diverge at an exponential rate, regardless of how they choose their step-sizes and momentum parameters.

**Theorem 3.2**.: _Consider a periodic game with period \(\mathcal{T}=2\), and described by the following payoff matrix_

\[A_{t}=\left\{\begin{array}{ll}\left[1,-1\right],&t\;\;\text{is}\;\;\text{odd }\\ \left[-1,1\right],&t\;\;\text{is}\;\;\text{even}\end{array}\right. \tag{6}\]

_with \(x_{t}\in\mathbb{R},\;y_{t}\in\mathbb{R}^{2}\). If two players use optimistic gradient descent ascent or negative momentum method, then regardless of how they choose step sizes and momentum parameters, we have_

\[\sup_{s\in[t]}\Delta_{i,s}\in\Omega(\lambda^{t}),\;\text{where}\;\lambda>1,\; i\in\{1,2\}.\]

_Here \(\lambda\) is determined by the largest modulus of the eigenvalues of the iterative matrix of optimistic gradient descent ascent or negative momentum method._

To prove theorem 3.2 we directly calculate the characteristic polynomials of the iterative matrices products in one period for optimistic gradient descent ascent and negative momentum method under the game defined by \(\textcircled{\raisebox{-0.86pt}{\rule{0.86pt}{0.86pt}}}\). To show that these systems have an exponential divergence rate, it is sufficient to demonstrate that their characteristic polynomials have a root with modulus larger than \(1\). We achieve this by using the Schur stable theorem, which is also employed to demonstrate the last iterate convergence of several learning dynamics in time-independent game (Zhang and Yu [2020]). The proof is deferred to Appendix C

In Figure 1, we present the function curves of \(\Delta_{1,t}\) for these three types of game dynamics under the periodic game defined by \(\textcircled{\raisebox{-0.86pt}{\rule{0.86pt}{0.86pt}}}\). From the experimental results, extra-gradient converges, while both optimistic gradient descent ascent and negative momentum method diverge.

### Convergent perturbed game

Recall that the payoff matrix of a convergent perturbed game has form \(A_{t}=A+B_{t}\), where \(\lim_{t\rightarrow\infty}B_{t}=0\), and we refer to the zero-sum game defined by payoff matrix \(A\) as the stable game. We denote

\[\Delta_{t}=\|A^{\top}x_{t}\|_{2}+\|Ay_{t}\|_{2}, \tag{7}\]

thus \(\Delta_{t}\) measures the distance between the strategy \((x_{t},y_{t})\) in the \(t\)-th round and the Nash equilibrium of the stable game defined by \(A\). Moreover, \(\Delta_{t}=0\) if and only if \((x_{t},y_{t})\) is an equilibrium of the stable game.

Figure 1: Function curves of \(\Delta_{1,t}\) of the game presented in Theorem 3.2 Extra-gradient converges, while the other two methods diverge.

In the literature on linear difference systems, a common assumption that needs to be added for convergence guarantee is the following bounded accumulated perturbations (BAP) assumption (Benzaid and Lutz [1987], Elaydi et al. [1999], Elaydi and Gyori [1995]):

\[\sum_{t=0}^{\infty}\lVert B_{t}\rVert_{2}\;\text{is bounded}.\] (BAP assumption)

In the following theorem, we prove that under BAP assumption, all three learning dynamics considered in this paper will make \(\Delta_{t}\) converge to \(0\), with a rate dependent on the vanishing rate of \(B_{t}\).

**Theorem 3.3**.: _Assume that the BAP assumption holds, i.e., \(\sum_{t=0}^{\infty}\lVert B_{t}\rVert_{2}\) is bounded, and let \(\sigma\) be the maximum modulus of the singular value of payoff matrix \(A\), then with parameters choice:_

* _for extra-gradient with step size_ \(\alpha=\eta<\frac{1}{2\sigma}\)_,_
* _for optimistic gradient descent ascent with step size_ \(\eta<\frac{1}{2\sigma}\)_,_
* _for negative momentum method with step size_ \(\eta<\frac{1}{\sigma}\) _and momentum parameters_ \(\beta_{1}=-\frac{1}{2}\) _and_ \(\beta_{2}=0\)_,_

_we have \(\Delta_{t}\) converge to \(0\) with rate \(\mathcal{O}(f(t))\). Here_

\[f(t)=\max\{\lambda^{t},\sum_{i=t/2}^{\infty}\lVert B_{i}\rVert_{2}\},\]

_and \(\lambda\in(0,1)\) is determined by the eigenvalues of the iterative matrix of corresponding learning dynamics and the payoff matrix \(A\) of the stable game._

There are two main ingredients in the proof of Theorem 3.3 firstly, we show that the iterative matrices associated with these learning dynamics are diagonalizable; secondly, these matrices do not have eigenvalues with modulus larger than \(1\). Moreover, we prove a general results which states any linear difference system satisfying these two conditions will converge. The details of proof are left to Appendix E

**Remark 3.2**.: _The BAP assumption can be converted into a constraint on the vanishing rate of \(B_{t}\): if \(B_{t}\) has a vanishing rate like \(\mathcal{O}(\frac{1}{t^{1+\epsilon}})\), for some arbitrary small \(\epsilon>0\), then \(\sum_{t=0}^{\infty}\lVert B_{t}\rVert_{2}\) is bounded. We also note that the condition for \(B_{t}\) to vanish at a rate \(\mathcal{O}(\frac{1}{t^{1+\epsilon}}),\;\forall\epsilon>0\) is necessary to ensure convergence in general linear difference system. For example, consider the 1-dimension system \(x_{t}=(1+\frac{1}{t})x_{t-1}\) where a \(\mathcal{O}(\frac{1}{t})\) convergence rate of perturbations leads to \(x_{t}\) diverging with a \(\Omega(t)\) rate._

Surprisingly, in the next theorem, we prove that Extra-gradient makes \(\Delta_{t}\) asymptotically converge to \(0\), without making any further assumptions about the converge rate of \(B_{t}\).

**Theorem 3.4**.: _In a convergent perturbed game, if two players use Extra-gradient, there holds \(\lim_{t\rightarrow\infty}\Delta_{t}=0\) with step size \(\alpha=\eta<\frac{1}{2\sigma}\) where \(\sigma\) is the maximum modulus of the singular value of payoff matrix \(A\)._

To prove Theorem 3.4 we first observe that \(\lVert x_{t}\rVert_{2}+\lVert y_{t}\rVert_{2}\) a non-increasing function of \(t\) since the iterative matrix of extra-gradient is a normal matrix. Next, we prove that if Theorem 3.4 doesn't hold, then \(\lVert x_{t}\rVert_{2}+\lVert y_{t}\rVert_{2}\) will decrease by a constant for infinite number of times, thus leading to a contradiction with the non-increasing property and the non-negativity of \(\lVert x_{t}\rVert_{2}+\lVert y_{t}\rVert_{2}\). The proof is deferred to Appendix E

## 4 Experiments

In this section we present numerical results for our theoretical results in Section E

**Experiments on Theorem 3.1** We verify Theorem 3.1 through a period-\(3\) game, the payoff matrices are chosen to be

\[A_{1}=\begin{bmatrix}1&2\\ 2&4\end{bmatrix},A_{2}=\begin{bmatrix}3&7\\ 7&1\end{bmatrix},A_{3}=\begin{bmatrix}4&2\\ 4&2\end{bmatrix}.\]We run extra-gradient and optimistic gradient descent ascent on this example, both with step size = 0.01, the experimental results are presented in Figure 2. We can see extra-gradient (left) makes \(\Delta_{i,t}\) converge, while optimistic gradient descent ascent (right) makes \(\Delta_{i,t}\) diverge. This result supports Theorem 3.1 and provides a numerical example of the separation of extra-gradient and optimistic gradient descent ascent in periodic games.

**Experiments on Theorem 3.3** We verify Theorem 3.3 by examples:

\[A=\begin{bmatrix}2&3\\ 4&6\end{bmatrix},B_{1,t}=B\cdot t^{-1.1},B_{2,t}=B\cdot t^{-4},\;\text{and}\;,B _{3,t}=B\cdot t^{-8}\]

where \(B=[[-15,70],[-90,90]]\). The step size is chosen to be \(0.005\). The initial points are chosen to be \(x_{0}=[15,13],x_{-1}=[11,3]\) and \(y_{0}=[35,1],y_{-1}=[35,1]\). The experimental results are presented in Figure 3 all of the three dynamics make \(\Delta_{t}\) converge to \(0\), and slower convergence rate of perturbations can decelerate the convergence rate of learning dynamics, thus support the convergence result in Theorem 3.3. We also observe that the convergence rate is faster than the upper bound provided in the theorem. Therefore, we conjecture the bound of convergence rate can be improved.

**Experiments on Theorem 3.4** We verify Theorem 3.4 by two group of examples. The perturbations are :

\[B_{1,t}=B\cdot t^{-0.4},\;B_{2,t}=B\cdot t^{-0.3},\;B_{3,t}=B\cdot t^{-0.2},\]

and

\[B_{4,t}=B\cdot\log(t)^{-1.8},\;B_{5,t}=B\cdot\log(t)^{-1.5},\;B_{6,t}=B\cdot \log(t)^{-1.3}.\]

where \(B=[[-10,10],[-10,10]]\). The payoff matrix of stable game is chosen to be \(A=[[2,3],[4,6]]\). Note that the perturbations do not satisfy (BAP assumption). The experimental results are shown in Figure 4. We can see all these curves converge to \(0\), thus support the result in Theorem 3.4 Furthermore, we can observe that large perturbations may lead to more oscillations during the convergence processes, which in turn slows down the rate of convergence. We present more experiments in Appendix C

Figure 3: Values of \(\Delta_{t}\) for extra-gradient (left), optimistic gradient descent ascent (middle), negative momentum method (right).

Figure 2: Function curves of \(\Delta_{t}\) for extra-gradient (left), and optimistic gradient descent ascent (right).

## 5 Discussion

In this paper, we study the last-iterate behavior of extra-gradient, optimistic gradient descent ascent, and negative momentum method in two types of time-varying games : periodic game and convergent perturbed game. In the case of periodic game, we prove that extra-gradient will converge to a Nash equilibrium while other two methods diverge. To the best of our knowledge, this is the first result that provides a clear separation between the behavior of extra-gradient methods and optimistic gradient descent ascent. In the case of convergent perturbed game, we prove all three learning dynamics converge to Nash equilibrium under the BAP assumption, which is commonly used in the literature on dynamical systems.

Our results leave many interesting open questions. Firstly, is the BAP assumption necessary for ensuring convergence in optimistic gradient descent ascent and negative momentum method? Secondly, the bound of convergence rate in Theorem 3.3 may rather slight as we shown in experiments section. Obtaining a tighter bound on the convergence rate is an important future research problem. Thirdly, can the results here be generalized to other settings, such as constrained zero-sum game?

## Acknowledgement

Yi Feng is supported by the Fundamental Research Funds for the Central Universities. Ioannis Panageas wants to thank a startup grant. Xiao Wang acknowledges Grant 202110458 from SUFE and support from the Shanghai Research Center for Data Science and Decision Technology.

## References

* Anagnostides et al. (2023) Ioannis Anagnostides, Ioannis Panageas, Gabriele Farina, and Tuomas Sandholm. On the convergence of no-regret learning dynamics in time-varying games. _arXiv preprint arXiv:2301.11241_, 2023.
* Benzaid and Lutz (1987) Z Benzaid and DA Lutz. Asymptotic representation of solutions of perturbed systems of linear difference equations. _Studies in Applied Mathematics_, 77(3):195-221, 1987.
* Blackwell (1956) David Blackwell. An analog of the minimax theorem for vector payoffs. In _Pacific J. Math._, pages 1-8, 1956.
* Brown (1951) George W Brown. Iterative solution of games by fictitious play. _Act. Anal. Prod Allocation_, 13(1):374, 1951.
* Cai et al. (2022) Yang Cai, Argyris Oikonomou, and Weiqiang Zheng. Finite-time last-iterate convergence for learning in multi-player games. In _NeurIPS_, 2022.
* Cardoso et al. (2019) Adrian Rivera Cardoso, Jacob D. Abernethy, He Wang, and Huan Xu. Competing against nash equilibria in adversarially changing zero-sum games. In _Proceedings of the 36th International Conference on Machine Learning, ICML 2019_, volume 97 of _Proceedings of Machine Learning Research_, pages 921-930. PMLR, 2019.
* Colonius and Kliemann (2014) Fritz Colonius and Wolfgang Kliemann. _Dynamical systems and linear algebra_, volume 158. American Mathematical Society, 2014.
* Colonius et al. (2019)Constantinos Daskalakis and Ioannis Panageas. The limit points of (optimistic) gradient descent in min-max optimization. _Advances in neural information processing systems_, 31, 2018.
* Leibniz-Zentrum fur Informatik, 2019.
* Daskalakis et al. (2018) Constantinos Daskalakis, Andrew Ilyas, Vasilis Syrgkanis, and Haoyang Zeng. Training GANs with Optimism. In _Proceedings of ICLR_, 2018.
* Duvocelle et al. (2022) Benoit Duvocelle, Panayotis Mertikopoulos, Mathias Staudigl, and Dries Vermeulen. Multiagent online learning in time-varying games. _Mathematics of Operations Research_, 2022.
* Elaydi and Gyori (1995) S Elaydi and I Gyori. Asymptotic theory for delay difference equations. _Journal of Difference Equations and Applications_, 1(2):99-116, 1995.
* Elaydi et al. (1999) Saber Elaydi, Satoru Murakami, and Etsuyo Kamiyama. Asymptotic equivalence for difference equations with infinite delay. _Journal of Difference Equations and Applications_, 5(1):1-23, 1999.
* Fiez et al. (2021) Tanner Fiez, Ryann Sim, Stratis Skoulakis, Georgios Piliouras, and Lillian Ratliff. Online learning in periodic zero-sum games. _Advances in Neural Information Processing Systems_, 34, 2021.
* Gidel et al. (2019) Gauthier Gidel, Reyhane Askari Hemmat, Mohammad Pezeshki, Remi Le Priol, Gabriel Huang, Simon Lacoste-Julien, and Ioannis Mitliagkas. Negative momentum for improved game dynamics. In _The 22nd International Conference on Artificial Intelligence and Statistics_, pages 1802-1811. PMLR, 2019.
* Gorbunov et al. (2022a) Eduard Gorbunov, Nicolas Loizou, and Gauthier Gidel. Extragradient method: O(1/K) last-iterate convergence for monotone variational inequalities and connections with cocoercivity. In Gustau Camps-Valls, Francisco J. R. Ruiz, and Isabel Valera, editors, _International Conference on Artificial Intelligence and Statistics, AISTATS 2022, 28-30 March 2022, Virtual Event_, volume 151 of _Proceedings of Machine Learning Research_, pages 366-402. PMLR, 2022a.
* Gorbunov et al. (2022b) Eduard Gorbunov, Adrien Taylor, and Gauthier Gidel. Last-iterate convergence of optimistic gradient method for monotone variational inequalities. In _NeurIPS_, 2022b.
* Korpelevich (1976) Galina M Korpelevich. The extragradient method for finding saddle points and other problems. _Matecon_, 12:747-756, 1976.
* Liang and Stokes (2019) Tengyuan Liang and James Stokes. Interaction matters: A note on non-asymptotic local convergence of generative adversarial networks. _AISTATS_, 2019a.
* Liang and Stokes (2019) Tengyuan Liang and James Stokes. Interaction matters: A note on non-asymptotic local convergence of generative adversarial networks. In _The 22nd International Conference on Artificial Intelligence and Statistics_, pages 907-915. PMLR, 2019b.
* Mai et al. (2018) Tung Mai, Milena Mihail, Ioannis Panageas, Will Ratcliff, Vijay V. Vazirani, and Peter Yunker. Cycles in zero-sum differential games and biological diversity. In Eva Tardos, Edith Elkind, and Rakesh Vohra, editors, _Proceedings of the 2018 ACM Conference on Economics and Computation, Ithaca, NY, USA, June 18-22, 2018_, pages 339-350. ACM, 2018.
* Mertikopoulos et al. (2019) Panayotis Mertikopoulos, Bruno Lecouat, Houssam Zenati, Chuan-Sheng Foo, Vijay Chandrasekhar, and Georgios Piliouras. Optimistic mirror descent in saddle-point problems: Going the extra (gradient) mile. In _7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019_. OpenReview.net, 2019.
* Monteiro and Svaiter (2010) Renato DC Monteiro and Benar Fux Svaiter. On the complexity of the hybrid proximal extragradient method for the iterates and the ergodic mean. _SIAM Journal on Optimization_, 20(6):2755-2787, 2010.
* Nemirovski (2004) Arkadi Nemirovski. Prox-method with rate of convergence o (1/t) for variational inequalities with lipschitz continuous monotone operators and smooth convex-concave saddle point problems. _SIAM Journal on Optimization_, 15(1):229-251, 2004.
* Nemirovski et al. (2019)Yurii Nesterov. Smooth minimization of non-smooth functions. _Math. Program._, 103(1):127-152, 2005.
* Pituk (2002) Mihaly Pituk. More on poincare's and perron's theorems for difference equations. _The Journal of Difference Equations and Applications_, 8(3):201-216, 2002.
* Popov (1980) Leonid Denisovich Popov. A modification of the arrow-hurwicz method for search of saddle points. _Mathematical notes of the Academy of Sciences of the USSR_, 28:845-848, 1980.
* Roy et al. (2019) Abhishek Roy, Yifang Chen, Krishnakumar Balasubramanian, and Prasant Mohapatra. Online and bandit algorithms for nonstationary stochastic saddle-point optimization. _CoRR_, abs/1912.01698, 2019.
* Syrgkanis et al. (2015) Vasilis Syrgkanis, Alekh Agarwal, Haipeng Luo, and Robert E. Schapire. Fast convergence of regularized learning in games. In Corinna Cortes, Neil D. Lawrence, Daniel D. Lee, Masashi Sugiyama, and Roman Garnett, editors, _Advances in Neural Information Processing Systems 28: Annual Conference on Neural Information Processing Systems 2015, December 7-12, 2015, Montreal, Quebec, Canada_, pages 2989-2997, 2015.
* Wei et al. (2021) Chen-Yu Wei, Chung-Wei Lee, Mengxiao Zhang, and Haipeng Luo. Linear last-iterate convergence in constrained saddle-point optimization. In _9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021_. OpenReview.net, 2021.
* Zhang and Wang (2021) Guodong Zhang and Yuanhao Wang. On the suboptimality of negative momentum for minimax optimization. In _International Conference on Artificial Intelligence and Statistics_, pages 2098-2106. PMLR, 2021.
* Zhang and Yu (2020) Guojun Zhang and Yaoliang Yu. Convergence of gradient methods on bilinear zero-sum games. In _International Conference on Learning Representations_, 2020.
* Zhang et al. (2022) Mengxiao Zhang, Peng Zhao, Haipeng Luo, and Zhi-Hua Zhou. No-regret learning in time-varying zero-sum games. In _International Conference on Machine Learning, ICML 2022_, volume 162 of _Proceedings of Machine Learning Research_, pages 26772-26808. PMLR, 2022.
* Zinkevich (2003) Martin Zinkevich. Online convex programming and generalized infinitesimal gradient ascent. In Tom Fawcett and Nina Mishra, editors, _Machine Learning, Proceedings of the Twentieth International Conference (ICML 2003), August 21-24, 2003, Washington, DC, USA_, pages 928-936. AAAI Press, 2003.

Response:
## Review

### Summary

This paper investigates the last-iterate convergence of three algorithms (Optimistic Gradient Descent Ascent, Extra Gradient, and Negative Momentum Method) in two types of time-varying zero-sum games: periodic and convergent perturbed games. It demonstrates that the Extra Gradient algorithm converges in periodic games, while the other two algorithms may diverge. Additionally, it shows that all algorithms converge in convergent perturbed games. The paper includes thorough theoretical analysis and supporting experimental results, contributing to the understanding of learning dynamics in zero-sum games.

### Strengths

- The problem studied is natural and clearly explained, with strong results, including the interesting split in behavior between the algorithms.
- The paper is well-written, organized, and easy to follow.
- The authors provide a significant contribution to the understanding of last-iterate convergence in time-varying games.
- The analysis is sound, and the empirical results validate the theoretical insights.
- The work leverages technical innovations from the literature on non-autonomous linear difference systems.

### Weaknesses

- The classes of games studied could benefit from better motivation. There is a lack of clarity on their significance beyond technical convenience.
- Some results lack intuitive explanations, particularly regarding the distinct behaviors of the algorithms.
- Figures could be improved for clarity, as some are not well-plotted.
- The notion of equilibrium is inconsistently defined in the paper.
- The negative momentum method may not apply in strictly online settings where both players must act first.

### Questions

- What is the range of $t$ in Theorem 3.1, and can the authors clarify the implications of choosing $t=T$?
- Could the authors provide more intuitive explanations for the convergence properties of the different algorithms?
- What specific reasons justify focusing on periodic and convergent perturbed games?
- Do the results extend to more general cases of time-varying games?
- Can the authors discuss whether the optimistic gradient descent ascent and negative momentum methods converge in the same instances as shown in the experiments?

### Soundness

**Score:** 15

**Description:** 3 = good; The theoretical foundation is solid with clear proofs and sound analysis.

### Presentation

**Score:** 15

**Description:** 3 = good; The paper is well-organized and accessible, though it could benefit from improved clarity in figures and terminology.

### Contribution

**Score:** 14

**Description:** 3 = good; The work represents a meaningful advancement in the field, though there are suggestions for broader applications.

### Rating

**Score:** 6

**Description:** 6 = Weak Accept; The paper is technically solid with moderate-to-high impact, though it has minor issues that need addressing.

### Paper Decision

**Decision:** Accept (poster)

**Reasons:** The paper presents original research with significant implications for understanding learning dynamics in zero-sum games. Despite some minor weaknesses and questions regarding motivation and clarity, the overall soundness and contribution of the work outweigh these concerns, justifying an acceptance decision.