Question:
You are a highly experienced, conscientious, and fair academic reviewer, please help me review this paper. The review should be organized into nine sections:
1. Summary: A summary of the paper in 100-150 words.
2. Strengths/Weaknesses/Questions: The Strengths/Weaknesses/Questions of paper, which should be listed in bullet points, with each point supported by specific examples from the article where possible.
3. Soundness/Contribution/Presentation: Rate the paper's Soundness/Contribution/Presentation, and match this score to the corresponding description from the list below and provide the result. The possible scores and their descriptions are:
    1 poor
    2 fair
    3 good
    4 excellent
4. Rating: Give this paper an appropriate rating, match this rating to the corresponding description from the list below and provide the result. The possible Ratings and their descriptions are:
    1 strong reject
    2 reject, significant issues present
    3 reject, not good enough
    4 possibly reject, but has redeeming facets
    5 marginally below the acceptance threshold
    6 marginally above the acceptance threshold
    7 accept, but needs minor improvements
    8 accept, good paper
    9 strong accept, excellent work
    10 strong accept, should be highlighted at the conference
5. Paper Decision: It must include the Decision itself(Accept or Reject) and the reasons for this decision, based on the criteria of originality, methodological soundness, significance of results, and clarity and logic of presentation.

Here is the template for a review format, you must follow this format to output your review result:
**Summary:**
Summary content

**Strengths:**
- Strength 1
- Strength 2
- ...

**Weaknesses:**
- Weakness 1
- Weakness 2
- ...

**Questions:**
- Question 1
- Question 2
- ...

**Soundness:**
Soundness result

**Presentation:**
Presentation result

**Contribution:**
Contribution result

**Rating:**
Rating result

**Paper Decision:**
- Decision: Accept/Reject
- Reasons: reasons content

Please ensure your feedback is objective and constructive. The paper is as follows:

# Robust Second-Order Nonconvex Optimization and

Its Application to Low Rank Matrix Sensing

Shuyao Li

University of Wisconsin-Madison

shuyao.li@wisc.edu

&Yu Cheng

Brown University

yu_cheng@brown.edu

&Ilias Diakonikolas

University of Wisconsin-Madison

ilias@cs.wisc.edu

&Jelena Diakonikolas

University of Wisconsin-Madison

jelena@cs.wisc.edu

&Rong Ge

Duke University

rongge@cs.duke.edu

&Stephen Wright

University of Wisconsin-Madison

wright@cs.wisc.edu

###### Abstract

Finding an approximate second-order stationary point (SOSP) is a well-studied and fundamental problem in stochastic nonconvex optimization with many applications in machine learning. However, this problem is poorly understood in the presence of outliers, limiting the use of existing nonconvex algorithms in adversarial settings. In this paper, we study the problem of finding SOSPs in the strong contamination model, where a constant fraction of datapoints are arbitrarily corrupted. We introduce a general framework for efficiently finding an approximate SOSP with _dimension-independent_ accuracy guarantees, using \(\widetilde{O}(D^{2}/\epsilon)\) samples where \(D\) is the ambient dimension and \(\epsilon\) is the fraction of corrupted datapoints. As a concrete application of our framework, we apply it to the problem of low rank matrix sensing, developing efficient and provably robust algorithms that can tolerate corruptions in both the sensing matrices and the measurements. In addition, we establish a Statistical Query lower bound providing evidence that the quadratic dependence on \(D\) in the sample complexity is necessary for computationally efficient algorithms.

## 1 Introduction

Learning in the presence of corrupted data is a significant challenge in machine learning (ML) with many applications, including ML security [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 120, 121, 122, 123, 124, 125, 126, 127, 128, 129, 130, 131, 132, 133, 134, 135, 136, 137, 138, 139, 140, 141, 142, 143, 144, 145, 146, 147, 148, 149, 150, 151, 152, 153, 154, 155, 156, 157, 158, 159, 160, 161, 162, 163, 164, 165, 166, 167, 168, 169, 170, 171, 172, 173, 174, 175, 176, 177, 178, 179, 180, 181, 182, 183, 184, 185, 186, 187, 188, 189, 190, 191, 192, 193, 194, 195, 196, 197, 198, 199, 199, 198, 199, 199, 190, 191, 190, 192, 193, 195, 196, 197, 198, 199, 199, 199, 199, 198, 199, 190, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 1999, 199, 199, 1999, 199, 199, 1999, 199, 199, 199, 199, 199, 199, 199, 1999, 199, 1999, 199, 1999, 199, 1999, 199, 1999, 1999, 199, 1999, 1999, 199, 1999, 199, 199, 199, 199, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 199, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 19999, 1999, 1999, 1999, 1999, 19999, 1999, 19999, 1999, 19999, 1999, 19999, 19999, 1999, 1999, 19999, 1999, 1999, 19999, 1999, 1999, 1999, 1999, 19999, 1999, 1999, 19999, 1999, 19999, 1999, 1999, 1999, 1999, 1999, 1999, 19999, 1999, 1999, 1999, 1999, 19999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 19999, 1999, 1999, 1999, 1999, 19999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 19999, 1999, 1999, 1999, 19999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 19999, 1999, 1999, 1999,In this paper, we study the general problem of smooth (with Lipschitz gradient and Hessian) stochastic nonconvex optimization \(\min_{x}\tilde{f}(x)\) in the outlier-robust setting, where \(\tilde{f}(x):=\mathbb{E}_{A\sim\mathcal{G}}\,f(x,A)\) and \(\mathcal{G}\) is a possibly unknown distribution of the random parameter \(A\). We will focus on the following standard adversarial contamination model (see, e.g., [2]).

**Definition 1.1** (Strong Contamination Model).: Given a parameter \(0<\epsilon<1/2\) and an inlier distribution \(\mathcal{G}\), an algorithm receives samples from \(\mathcal{G}\) with \(\epsilon\)-contamination as follows: The algorithm first specifies the number of samples \(n\), and \(n\) samples are drawn independently from \(\mathcal{G}\). An adversary is then allowed to inspect these samples, and replace an \(\epsilon\)-fraction of the samples with arbitrary points. This modified set of \(n\) points is said to be \(\epsilon\)-corrupted, which is then given to the algorithm.

The stochastic optimization problem we consider is computationally intractable in full generality -- even without corruption -- if the goal is to obtain globally optimal solutions. At a high level, an achievable goal is to design sample and computationally efficient robust algorithms for finding _locally_ optimal solutions. Prior work [14][2] studied outlier-robust stochastic optimization and obtained efficient algorithms for finding approximate _first-order_ stationary points. While first-order guarantees suffice for convex problems, it is known that in many tractable non-convex problems, first-order stationary points may be bad solutions, but all _second-order_ stationary points (SOSPs) are globally optimal. This motivates us to study the following questions:

_Can we develop a general framework for finding **second-order** stationary points in outlier-robust stochastic optimization?_

_Can we obtain sample and computationally efficient algorithms for outlier-robust versions of tractable nonconvex problems using this framework?_

In this work, we answer both questions affirmatively. We introduce a framework for efficiently finding an approximate SOSP when \(\epsilon\)-fraction of the functions are corrupted and then use our framework to solve the problem of outlier-robust low rank matrix sensing.

In addition to the gradient being zero, a SOSP requires the Hessian matrix to not have negative eigenvalues. The second-order optimality condition is important because it rules out suboptimal solutions such as strict saddle points. It is known that all SOSPs are globally optimal in nonconvex formulations of many important machine learning problems, such as matrix completion [1], matrix sensing [1], phase retrieval [2], phase synchronization [3], dictionary learning [2], and tensor decomposition [4] (see also [13] Chapter 7). However, the properties of SOSPs are highly sensitive to perturbation in the input data. For example, it is possible to create spurious SOSPs for nonconvex formulations of low rank matrix recovery problems, even for a semi-random adversary that can add additional sensing matrices but cannot corrupt the measurements in matrix sensing [4] or an adversary who can only reveal more entries of the ground-truth matrix in matrix completion [4]. Those spurious SOSPs correspond to highly suboptimal solutions.

Finding SOSPs in stochastic nonconvex optimization problems in the presence of arbitrary outliers was largely unaddressed prior to our work. Prior works [14][2][3] obtained efficient and robust algorithms for finding _first-order_ stationary points with dimension-independent accuracy guarantees. These works relied on the following simple idea: Under certain smoothness assumptions, projected gradient descent with an _approximate_ gradient oracle efficiently converges to an _approximate_ first-order stationary point. Moreover, in the outlier-robust setting, approximating the gradient at a specific point amounts to a robust mean estimation problem (for the underlying distribution of the gradients), which can be solved by leveraging existing algorithms for robust mean estimation.

Our work is the first to find approximate SOSPs with dimension-independent errors in outlier-robust settings. Note that in standard non-robust settings, approximate SOSPs can be computed using first-order methods such as perturbed gradient descent [15][16][17]. This strategy might seem extendable to outlier-robust settings through perturbed approximate gradient descent, utilizing robust mean estimation algorithms to approximate gradients. The approach in [16][17] follows this idea, but unfortunately their second-order guarantees scale polynomially with dimension, even under very strong distributional assumptions (e.g., subgaussianity). Our lower bound result provides evidence that approximating SOSPs with dimension-independent error is as hard as approximating _full_ Hessian, suggesting that solely approximating the gradients is not sufficient. On a different note, [15] recently employed robust estimators for both gradient and Hessian in solving certain convex stochastic optimization problems, which has a different focus than ours and does not provide SOSPs with the guarantees that we achieve.

### Our Results and Contributions

The notation we use in this section is defined in Section2. To state our results, we first formally define our generic nonconvex optimization problem. Suppose there is a true distribution over functions \(f:\mathbb{R}^{D}\times\mathcal{A}\to\mathbb{R}\), where \(f(x,A)\) takes an argument \(x\in\mathbb{R}^{D}\) and is parameterized by a random variable \(A\in\mathcal{A}\) drawn from a distribution \(\mathcal{G}\). Our goal is to find an \((\epsilon_{g},\epsilon_{H})\)-approximate SOSP of the function \(\bar{f}(x):=\mathbb{E}_{A\sim\mathcal{G}}\,f(x,A)\).

**Definition 1.2** (\(\epsilon\)-Corrupted Stochastic Optimization).: The algorithm has access to \(n\) functions \((f_{i})_{i=1}^{n}\) generated as follows. First \(n\) random variables \((A_{i})_{i=1}^{n}\) are drawn independently from \(\mathcal{G}\). Then an adversary arbitrarily corrupts an \(\epsilon\) fraction of the \(A_{i}\)'s. Finally, the \(\epsilon\)-corrupted version of \(f_{i}(\cdot)=f(\cdot,A_{i})\) is sent to the algorithm as input. The task is to find an approximate SOSP of the ground-truth average function \(\bar{f}(\cdot):=\mathbb{E}_{A\sim\mathcal{G}}\,f(\cdot,A)\).

**Definition 1.3** (Approximate SOSPs).: A point \(x\) is an \((\epsilon_{g},\epsilon_{H})\)-approximate second-order stationary point (SOSP) of \(\bar{f}\) if \(\left\|\nabla\bar{f}(x)\right\|\leq\epsilon_{g}\) and \(\lambda_{\min}\left(\nabla^{2}\bar{f}(x)\right)\geq-\epsilon_{H}\).

We make the following additional assumptions on \(f\) and \(\mathcal{G}\).

**Assumption 1.4**.: _There exists a bounded region \(\mathcal{B}\) such that the following conditions hold:_

1. _There exists a lower bound_ \(f^{*}>-\infty\) _such that for all_ \(x\in\mathcal{B}\)_,_ \(f(x,A)\geq f^{*}\) _with probability_ \(1\)_._
2. _There exist parameters_ \(L_{D_{g}}\)_,_ \(L_{D_{H}}\)_,_ \(B_{D_{g}}\)_, and_ \(B_{D_{H}}\) _such that, with high probability over the randomness in_ \(A\sim\mathcal{G}\)_, letting_ \(g(x)=f(x,A)\)_, we have that_ \(g(x)\) _is_ \(L_{D_{g}}\)_-gradient Lipschitz and_ \(L_{D_{H}}\)_-Hessian Lipschitz over_ \(\mathcal{B}\)_, and_ \(\left\|\nabla g(x)\right\|\leq B_{D_{g}}\) _and_ \(\left\|\nabla^{2}g(x)\right\|_{F}\leq B_{D_{H}}\) _for all_ \(x\in\mathcal{B}\)_._
3. _There exist parameters_ \(\sigma_{g},\sigma_{H}>0\) _such that for all_ \(x\in\mathcal{B}\)_,_ \[\left\|\mathrm{Cov}_{A\sim\mathcal{G}}(\nabla f(x,A))\right\|_{\mathrm{op}} \leq\sigma_{g}^{2}\text{ and }\left\|\mathrm{Cov}_{A\sim\mathcal{G}}( \mathrm{vec}(\nabla^{2}f(x,A)))\right\|_{\mathrm{op}}\leq\sigma_{H}^{2}.\]

Note that the radius of \(\mathcal{B}\) and the parameters \(L_{D_{g}}\), \(L_{D_{H}}\), \(B_{D_{g}}\), \(B_{D_{H}}\) are all allowed to depend polynomially on \(D\) and \(\epsilon\) (but not on \(x\) and \(A\)).

Our main algorithmic result for \(\epsilon\)-corrupted stochastic optimization is summarized in the following theorem. A formal version of this theorem is stated as Theorem3.1 in Section3.

**Theorem 1.5** (Finding an Outlier-Robust SOSP, informal).: _Suppose \(f\) satisfies Assumption1.4 in a region \(\mathcal{B}\) with parameters \(\sigma_{g}\) and \(\sigma_{H}\). Given an arbitrary initial point \(x_{0}\in\mathcal{B}\) and an \(\epsilon\)-corrupted set of \(n=\widetilde{\Omega}\big{(}D^{2}/\epsilon\big{)}\) functions where \(D\) is the ambient dimension, there exists a polynomial-time algorithm that with high probability outputs an \((O(\sigma_{g}\sqrt{\epsilon})\,,O(\sigma_{H}\sqrt{\epsilon}))\)-approximate SOSP of \(\bar{f}\), provided that all iterates of the algorithm stay inside \(\mathcal{B}\)._

Although the bounded iterate condition in Theorem1.5 appears restrictive, this assumption holds if the objective function satisfies a "dissipativity" property, which is a fairly general phenomenon [11]. Moreover, adding an \(\ell_{2}\)-regularization term enables any Lipschitz function to satisfy the dissipativity property [12]. Section4. As an illustrating example, a simple problem-specific analysis shows that this bounded iterate condition holds for outlier-robust matrix sensing by exploiting the fact that the matrix sensing objective satisfies the dissipativity property.

In this paper, we consider the problem of outlier-robust symmetric low rank matrix sensing, which we formally define below. We focus on the setting with Gaussian design.

**Definition 1.6** (Outlier-Robust Matrix Sensing).: There is an unknown rank-\(r\) ground-truth matrix \(M^{*}\in\mathbb{R}^{d\times d}\) that can be factored into \(U^{*}U^{*}{}^{\top}\) where \(U^{*}\in\mathbb{R}^{d\times r}\). The (clean) sensing matrices \(\{A_{i}\}_{i\in[n]}\) have i.i.d. standard Gaussian entries. The (clean) measurements \(y_{i}\) are obtained as \(y_{i}=\langle A_{i},M^{*}\rangle+\zeta_{i}\), where the noise \(\zeta_{i}\sim\mathcal{N}(0,\sigma^{2})\) is independent from all other randomness. We denote the (clean) data generation process by \((A_{i},y_{i})\sim\mathcal{G}_{\sigma}\). When \(\sigma=0\), we have \(\zeta_{i}=0\) and we write \(\mathcal{G}:=\mathcal{G}_{0}\) for this noiseless (measurement) setting. In outlier robust matrix sensing, an adversary can arbitrarily change any \(\epsilon\)-fraction of the sensing matrices and the corresponding measurements. This corrupted set of \((A_{i},y_{i})\)'s is then given to the algorithm as input, where the goal is to recover \(M^{*}\).

We highlight that in our setting, both the sensing matrices \(A_{i}\in\mathbb{R}^{d\times d}\) and the measurements \(y_{i}\in\mathbb{R}\) can be corrupted, presenting a substantially more challenging problem compared to prior works (e.g., [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 120, 121, 122, 123, 124, 125, 126, 127, 128, 129, 130, 131, 140, 132, 133, 141, 142, 143, 144, 145, 146, 147, 148, 149, 150, 151, 152, 153, 154, 155, 156, 157, 158, 159, 160, 161, 162, 163, 164, 165, 166, 167, 168, 169, 170, 171, 172, 173, 174, 175, 176, 177, 178, 179, 180, 181, 182, 183, 184, 185, 186, 187, 188, 189, 190, 191, 192, 193, 194, 195, 196, 197, 198, 199, 199, 190, 191, 193, 195, 196, 197, 198, 199, 199, 198, 190, 199, 199, 191, 194, 196, 198, 199, 199, 197, 199, 198, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 1999, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 1999, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 1999, 199, 199, 1999, 199, 199, 1999, 199, 199, 199, 1999, 199, 1999, 199, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 199, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 19999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 19999, 1999, 19999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 19999, 1999, 1999, 1999, 1999, 1999, 19999, 1999, 19999, 19999, 19999, 19999, 1999,similar to strong convexity but holds only locally). This local regularity condition bounds below a measure of stationarity, which allows us to prove that gradient descent-type updates contract the distance to the closest global optimum under appropriate stepsize. We leverage this local regularity condition to prove that the iterates of the algorithm stay near a global optimum, so that the regularity condition continues to hold, and moreover, the distance between the current solution and the closest global optimum contracts, as long as it is larger than a function of \(\sigma\). Consequently, the distance-dependent component of the gradient and Hessian covariance bound contracts as well, which allows us to obtain more accurate gradient and Hessian estimates. While such a statement may seem evident to readers familiar with linear convergence arguments, we note that proving it is quite challenging, due to the circular dependence between the distance from the current solution to global optima, the inexactness in the gradient and Hessian estimates, and the progress made by our algorithm.

The described distance-contracting argument allows us to control the covariance of the gradient and Hessian, which we utilize to recover \(M^{*}\) exactly when \(\sigma=0\), and recover \(M^{*}\) with error roughly \(O(\sigma\sqrt{\epsilon})\) when \(0\neq\sigma\leq r\Gamma\). We note that the \(\sigma\sqrt{\epsilon}\) appears unavoidable in the \(\sigma\neq 0\) case, due to known limits of robust mean estimation algorithms [1].

SQ lower bound.We exhibit a hard instance of low rank matrix sensing problem to show that quadratic dependence on the dimension in sample complexity is unavoidable for computationally efficient algorithms. Our SQ lower bound proceeds by constructing a family of distributions, corresponding to corruptions of low rank matrix sensing, that are nearly uncorrelated in a well-defined technical sense [11]. To achieve this, we follow the framework of [1] which considered a family of distributions that are rotations of a carefully constructed one-dimensional distribution. The proof builds on [1][2], using a new univariate moment-matching construction which yields a family of corrupted conditional distributions. These induce a family of joint distributions that are SQ-hard to learn.

### Roadmap

Section 2 defines the necessary notation and discusses relevant building blocks of our algorithm and analysis. Section 3 introduces our framework for finding SOSPs in the outlier-robust setting. Section 4 presents how to extend and apply our framework to solve outlier-robust low rank matrix sensing. Section 5 proves that our sample complexity has optimal dimensional dependence for SQ algorithms. Most proofs are deferred to the supplementary material due to space limitations.

## 2 Preliminaries

For an integer \(n\), we use \([n]\) to denote the ordered set \(\{1,2,\ldots,n\}\). We use \([a_{i}]_{i\in\mathcal{I}}\) to denote the matrix whose columns are vectors \(a_{i}\), where \(\mathcal{I}\) is an ordered set. We use \(\mathbbm{1}_{E}(x)\) to denote the indicator function that is equal to \(1\) if \(x\in E\) and \(0\) otherwise. For two functions \(f\) and \(g\), we say \(f=\widetilde{O}(g)\) if \(f=O(g\log^{k}(g))\) for some constant \(k\), and we similarly define \(\widetilde{\Omega}\).

For vectors \(x\) and \(y\), we let \(\left\langle x,y\right\rangle\) denote the inner product \(x^{\top}y\) and \(\left\|x\right\|\) denote the \(\ell_{2}\) norm of \(x\). For \(d\in\mathbb{Z}_{+}\), we use \(I_{d}\) to denote the identity matrix of size \(d\times d\). For matrices \(A\) and \(B\), we use \(\left\|A\right\|_{\mathrm{op}}\) and \(\left\|A\right\|_{F}\) to denote the spectral norm and Frobenius norm of \(A\) respectively. We use \(\lambda_{\max}(A)\) and \(\lambda_{\min}(A)\) to denote the maximum and minimum eigenvalue of \(A\) respectively. We use \(\operatorname{tr}(A)\) to denote the trace of a matrix \(A\). We use \(\left\langle A,B\right\rangle=\operatorname{tr}(A^{\top}B)\) to denote the entry-wise inner product of two matrices of the same dimension. We use \(\operatorname{vec}(A)=[a_{1}^{\top},a_{2}^{\top},\ldots,a_{d}^{\top}]^{\top}\) to denote the canonical flattening of \(A\) into a vector, where \(a_{1},a_{2},\ldots,a_{d}\) are columns of \(A\).

**Definition 2.1** (Lipschitz Continuity).: Let \(\mathcal{X}\) and \(\mathcal{Y}\) be normed vector spaces. A function \(h:\mathcal{X}\rightarrow\mathcal{Y}\) is \(\ell\)-Lipschitz if \(\left\|h(x_{1})-h(x_{2})\right\|_{\mathcal{Y}}\leq\ell\left\|x_{1}-x_{2}\right\| _{\mathcal{X}},\forall x_{1},x_{2}\).

In this paper, when \(\mathcal{Y}\) is a space of matrices, we take \(\left\|\cdot\right\|_{\mathcal{Y}}\) to be the spectral norm \(\left\|\cdot\right\|_{\mathrm{op}}\). When \(\mathcal{X}\) is a space of matrices, we take \(\left\|\cdot\right\|_{\mathcal{X}}\) to be the Frobenius norm \(\left\|\cdot\right\|_{F}\); this essentially views the function \(h\) as operating on the vectorized matrices endowed with the usual \(\ell_{2}\) norm. When \(\mathcal{X}\) or \(\mathcal{Y}\) is the Euclidean space, we take the corresponding norm to be the \(\ell_{2}\) norm.

A Randomized Algorithm with Inexact Gradients and Hessians.We now discuss how to solve the unconstrained nonconvex optimization problem \(\min_{x\in\mathbb{R}^{D}}f(x),\) where \(f(\cdot)\) is a smooth function with Lipschitz gradients and Lipschitz Hessians. The goal of this section is to find an approximate SOSP as defined in Definition 1.3

**Proposition 2.2** ([11]).: _Suppose a function \(f\) is bounded below by \(f^{*}>-\infty,\) has \(L_{g}\)-Lipschitz gradient and \(L_{H}\)-Lipschitz Hessian, and its inexact gradient and Hessian computations \(\widetilde{g}_{t}\) and \(\widetilde{H}_{t}\) satisfy \(\left\lVert\widetilde{g}_{t}-\nabla f(x_{t})\right\rVert\leq\frac{1}{3} \epsilon_{g}\) and \(\left\lVert\widetilde{H}_{t}-\nabla^{2}f(x_{t})\right\rVert_{\mathrm{op}}\leq \frac{2}{9}\epsilon_{H}\). Then there exists an algorithm (Algorithm 1) with the following guarantees:_

1. _(Correctness) If Algorithm_ 1_._ _[_1_]_ _terminates and outputs_ \(x_{n}\)_, then_ \(x_{n}\) _is a_ \((\frac{4}{3}\epsilon_{g},\frac{4}{3}\epsilon_{H})\)_-approximate SOSP._
2. _(Runtime) Algorithm_ 1_._ _[_1_]_ _terminates with probability_ \(1\)_. Let_ \(C_{\epsilon}:=\min\left(\frac{\epsilon_{g}^{2}}{6L_{g}},\frac{2\epsilon_{H}^{2 }}{9L_{H}^{2}}\right)\)_. With probability at least_ \(1-\delta\)_, Algorithm_ 1_._ _[_1_]_ _terminates after_ \(k\) _iterations for_ \[k=O\Big{(}\frac{f(x_{0})-f^{*}}{C_{\epsilon}}+\frac{L_{H}^{2}L_{g}^{2}\epsilon _{g}^{2}}{\epsilon_{H}^{6}}\log\Bigl{(}\frac{1}{\delta}\Bigr{)}\Big{)}\;.\] (1)

The constants \(1/3\) and \(2/9\) are chosen for ease of presentation. For all constructions of Hessian oracles in this paper, we take the straightforward relaxation \(\left\lVert\widetilde{H}_{t}-\nabla^{2}f(x_{t})\right\rVert_{\mathrm{op}}\leq \left\lVert\widetilde{H}_{t}-\nabla^{2}f(x_{t})\right\rVert_{F}\) and upper bound Hessian inexactness using Frobenius norm. Proof of a simplified version of Proposition 2.2 with a weaker high probability bound that is sufficient for our purposes is provided in Appendix A.2 for completeness.

Robust Mean Estimation.Recent algorithms in robust mean estimation give dimension-independent error in the presence of outliers under strong contamination model.

We use the following results, see, e.g., [12], where the upper bound \(\sigma\) on the spectral norm of covariance matrix is unknown to the algorithm.

**Proposition 2.3** (Robust Mean Estimation).: _Fix any \(0<\xi<1\). Let \(S\) be a multiset of \(n=O((k\log k+\log(1/\xi))/\epsilon)\) i.i.d. samples from a distribution on \(\mathbb{R}^{k}\) with mean \(\mu_{S}\) and covariance \(\Sigma\). Let \(T\subset\mathbb{R}^{k}\) be an \(\epsilon\)-corrupted version of \(S\) as in Definition 1.1. There exists an algorithm (Algorithm 2) such that, with probability at least \(1-\xi\), on input \(\epsilon\) and \(T\) (but not \(\left\lVert\Sigma\right\rVert_{\mathrm{op}}\)) returns a vector \(\widehat{\mu}\) in polynomial time so that \(\left\lVert\mu_{S}-\widehat{\mu}\right\rVert=O(\sqrt{\left\lVert\Sigma\right\rVert _{\mathrm{op}}\epsilon})\)._

Algorithm 2 is given in Appendix A.3 Proposition 2.3 states that for Algorithm 2 to succeed with high probability, \(\widetilde{O}(k/\epsilon)\) i.i.d. samples need to be drawn from a \(k\)-dimensional distribution of bounded covariance. State of the art algorithms for robust mean estimation can be implemented in near-linear time, requiring only a logarithmic number of passes on the data, see, e.g. [13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 52, 54, 59, 61, 53, 56, 58, 59, 70, 54, 51, 52, 55, 57, 59, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 109, 111, 110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 120, 121, 122, 123, 124, 125, 126, 127, 128, 129, 130, 131, 140, 132, 133, 134, 135, 136, 137, 138, 139, 141, 142, 139, 143, 144, 145, 146, 147, 148, 149, 150, 151, 152, 153, 154, 155, 156, 157, 158, 159, 160, 161, 162, 163, 164, 165, 166, 167, 168, 169, 170, 171, 172, 173, 174, 175, 176, 177, 178, 179, 180, 181, 182, 183, 184, 185, 186, 187, 188, 189, 190, 191, 192, 193, 194, 195, 196, 197, 198, 199, 199, 199, 197, 199, 198, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 1999, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 1999, 1999, 199, 199, 199, 1999, 199, 1999, 1999, 1999, 1999, 1999, 1999, 199, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 19999, 1999, 19999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 1999, 19999, 19999, 1999, 1999, 19999, 1999, 19999, 1999, 1999, 1999, 1999, 19999, 1999, 19999, 1999, 1999, 19999, 19999, 1999, 19999, 19999, 1999, 19999, 1999, 1999, 1999, 1999, 19999, 19999, 19999, 19999, 1999, 19999, 1999, 1999, 19999, 1999, 1999, 19999, 1999, 19999, 19999, 19999, 1999, 1999, 19999, 1999, 19999, 19999, 19999, 19999, 19999, 19999, 19999, 1999, 19999, 19999, 19999, 19999, 19999, 19999, 19999, 19999, 19999, 19999, 19999, 1999, 19999, 19999, 19999, 19999, 19999, 19999, 19999, 19999, 19999, 19999, 19999, 19999, 199999, 19999, 19999, 19999, 19999, 19999, 19999, 19999, 19999, 19999, 19999, 19999, 19999, 19999, 19999, 19999, 19999, 19999, 19999, 19999, 19999, 19999, 19999, 19999, 19999, 19999, 19999, 19999, 1999, 19999, 19999, 19999, 19999, 19999, 19999, 19999, 19999, 19999, 19999, 19999, 19999, 19999, 19999, 19999, 19999, 19999, 1999, 19999, 19999, 1999, 19999, 19999, 19999, 19999, 19999, 1Then we have the following guarantee:

**Theorem 3.1**.: _Suppose we are given \(\epsilon\)-corrupted set of functions \(\{f_{i}\}_{i=1}^{n}\) for sample size \(n\), generated according to Definition1.2 Suppose Assumption1.4 holds in a bounded region \(\mathcal{B}\subset\mathbb{R}^{D}\) of diameter \(\gamma\) with gradient and Hessian covariance bound \(\sigma_{g}\) and \(\sigma_{H}\) respectively, and we have an arbitrary initialization \(x_{0}\in\mathcal{B}\). Algorithm1.1 initialized at \(x_{0}\) outputs an \((\epsilon_{g},\epsilon_{H})\)-approximate SOSP for a sufficiently large sample with probability at least \(1-\xi\) if the following conditions hold:_

1. _All iterates_ \(x_{t}\) _in Algorithm_1.1_stay inside the bounded region_ \(\mathcal{B}\)_._
2. _For an absolute constant_ \(c>0\)_, it holds that_ \(\sigma_{g}\sqrt{\epsilon}\leq c\epsilon_{g}\) _and_ \(\sigma_{H}\sqrt{\epsilon}\leq c\epsilon_{H}\)_._

_The algorithm uses \(n=\widetilde{O}(D^{2}/\epsilon)\) samples, where \(\widetilde{O}(\cdot)\) hides logarithmic dependence on \(D,\epsilon,L_{D_{y}},L_{D_{H}},B_{D_{y}},B_{D_{H}},\gamma/\sigma_{H},\gamma/ \sigma_{g},\) and \(1/\xi\). The algorithm runs in time polynomial in the above parameters._

Note that we are able to obtain dimension-independent errors \(\epsilon_{g}\) and \(\epsilon_{H}\), provided that \(\sigma_{g}\) and \(\sigma_{H}\) are dimension-independent.

### Low Rank Matrix Sensing Problems

In this section, we study the problem of outlier-robust low rank matrix sensing as formally defined in Definition1.6. We first apply the above framework to obtain an approximate SOSP in Section8.1.2. Then we make use of the approximate SOSP to obtain a solution that is close to the ground-truth matrix \(M^{*}\) in Section8.1.3 this demonstrates the usefulness of approximate SOSPs.

#### 3.1.1 Main results for Robust Low Rank Matrix Sensing

The following are the main results that we obtain in this section:

**Theorem 3.2** (Main Theorem Under Noiseless Measurements).: _Consider the noiseless setting as in Theorem1.7 with \(\sigma=0\). For some sample size \(n=\widetilde{O}((d^{2}r^{2}+dr\log(\Gamma/\xi))/\epsilon)\) and with probability at least \(1-\xi\), there exists an algorithm that outputs a solution that is \(\iota\)-close to \(M^{*}\) in Frobenius norm in \(O(r^{2}\kappa^{3}\log(1/\xi)+\kappa\log(\sigma_{r}^{*}/\iota))\) calls to the robust mean estimation subroutine (Algorithm1.2)._

This result achieves exact recovery of \(M^{*}\), despite the strong contamination of samples. Each iteration involves a subroutine call to robust mean estimation. Algorithm1.2 presented here is one simple example of robust mean estimation; there are refinements [12][11] that run in nearly linear time, so the total computation utilizing those more efficient algorithms indeed requires \(\widetilde{O}\left(r^{2}\kappa^{3}\right)\) passes of data (computed gradients and Hessians).

**Theorem 3.3** (Main Theorem Under Noisy Measurements).: _Consider the same setting as in Theorem1.7 with \(\sigma\neq 0\). There exists a sample size \(n=\widetilde{O}((d^{2}r^{2}+dr\log(\Gamma/\xi))/\epsilon)\) such that_

* _if_ \(\sigma\leq r\Gamma\)_, then with probability at least_ \(1-\xi\)_, there exists an algorithm that outputs a solution_ \(\widehat{M}\) _in_ \(\widetilde{O}(r^{2}\kappa^{3})\) _calls to robust mean estimation routine_ _[_4.2_]_ _with error_ \(\left\|\widehat{M}-M^{*}\right\|_{F}=O(\kappa\sigma\sqrt{\epsilon})\)_;_
* _if_ \(\sigma\geq r\Gamma\)_, then with probability at least_ \(1-\xi\)_, there exists a (different) algorithm that outputs a solution_ \(\widehat{M}\) _in one call to robust mean estimation routine_ _[_4.2_]_ _with error_ \(\left\|\widehat{M}-M^{*}\right\|_{F}=O(\sigma\sqrt{\epsilon})\)_._

We prove Theorem5.3 in AppendixC.4 and instead focus on the noiseless measurements with \(\sigma=0\) when we develop our algorithms in this section; the two share many common techniques. In the remaining part of Section8.1 we use \(\mathcal{G}_{0}\) in Definition1.6 for the data generation process.

We now describe how we obtain the solution via nonconvex optimization. Consider the following objective function for (uncorrupted) matrix sensing:

\[\min_{\begin{subarray}{c}M\in\mathbb{R}^{d\times d}\\ \mathrm{rank}(M)=r\end{subarray}}\frac{1}{2}\operatorname*{\mathbb{E}}_{(A_{ i},y_{i})\sim\mathcal{G}_{0}}(\langle M,A_{i}\rangle-y_{i})^{2}. \tag{2}\]

We can write \(M=UU^{\top}\) for some \(U\in\mathbb{R}^{d\times r}\) to reparameterize the objective function. Let

\[f_{i}(U):=\frac{1}{2}\left(\langle UU^{\top},A_{i}\rangle-y_{i}\right)^{2}. \tag{3}\]We can compute

\[\bar{f}(U):=\mathop{\mathbb{E}}_{(A_{i},y_{i})\sim\mathcal{G}_{0}}f_{i}(U)=\frac{1 }{2}\mathop{\mathrm{Var}}\langle UU^{\top}-M^{*},A_{i}\rangle=\frac{1}{2}\left\| UU^{\top}-M^{*}\right\|_{F}^{2}. \tag{4}\]

We seek to solve the following optimization problem under the corruption model in Definition 1.2

\[\min_{U\in\mathbb{R}^{d\times r}}\bar{f}(U). \tag{5}\]

The gradient Lipschitz constant and Hessian Lipschitz constant of \(\bar{f}\) are given by the following result.

**Fact 3.4** ([17], Lemma 6).: _For any \(\Gamma>\sigma_{1}^{*}\), \(\bar{f}(U)\) has gradient Lipschitz constant \(L_{g}=16\Gamma\) and Hessian Lipschitz constant \(L_{H}=24\Gamma^{\frac{1}{2}}\) inside the region \(\{U:\left\|U\right\|_{\mathrm{op}}^{2}<\Gamma\}\)._

#### 3.1.2 Global Convergence to an Approximate SOSP

In this section, we apply our framework Theorem 6.1 to obtain global convergence from an arbitrary initialization to an approximate SOSP, by providing problem-specific analysis to guarantee that both Assumption 1.4 and algorithmic assumptions (I) and (II) required by Theorem 6.1 are satisfied.

**Theorem 3.5** (Global Convergence to a SOSP).: _Consider the noiseless setting as in Theorem 1.7 with \(\sigma=0\) and \(\epsilon=O(1/(\kappa^{3}r^{3}))\). Assume we have an arbitrary initialization \(U_{0}\) inside \(\{U:\left\|U\right\|_{\mathrm{op}}^{2}\leq\Gamma\}\). There exists a sample size \(n=\widetilde{O}\left((d^{2}r^{2}+dr\log(\Gamma/\xi))/\epsilon\right)\) such that with probability at least \(1-\xi\), Algorithm 1.1 initialized at \(U_{0}\) outputs a \((\frac{1}{24}\sigma_{r}^{3/2},\frac{1}{3}\sigma_{r}^{*})\)-approximate SOSP using at most \(O\big{(}r^{2}\kappa^{3}\log(1/\xi)\big{)}\) calls to robust mean estimation subroutine (Algorithm 2.2)._

Proof of Theorem 3.5.: To apply Theorem 3.1 we verify Assumption 1.4 first. To verify (i), for all \(U\) and \(A_{i}\), \(f_{i}(U)=\frac{1}{2}\left(\langle UU^{\top},A_{i}\rangle-y_{i}\right)^{2}\geq 0\), so \(f^{*}=0\) is a uniform lower bound. We verify (ii) in Appendix C.2 conceptually, by Fact 3.4\(\bar{f}\) is gradient and Hessian Lipschitz; both gradient and Hessian of \(f_{i}\) are sub-exponential and concentrate around those of \(\bar{f}\). To check (iii), we calculate the gradients and Hessians of \(f_{i}\) in Appendix C.1.1 and bound their covariances from above in Appendix C.1.2 and 1.3. The result is summarized in the following lemma. Note that the domain of the target function in Algorithm 1.1 and Theorem 9.1 is the Euclidean space \(\mathbb{R}^{D}\), so we vectorize \(U\) and let \(D=dr\). The gradient becomes a vector in \(\mathbb{R}^{dr}\) and the Hessian becomes a matrix in \(\mathbb{R}^{dr\times dr}\).

**Lemma 3.6** (Gradient and Hessian Covariance Bounds).: _For all \(U\in\mathbb{R}^{d\times r}\) with \(\left\|U\right\|_{\mathrm{op}}^{2}\leq\Gamma\) and \(f_{i}\) defined in Equation 3.1, it holds_

\[\left\|\mathrm{Cov}(\mathrm{vec}(\nabla f_{i}(U)))\right\|_{ \mathrm{op}} \leq 8\left\|UU^{\top}-M^{*}\right\|_{F}^{2}\left\|U\right\|_{ \mathrm{op}}^{2}\leq 32r^{2}\Gamma^{3} \tag{6}\] \[\left\|\mathrm{Cov}(\mathrm{vec}(H_{i}))\right\|_{\mathrm{op}} \leq 16r\left\|UU^{\top}-M^{*}\right\|_{F}^{2}+128\left\|U\right\|_{ \mathrm{op}}^{4}\leq 192r^{3}\Gamma^{2} \tag{7}\]

We proceed to verify the algorithmic assumptions in Theorem 9.1.1 For the assumption (I), we prove the following Lemma in Appendix C.2 to show that all iterates stay inside the bounded region in which we compute the covariance bounds.

**Lemma 3.7**.: _All iterates of Algorithm 1.1 stay inside the region \(\{U:\left\|U\right\|_{\mathrm{op}}^{2}\leq\Gamma\}\)._

To verify Theorem 9.1.1(II), we let \(\epsilon_{g}=\frac{1}{32}\sigma_{r}^{*3/2},\epsilon_{H}=\frac{1}{4}\sigma_{r} ^{*}\) and \(\sigma_{g}=8r\Gamma^{1.5},\sigma_{H}=16r^{1.5}\Gamma\). So if we assume \(\epsilon=O(\overline{1/(\kappa^{3}r^{3})})\), then for the absolute constant \(c\) in Theorem 9.1 it holds that

\[\sigma_{g}\sqrt{\epsilon}\leq c\epsilon_{g}\qquad\sigma_{H}\sqrt{\epsilon}\leq c \epsilon_{H}.\]

Hence, Theorem 9.1.1 applies and Algorithm 1.1 outputs an \((\epsilon_{g},\epsilon_{H})\)-approximate SOSP with high probability in polynomial time. To bound the runtime, since \(\bar{f}(U_{0})=1/2\left\|U_{0}U_{0}^{\top}-M^{*}\right\|_{F}^{2}=O(r^{2} \Gamma^{2})\) for an arbitrary initialization \(U_{0}\) with \(\left\|U_{0}\right\|_{\mathrm{op}}^{2}<\Gamma\), the initial distance can be bounded by \(O(r^{2}\Gamma^{2})\). Setting \(L_{g}=16\Gamma,L_{H}=24\Gamma^{1/2},\bar{f}(U_{0})=O(r^{2}\Gamma^{2}),f^{*}=0\) and thus \(C_{\epsilon}=O(\sigma_{r}^{*3}/\Gamma)\), Proposition 2.2 implies that Algorithm 1.1 outputs a \((\frac{1}{24}\sigma_{r}^{*3/2},\frac{1}{3}\sigma_{r}^{*})\)-approximate second order stationary point \(U_{SOSP}\) in \(O(r^{2}\kappa^{3}\log(1/\xi))\) steps with high probability.

#### 3.1.3 Local Linear Convergence

In this section, we describe a local search algorithm that takes a \((\frac{1}{2d}\sigma_{r}^{*3/2},\frac{1}{3}\sigma_{r}^{\star})\)-approximate second-order stationary point as its initialization and achieves exact recovery even in the presence of outliers.

```
Data: The initialization \(U_{SOSP}\) is a \((\frac{1}{3d}\sigma_{r}^{*3/2},\frac{1}{3}\sigma_{r}^{\star})\)-approximate SOSP, corruption fraction is \(\epsilon\), corrupted samples are \(\{(A_{i},y_{i})\}_{i=1}^{n}\), target distance to optima is \(\iota\) Result:\(U\) that is \(\iota\)-close in Frobenius norm to some global minimum
1\(\eta=1/\Gamma\), \(U_{0}=U_{SOSP}\)
2fort =0, 1,...do
3\(\widehat{g}_{t}:=\mathbf{RobustMeanEstimation}(\{\nabla f_{i}(U_{t})\}_{i=1}^{n}, 4\epsilon)\)
4\(U_{t+1}\gets U_{t}-\eta\widehat{g}_{t}\)
```

**Algorithm 3.1**Local Inexact Gradient Descent

**Theorem 3.8** (Local Linear Convergence).: _Consider the same noiseless setting as in Theorem [1]. Assume we already found a \((\frac{1}{2}\sigma_{r}^{*3/2},\frac{1}{3}\sigma_{r}^{\star})\)-approximate SOSP \(U_{SOSP}\) of \(\bar{f}\). Then there exists a sample size \(n=\widetilde{O}\left(dr\log(1/\xi)/\epsilon\right)\) such that with probability at least \(1-\xi\), Algorithm [3] initialized at \(U_{SOSP}\) outputs a solution that is \(\iota\)-close to some global minimum in Frobenius norm after \(O(\kappa\log(\sigma_{r}^{\star}/\iota))\) calls to robust mean estimation subroutine (Algorithm [4]. Moreover, all iterates \(U_{t}\) are \(\frac{1}{3}\sigma_{r}^{*1/2}\)-close to some global minimum in Frobenius norm._

Proof Sketch.: First we use known properties of \(\bar{f}=\mathbb{E}_{(A_{i},y_{i})\sim\mathcal{G}_{0}}\,f_{i}\) from the literature [11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 120, 121, 122, 123, 124, 125, 126, 127, 128, 129, 130, 131, 132, 133, 134, 135, 136, 137, 138, 139, 140, 141, 142, 143, 144, 145, 146, 147, 148, 149, 150, 151, 152, 153, 154, 155, 156, 157, 158, 159, 160, 161, 162, 163, 164, 165, 166, 167, 168, 169, 170, 171, 172, 173, 174, 175, 176, 177, 178, 179, 180, 181, 182, 183, 184, 185, 186, 187, 188, 189, 190, 191, 192, 193, 194, 195, 196, 197, 198, 199, 199, 199, 198, 199, 199, 199, 190, 191, 190, 192, 193, 194, 195, 196, 197, 198, 199, 199, 199, 199, 198, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 199, 1999, 199,responds to the query \(q\) with a value \(v\) such that \(|v-\mathbb{E}_{X\sim\mathcal{P}}[q(X)]|\leq\tau\). An SQ algorithm is an algorithm whose objective is to learn some information about an unknown distribution \(\mathcal{P}\) by making adaptive calls to the corresponding \(\mathsf{STAT}(\tau)\) oracle.

In this section, we consider \(\mathcal{P}\) as the unknown corrupted distribution where \((A_{i},y_{i})\) are drawn. The SQ algorithm tries to learn the ground truth matrix \(M^{*}\) from this corrupted distribution; the goal of the lower bound result is to show that this is hard.

The SQ model has the capability to implement a diverse set of algorithmic techniques in machine learning such as spectral techniques, moment and tensor methods, local search (e.g., Expectation Maximization), and several others [11]. A lower bound on the SQ complexity of a problem provides evidence of hardness for the problem. [12] established that (under certain assumptions) an SQ lower bound also implies a qualitatively similar lower bound in the low-degree polynomial testing model. This connection can be used to show a similar lower bound for low-degree polynomials.

Our main result here is a near-optimal SQ lower bound for robust low rank matrix sensing that applies even for rank \(r=1\), i.e., when the ground truth matrix is \(M^{*}=uu^{\top}\) for some \(u\in\mathbb{R}^{d}\). The choice of rank \(r=1\) yields the strongest possible lower bound in our setting because it is the easiest parameter regime: Recall that the sample complexity of our algorithm is \(\widetilde{O}(d^{2}r^{2})\) as in Theorems 3.2 and 3.3 and the main message of our SQ lower bound is to provide evidence that the \(d^{2}\) factor is necessary for computationally efficient algorithms _even if_\(r=1\).

**Theorem 4.2** (SQ Lower Bound for Robust Rank-One Matrix Sensing).: _Let \(\epsilon\in(0,1/2)\) be the fraction of corruptions and let \(c\in(0,1/2)\). Assume the dimension \(d\in\mathbb{N}\) is sufficiently large. Consider the \(\epsilon\)-corrupted rank-one matrix sensing problem with ground-truth matrix \(M^{*}=uu^{\top}\) and noise \(\sigma^{2}=O(1)\). Any SQ algorithm that outputs \(\widehat{u}\) with \(\|\widehat{u}-u\|=O(\epsilon^{1/4})\) either requires \(2^{\Omega(d^{\epsilon})}/d^{2-4c}\) queries or makes at least one query to \(\mathsf{STAT}\left(e^{O(1/\sqrt{\epsilon})}/O\left(d^{1-2c}\right)\right)\)._

In other words, we show that, when provided with SQ access to an \(\epsilon\)-corrupted distribution, approximating \(u\) is impossible unless employing a statistical query of higher precision than what can be achieved with a strictly sub-quadratic number (e.g., \(d^{1.99}\)) of samples. Note that the SQ oracle \(\mathsf{STAT}(e^{O\left(1/\sqrt{\epsilon}\right)}/O(d^{1-2c}))\) can be simulated with \(O(d^{2-4c})/e^{O(1/\epsilon)}\) samples, and this bound is tight in general. Informally speaking, this theorem implies that improving the sample complexity from \(d^{2}\) to \(d^{2-4c}\) requires exponentially many queries. This result can be viewed as a near-optimal information-computation tradeoff for the problem, within the class of SQ algorithms.

The proof follows a similar analysis as in [13], using one-dimensional moment matching to construct a family of corrupted conditional distributions, which induce a family of corrupted joint distributions that are SQ-hard to learn. We provide the details of the proof in Appendix A Apart from the formal proof, in Appendix E we also informally discuss the intuition for why some simple algorithms that require \(O(d)\) samples do not provide dimension-independent error guarantees.

## Acknowledgements

Shuyao Li was supported in part by NSF Awards DMS-2023239, NSF Award CCF-2007757 and the U. S. Office of Naval Research under award number N00014-22-1-2348. Yu Cheng was supported in part by NSF Award CCF-2307106. Ilias Diakonikolas was supported in part by NSF Medium Award CCF-2107079, NSF Award CCF-1652862 (CAREER), a Sloan Research Fellowship, and a DARPA Learning with Less Labels (LwLL) grant. Jelena Diakonikolas was supported in part by NSF Award CCF-2007757 and by the U. S. Office of Naval Research under award number N00014-22-1-2348. Rong Ge was supported in part by NSF Award DMS-2031849, CCF-1845171 (CAREER) and a Sloan Research Fellowship. Stephen Wright was supported in part by NSF Awards DMS-2023239 and CCF-2224213 and AFOSR via subcontract UTA20-001224 from UT-Austin.

## References

* [Bar+10] M. Barreno, B. Nelson, A. D. Joseph, and J. D. Tygar. "The security of machine learning". In: _Machine Learning_ 81.2 (2010), pp. 121-148.
* [BBV16] A. S. Bandeira, N. Boumal, and V. Voroninski. "On the low-rank approach for semidefinite programs arising in synchronization and community detection". In: _Conference on learning theory_. PMLR. 2016, pp. 361-382.
* [Ber+22] E. H. Bergou, Y. Diouane, V. Kunc, V. Kungurtsev, and C. W. Royer. "A subsampling line-search method with second-order results". In: _INFORMS Journal on Optimization_ 4.4 (2022), pp. 403-425.
* [BNL12] B. Biggio, B. Nelson, and P. Laskov. "Poisoning Attacks against Support Vector Machines". In: _Proceedings of the 29th International Coference on International Conference on Machine Learning_. Omnipress, 2012, pp. 1467-1474.
* [BNS16] S. Bhojanapalli, B. Neyshabur, and N. Srebro. "Global optimality of local search for low rank matrix recovery". In: _Advances in Neural Information Processing Systems_. 2016, pp. 3873-3881.
* [Bre+21] M. S. Brennan, G. Bresler, S. Hopkins, J. Li, and T. Schramm. "Statistical Query Algorithms and Low Degree Tests Are Almost Equivalent". In: _Proceedings of Thirty Fourth Conference on Learning Theory_. Ed. by M. Belkin and S. Kpotufe. Vol. 134. Proceedings of Machine Learning Research. PMLR, 15-19 Aug 2021, pp. 774-774.
* [CDG19] Y. Cheng, I. Diakonikolas, and R. Ge. "High-Dimensional Robust Mean Estimation in Nearly-Linear Time". In: _Proceedings of the 30th ACM-SIAM Symposium on Discrete Algorithms (SODA)_. SIAM, 2019, pp. 2755-2771.
* [CG18] Y. Cheng and R. Ge. "Non-convex matrix completion against a semi-random adversary". In: _Conference On Learning Theory_. PMLR. 2018, pp. 1362-1394.
* [DHL19] Y. Dong, S. Hopkins, and J. Li. "Quantum entropy scoring for fast robust mean estimation and improved outlier detection". In: _Advances in Neural Information Processing Systems_ 32 (2019).
* [Dia+16] I. Diakonikolas, G. Kamath, D. M. Kane, J. Li, A. Moitra, and A. Stewart. "Robust estimators in high dimensions without the computational intractability". In: _57th Annual IEEE Symposium on Foundations of Computer Science--FOCS 2016_. IEEE Computer Soc., Los Alamitos, CA, 2016, pp. 655-664.
* [Dia+17] I. Diakonikolas, G. Kamath, D. M. Kane, J. Li, A. Moitra, and A. Stewart. "Being Robust (in High Dimensions) Can Be Practical". In: _Proceedings of the 34th International Conference on Machine Learning_. Ed. by D. Precup and Y. W. Teh. Vol. 70. Proceedings of Machine Learning Research. PMLR, June 2017, pp. 999-1008.
* [Dia+19] I. Diakonikolas, G. Kamath, D. Kane, J. Li, J. Steinhardt, and A. Stewart. "Sever: A Robust Meta-Algorithm for Stochastic Optimization". In: _Proceedings of the 36th International Conference on Machine Learning_. Ed. by K. Chaudhuri and R. Salakhutdinov. Vol. 97. Proceedings of Machine Learning Research. PMLR, Sept. 2019, pp. 1596-1606.
* [Dia+21] I. Diakonikolas, D. Kane, A. Pensia, T. Pittas, and A. Stewart. "Statistical Query Lower Bounds for List-Decodable Linear Regression". In: _Advances in Neural Information Processing Systems_. Ed. by M. Ramzato, A. Beygelzimer, Y. Dauphin, P. Liang, and J. W. Vaughan. Vol. 34. Curran Associates, Inc., 2021, pp. 3191-3204.
* [Dia+22] I. Diakonikolas, D. M. Kane, A. Pensia, and T. Pittas. "Streaming Algorithms for High-Dimensional Robust Statistics". In: _Proceedings of the 39th International Conference on Machine Learning_. Ed. by K. Chaudhuri, S. Jegelka, L. Song, C. Szepesvari, G. Niu, and S. Sabato. Vol. 162. Proceedings of Machine Learning Research. PMLR, July 2022, pp. 5061-5117.
* [DK19] I. Diakonikolas and D. M. Kane. "Recent advances in algorithmic high-dimensional robust statistics". In: _arXiv preprint arXiv:1911.05911_ (2019).
* [DK23] I. Diakonikolas and D. M. Kane. _Algorithmic High-Dimensional Robust Statistics_. Cambridge University Press, 2023.
* [DKP20] I. Diakonikolas, D. M. Kane, and A. Pensia. "Outlier Robust Mean Estimation with Subgaussian Rates via Stability". In: _Advances in Neural Information Processing Systems_. Ed. by H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin. Vol. 33. Curran Associates, Inc., 2020, pp. 1830-1840.

* [DKS17] I. Diakonikolas, D. M. Kane, and A. Stewart. "Statistical Query Lower Bounds for Robust Estimation of High-Dimensional Gaussians and Gaussian Mixtures". In: _2017 IEEE 58th Annual Symposium on Foundations of Computer Science (FOCS)_. 2017, pp. 73-84.
* [DKS19] I. Diakonikolas, W. Kong, and A. Stewart. "Efficient algorithms and lower bounds for robust linear regression". In: _Proceedings of the Thirtieth Annual ACM-SIAM Symposium on Discrete Algorithms_. SIAM, Philadelphia, PA, 2019, pp. 2745-2754.
* [Dur19] R. Durrett. _Probability--theory and examples_. Vol. 49. Cambridge Series in Statistical and Probabilistic Mathematics. Cambridge University Press, Cambridge, 2019.
* [Fel+17] V. Feldman, E. Grigorescu, L. Reyzin, S. Vempala, and Y. Xiao. "Statistical Algorithms and a Lower Bound for Detecting Planted Cliques". In: _J. ACM_ 64.2 (2017), 8:1-8:37.
* [Gao20] C. Gao. "Robust regression via mutivariate regression depth". In: _Bernoulli_ 26.2 (2020), pp. 1139-1170.
* [GC23] X. Gao and Y. Cheng. "Robust Matrix Sensing in the Semi-Random Model". In: _Proceedings of the 37th Conference on Neural Information Processing Systems (NeurIPS)_ (2023).
* [Ge+15] R. Ge, F. Huang, C. Jin, and Y. Yuan. "Escaping from saddle points--online stochastic gradient for tensor decomposition". In: _Conference on Learning Theory_. 2015, pp. 797-842.
* [GJZ17] R. Ge, C. Jin, and Y. Zheng. "No Spurious Local Minima in Nonconvex Low Rank Problems: A Unified Geometric Analysis". In: _Proceedings of the 34th International Conference on Machine Learning_. Ed. by D. Precup and Y. W. Teh. Vol. 70. Proceedings of Machine Learning Research. PMLR, June 2017, pp. 1233-1242.
* [GLM16] R. Ge, J. D. Lee, and T. Ma. "Matrix completion has no spurious local minimum". In: _Advances in Neural Information Processing Systems_. 2016, pp. 2973-2981.
* [Hal10] J. K. Hale. _Asymptotic behavior of dissipative systems_. 25. American Mathematical Soc., 2010.
* [Ham+86] F. R. Hampel, E. M. Ronchetti, P. J. Rousseeuw, and W. A. Stahel. _Robust statistics. The approach based on influence functions_. Wiley New York, 1986.
* [HR09] P. J. Huber and E. M. Ronchetti. _Robust statistics_. Wiley New York, 2009.
* [Hub64] P. J. Huber. "Robust estimation of a location parameter". In: _Annals of Mathematical Statistics_ 35 (1964), pp. 73-101.
* [IPL23] E. Ioannou, M. S. Pydi, and P.-L. Loh. "Robust empirical risk minimization via Newton's method". In: _arXiv preprint arXiv:2301.13192_ (2023).
* [Jin+17] C. Jin, R. Ge, P. Netrapalli, S. M. Kakade, and M. I. Jordan. "How to Escape Saddle Points Efficiently". In: _Proceedings of the 34th International Conference on Machine Learning_. Ed. by D. Precup and Y. W. Teh. Vol. 70. Proceedings of Machine Learning Research. PMLR, June 2017, pp. 1724-1732.
* [Jin+21] C. Jin, P. Netrapalli, R. Ge, S. M. Kakade, and M. I. Jordan. "On nonconvex optimization for machine learning: gradients, stochasticity, and saddle points". In: _Journal of the ACM_ 68.2 (2021), Art. 11, 29.
* [Kea98] M. J. Kearns. "Efficient noise-tolerant Learning from Statistical Queries". In: _Journal of the ACM_ 45.6 (1998), pp. 983-1006.
* [Li+08] J. Li, D. Absher, H. Tang, A. Southwick, A. Casto, S. Ramachandran, H. Cann, G. Barsh, M. Feldman, L. Cavalli-Sforza, and R. Myers. "Worldwide human relationships inferred from genome-wide patterns of variation". In: _Science_ 319 (2008), pp. 1100-1104.
* [Li+20a] X. Li, Z. Zhu, A. Man-Cho So, and R. Vidal. "Nonconvex robust low-rank matrix recovery". In: _SIAM Journal on Optimization_ 30.1 (2020), pp. 660-686.
* [Li+20b] Y. Li, Y. Chi, H. Zhang, and Y. Liang. "Non-convex low-rank matrix recovery with arbitrary outliers via median-truncated gradient descent". In: _Information and Inference: A Journal of the IMA_ 9.2 (2020), pp. 289-325.
* [LRV16] K. A. Lai, A. B. Rao, and S. Vempala. "Agnostic Estimation of Mean and Covariance". In: _focs2016_. 2016, pp. 665-674.
* [LW23] S. Li and S. J. Wright. "A randomized algorithm for nonconvex minimization with inexact evaluations and complexity guarantees". In: _arXiv preprint arXiv:2310.18841_ (2023).

* [Pas+10] P. Paschou, J. Lewis, A. Javed, and P. Drineas. "Ancestry Informative Markers for Fine-Scale Individual Assignment to Worldwide Populations". In: _Journal of Medical Genetics_ 47 (2010), pp. 835-847.
* [Pra+20] A. Prasad, A. S. Suggala, S. Balakrishnan, and P. Ravikumar. "Robust estimation via robust gradient estimation". In: _Journal of the Royal Statistical Society. Series B. Statistical Methodology_ 82.3 (2020), pp. 601-627.
* [Ros+02] N. Rosenberg, J. Pritchard, J. Weber, H. Cann, K. Kidd, L. Zhivotovsky, and M. Feldman. "Genetic structure of human populations". In: _Science_ 298 (2002), pp. 2381-2385.
* [RRT17] M. Raginsky, A. Rakhlin, and M. Telgarsky. "Non-convex learning via Stochastic Gradient Langevin Dynamics: a nonasymptotic analysis". In: _Proceedings of the 2017 Conference on Learning Theory_. Ed. by S. Kale and O. Shamir. Vol. 65. Proceedings of Machine Learning Research. PMLR, July 2017, pp. 1674-1703.
* [SKL17] J. Steinhardt, P. W. Koh, and P. S. Liang. "Certified Defenses for Data Poisoning Attacks". In: _Advances in Neural Information Processing Systems 30_. 2017, pp. 3520-3532.
* [SQW16] J. Sun, Q. Qu, and J. Wright. "A geometric analysis of phase retrieval". In: _Information Theory (ISIT), 2016 IEEE International Symposium on_. IEEE. 2016, pp. 2379-2383.
* [SQW17] J. Sun, Q. Qu, and J. Wright. "Complete Dictionary Recovery Over the Sphere I: Overview and the Geometric Picture". In: _IEEE Trans. Inf. Theor._ 63.2 (Feb. 2017), pp. 853-884.
* [Tuk75] J. Tukey. "Mathematics and picturing of data". In: _Proceedings of ICM_. Vol. 6. 1975, pp. 523-531.
* [WM22] J. Wright and Y. Ma. _High-dimensional data analysis with low-dimensional models: Principles, computation, and applications_. Cambridge University Press, 2022.
* [Yin+19] D. Yin, Y. Chen, R. Kannan, and P. Bartlett. "Defending against saddle point attack in Byzantine-robust distributed learning". In: _International Conference on Machine Learning_. PMLR. 2019, pp. 7074-7084.

Response:
## Review

### Summary

This paper addresses the problem of finding approximate second-order stationary points in the presence of outliers in stochastic optimization problems. It introduces a novel algorithm that achieves provable guarantees and applies it specifically to the robust low-rank matrix sensing problem. The algorithm demonstrates dimension-independent accuracy and provides a statistical query lower bound for rank one matrix sensing. Despite the theoretical contributions, the paper has been noted for insufficient numerical evaluations. The reviewers appreciate the clarity of the writing and theoretical backing but suggest improvements in addressing certain counterintuitive results and providing practical evaluations.

### Strengths

- The paper proposes a general result for finding approximate second-order stationary points (SOSP) and applies it to a widely relevant problem in low-rank matrix sensing.
- The algorithm is capable of finding an approximate SOSP in polynomial time, even under a constant proportion of sample corruption.
- The theoretical guarantees are well-supported and offer strong results for finding second-order stationary points in adversarial settings.
- The writing is clear, and the approach is well supported by theoretical analysis.

### Weaknesses

- The results suggest that increasing the sample size does not enhance performance, which is counterintuitive and needs clarification.
- There are concerns about the paper being a combination of previous works without sufficient new theoretical contributions.
- The presentation could be improved by including more details on the algorithms and theoretical results.
- The assumptions in the theorems are seen as restrictive and may not generalize well to other nonconvex optimization problems.

### Questions

- How do the results differ from existing literature on robust matrix sensing?
- Could the analysis be extended to matrix completion?
- What are the practical implications of requiring knowledge of a multiplicative upper bound of certain parameters?
- Can the authors clarify the motivation for using second-order methods over first-order methods in high-dimensional settings?

### Soundness

**Score:** 3

**Description:** Good; the theoretical contributions are sound but require clarification on certain counterintuitive results.

### Presentation

**Score:** 3

**Description:** Good; the paper is generally well-written but could benefit from improved clarity in presenting technical details.

### Contribution

**Score:** 3

**Description:** Good; the contributions are significant in the context of the problem addressed, though questions about originality remain.

### Rating

**Score:** 6

**Description:** Weak Accept; the paper is technically solid and has moderate-to-high impact potential, but it requires some improvements and clarifications.

### Paper Decision

**Decision:** Accept (poster)

**Reasons:** The paper presents original theoretical contributions in the area of robust stochastic optimization, demonstrating a clear understanding of the problem. While the paper's soundness and contribution are generally good, some aspects regarding originality and practical evaluations are less clear. Addressing the weaknesses and questions raised by reviewers will strengthen the paper further.