The theoretical framework relies on several significant assumptions:
- **Types of Features:** The paper assumes only two types of features: robust and non-robust, both of which contribute meaningfully to model predictions. However, this simplification may not fully capture the complexity of real data, as observed in [1], which categorizes features into robust/non-robust and useful/non-useful, based on their contribution to the model’s predictions. The set of non-robust features could be infinite, whereas this paper imposes a limit.
- **Orthogonality Assumption:** The core theorem relies on the assumption that features are orthogonal. However, in practical settings, features are not typically orthogonal. The proofs, particularly of Propositions 3.1 and 3.2 (provided in Appendix C), depend on this orthogonal assumption by considering robust features $(\(u_y\))$ orthogonal to non-robust features $(\(v_y\))$.
- **Data Construction Assumption:** Each data point $\(X\)$ is composed of multiple patches $\(\{x_1, \dots, x_P\}\)$, where each patch represents either a robust or non-robust feature (Definition 2.2, Step 3), with certain scaling and randomness. These features are modeled in the input space, but to satisfy the strict assumptions made in the paper, it would seem more appropriate to consider them in the latent space.
- **Definitions of Robust and Non-Robust Features (Assumption 2.3):**
- Robust features are defined as stronger than non-robust features.
- Non-robust features are defined as denser than robust features.
- However, these definitions lack interpretability, which would be beneficial for a more complete understanding.
- Under this assumption, standard training picks up denser features, while adversarial training picks up stronger features, assumed to be robust features.
- The paper lacks experiments to demonstrate the existence of these properties (strength and density) in real-world datasets.
- However, these definitions lack interpretability, which would be beneficial for a more complete understanding.