Achieving Constraints in Neural Networks: A Stochastic Augmented
  Lagrangian Approach
Regularizing Deep Neural Networks (DNNs) is essential for improving
generalizability and preventing overfitting. Fixed penalty methods, though
common, lack adaptability and suffer from hyperparameter sensitivity. In this
paper, we propose a novel approach to DNN regularization by framing the
training process as a constrained optimization problem. Where the data fidelity
term is the minimization objective and the regularization terms serve as
constraints. Then, we employ the Stochastic Augmented Lagrangian (SAL) method
to achieve a more flexible and efficient regularization mechanism. Our approach
extends beyond black-box regularization, demonstrating significant improvements
in white-box models, where weights are often subject to hard constraints to
ensure interpretability. Experimental results on image-based classification on
MNIST, CIFAR10, and CIFAR100 datasets validate the effectiveness of our
approach. SAL consistently achieves higher Accuracy while also achieving better
constraint satisfaction, thus showcasing its potential for optimizing DNNs
under constrained settings.
Diogo Lavado, Cláudia Soares, Alessandra Micheletti
# Achieving Constraints in Neural Networks: A Stochastic Augmented Lagrangian Approach

###### Abstract

Regularizing Deep Neural Networks (DNNs) is essential for improving generalizability and preventing overfitting. Fixed penalty methods, though common, lack adaptability and suffer from hyperparameter sensitivity. In this paper, we propose a novel approach to DNN regularization by framing the training process as a constrained optimization problem. Where the data fidelity term is the minimization objective and the regularization terms serve as constraints. Then, we employ the Stochastic Augmented Lagrangian (SAL) method to achieve a more flexible and efficient regularization mechanism. Our approach extends beyond black-box regularization, demonstrating significant improvements in white-box models, where weights are often subject to hard constraints to ensure interpretability. Experimental results on image-based classification on MNIST, CIFAR10, and CIFAR100 datasets validate the effectiveness of our approach. SAL consistently achieves higher Accuracy while also achieving better constraint satisfaction, thus showcasing its potential for optimizing DNNs under constrained settings.

## 1 Introduction

Deep Neural Networks (DNNs) have shown remarkable success in diverse applications. However, overfitting remains a challenge, which calls for effective regularization techniques. Fixed penalty (FP) methods, like \(L_{1}\) or \(L_{2}\) regularization, are commonly used but lack adaptability since different layers in DNNs may require different amounts of regularization, FP methods induce the same amount of regularization to every layer. In addition, the effectiveness of fixed penalties often depends on hyperparameter tuning, specifically of the regularization coefficients, which can not only be challenging and sensitive to the dataset and network architecture but also make it difficult to explore various parameters due to the time-consuming nature of training DNNs.

The Augmented Lagrangian method (ALM) is an optimization technique designed to solve constrained optimization problems. By viewing DNN training as a constrained optimization problem, we can leverage the Augmented Lagrangian method to enforce regularization as constraints. By iteratively updating the Lagrange multipliers and the penalty parameters, the Augmented Lagrangian method dynamically adapts regularization strengths during the training process. This adaptability allows for a better balance between preventing overfitting and retaining model performance. Adopting ALM to train neural networks under constraints has been widely used, namely, [1] showcased the power of ALM by demonstrating state-of-the-art performance in three NLP benchmarks using a constrained formulation. [2] harnessed ALM to address class-imbalanced binary classification, while [3, 4] applied it to optimal power flow prediction problems and energy domains. For problems involving partial differential equations (PDEs), [5] introduced an ALM approach to enforce physical conservation laws of kinetic PDEs on neural networks. Additionally, [6] proposed an ALM strategy to train Physically informed neural networks (PINNs) by deriving a novel sequence of loss functions with adaptively balanced loss components.

The Alternating Direction Method of Multipliers (ADMM), introduced and popularized by the work of [7], has emerged as a powerful tool for statistics and machine learning challenges involving numerous features or training examples. ADMM adopts a decomposition-alternating approach, where solutions to smaller local subproblems are harmonized to derive a solution to the larger global problem. By combining the benefits of dual decomposition
and augmented Lagrangian methods, ADMM excels in handling combinatorial constraints and supports efficient parallelization. Various applications use ADMM to train DNNs under constraints, for instance, [8; 9] developed an ADMM-based framework against adversarial attacks, while [10] used ADMM for weight pruning in DNNs.

In the context of white-box models, constrained optimization introduces a delicate balance between interpretability and performance. Unlike black-box DNNs, white-box models often involve hard constraints, where parameters hold meaningful interpretations within specific feasible sets. Fixed penalty methods prove inadequate for upholding such hard constraints due to their lack of adaptability and fixed balance between data fidelity and penalties. By formulating DNN training as a constrained optimization problem, SAL and ADMM effectively handle complex hard constraints. The introduction of Lagrange multipliers in SAL and the decomposition-alternating approach in ADMM ensures the enforcement of hard constraints while maintaining model performance.

The contributions of this work are as follows: (1) a novel approach to DNN regularization by formulating the training process as a constrained optimization problem. By leveraging the SAL method, we achieve more flexible and efficient regularization. (2) Our method improves white-box models' performance, ensuring interpretability while effectively handling hard constraints. Experimental results on various datasets validate its effectiveness. (3) We demonstrate the applicability of ADMM for DNN training under constraints. This work presents a comprehensive and versatile regularization framework, paving the way for insights into constrained optimization and interpretable DNNs.

## 2 Theoretical Background

### Augmented Lagrangian Method

Consider a generic optimization problem for an objective function \(F:\Theta\rightarrow\mathbb{R}\) subject to constraints \(C(\theta)=[c_{1}(\theta),\ldots,c_{m}(\theta)]\):

\[\arg\min_{\theta\in\Theta}F(\theta);\quad\text{s.t. }C(\theta)=0. \tag{1}\]

The augmented Lagrangian method (ALM) [11] relaxes the problem in 1 into an unconstrained optimization problem. Specifically, it harmonizes two earlier methods, the quadratic penalty method and the method of Lagrangian multipliers, which suffer from training instability and non-convergence due to the difficulty of convexifying loss functions. ALM augments the Lagrangian function with a quadratic term that helps drive the solution toward the constraint, thus forming the augmented Lagrangian function:

\[L_{\rho}(\theta,\lambda)=F(\theta)+\big{\langle}\lambda,C(\theta)\big{\rangle} +\frac{\rho}{2}\|C(\theta)\|_{2}^{2}, \tag{2}\]

where \(\lambda\) is the Lagrangian multiplier vector and \(\rho\) is the penalty coefficient that controls the trade-off between the objective function and the constraint violation. Similarly to the Fixed Penalty (FP) method, if \(\rho\) is chosen to be too large, then the optimization problem can become stiff and convergence will be very slow. If chosen to be too small, the solution will deviate from the feasible space enforced by the constraints.

The augmented Lagrangian method solves problem (1) by executing the following recursion in \(k\)

\[\min_{\theta\in\Omega}L(\theta^{k},\lambda^{k}), \tag{3}\]

which can be translated into the following steps:

\[\theta^{k+1} \leftarrow\arg\min_{\theta\in\Omega}L_{\rho}(\theta,\lambda^{k}), \tag{4}\] \[\lambda^{k+1} \leftarrow\lambda^{k}+\rho C(\theta^{k+1}) \tag{5}\]

The optimization step on (4) computes the minimum of the augmented Lagrangian cost function with respect to \(\theta\), and the update step on (5) updates the Lagrangian multipliers based on the error in the constraints. The convergence of ALM does not depend on the choice of \(\rho\). A large \(\rho\) leads to faster convergence in terms of needed iterations, however, each iteration becomes more difficult to compute because the optimization step (4) becomes more ill-conditioned. Thus, it becomes crucial to find a penalty that balances these objectives of fast convergence and well-conditioned minimization.
```
1:Initialize \(\theta^{0},\lambda^{0}\gets 0\)
2:for\(k=0,1,\dots,K-1\)do
3:\(\theta^{k+1}\leftarrow\arg\min_{\theta}L_{\rho}(\theta,\lambda^{k})\)
4:\(\lambda^{k+1}\leftarrow\lambda^{k}+\rho C(\theta^{k+1})\)
5:endfor
```

**Algorithm 1** Deterministic Augmented Lagrangian Method

### Alternating Direction Method of Multipliers

The Alternating Direction Method of Multipliers (ADMM) [11; 7] is particularly effective for solving problems that can be decomposed into smaller, simpler subproblems, such as those arising in machine learning and signal processing. ADMM splits the primal variables into two sets, one with respect to the primal function \(F\) and the other to the constraints. Thus, we cast the problem in (1) into a new problem via variable splitting:

\[\arg\min_{\theta\in\Omega,\mu\in\Omega} F(\theta)\ +C(\mu)\] (6) s.t. \[\theta=\mu.\]

The augmented Lagrangian function for (6) is now defined as:

\[L_{\rho}(\theta,\mu,\lambda) =F(\theta)+C(\mu)+\left\langle\lambda,(\theta-\mu)\right\rangle+ \frac{\rho}{2}\|\theta-\mu\|_{2}^{2}\] \[=F(\theta)+C(\mu)+\frac{\rho}{2}\|\theta-\mu+u\|_{2}^{2}, \tag{7}\]

where \(u=\lambda/\rho\) is the scaled Lagrangian multiplier vector. To determine \(u\), we follow the strategy of the augmented Lagrangian algorithm with the following iteration

\[(\theta^{k+1},\mu^{k+1}) \leftarrow\arg\min_{(\theta,\mu)}L_{\rho}(\theta,\mu,\lambda^{k}) \tag{8}\] \[\lambda^{k+1} \leftarrow\lambda^{k}+\rho(\theta^{k+1}-\mu^{k+1}) \tag{9}\]

Then, the method of alternating minimization lets us achieve the following equivalent algorithm

\[\theta^{k+1} \leftarrow\arg\min_{\theta}F(\theta)+\frac{\rho}{2}\|\theta-\mu^{k }+u^{k}\|_{2}^{2} \tag{10}\] \[\mu^{k+1} \leftarrow\arg\min_{\mu}C(\mu)+\frac{\rho}{2}\|\theta^{k+1}-\mu+u^ {k}\|_{2}^{2}\] (11) \[u^{k+1} \gets u^{k}+\rho(\theta^{k+1}-\mu^{k+1}). \tag{12}\]

Since alternating minimization can be seen as a form of coordinated descent [12], this algorithm converges to the desired solution of equation (6).

## 3 Related Work

### Fixed Penalty Method

Fixed penalty (FP) methods attempt to convert the original optimization problem in (1) into an unconstrained optimization problem by augmenting the loss function with a penalty term to handle constraints:

\[\min_{\theta\in\Omega} F(\theta)+\rho\|C(\theta)\|_{2}^{2}, \tag{13}\]
where \(F\) denotes the loss function, \(\theta\) represents the trainable parameters of the model, and \(\rho>0\) serves as the penalty parameter, balancing the trade-off between data fidelity and constraint enforcement during the training process. However, the effectiveness of this method relies heavily on manual fine-tuning of the penalty parameter \(\rho\), as an inappropriate value can lead to an unstable optimization process. Moreover, the FP approach lacks adaptability since it induces the same amount of regularization across all layers of the DNN, regardless of their specific requirements. This lack of adaptability can hinder its performance on complex tasks with varying regularization needs. Additionally, the FP method does not guarantee constraint satisfaction below a certain threshold of interest, \(\|C(\theta)\|_{2}^{2}\leq\epsilon\). This limitation further motivates the search for more robust and efficient regularization methods for training DNNs.

### Stochastic Augmented Lagrangian (SAL)

The Stochastic Augmented Lagrangian (SAL) applies the Augmented Lagrangian Method (ALM) to train neural networks and has garnered attention for its effectiveness in handling constrained optimization problems. [1] demonstrated the power of SAL in achieving state-of-the-art performance in three Natural Language Processing (NLP) benchmarks, while [2] utilized SAL to address class-imbalanced binary classification, improving performance on imbalanced datasets. In energy domains, [3; 4] successfully applied SAL to optimal power flow prediction problems. In the context of Partial Differential Equations (PDEs), [5] employed SAL to enforce physical conservation laws of kinetic PDEs on neural networks. Additionally, [6] proposed a SAL strategy for training Physically Informed Neural Networks (PINNs) with adaptively balanced loss components.

Although these approaches have achieved state-of-the-art results in their respective domains, we recognize that the use of stochastic mini-batch gradients in machine learning problems calls for modifications to the original ALM algorithm. In our work, we draw inspiration from the research of [13], where they address this setting by designing a SAL algorithm that adapts to stochastic mini-batch gradients. While [13] applied their SAL strategy to train DNNs under physical constraints, our work aims to extend and optimize this approach for training DNNs under general constraints and regularization.

### Stochastic ADMM

Stochastic ADMM (S-ADMM) extends the traditional ADMM algorithm to handle large-scale optimization problems with noisy data and a large number of constraints and variables. In S-ADMM, the primal and dual variables are updated following the traditional ADMM strategy, i.e., the steps (8) and (9), but with an approximated augmented Lagrangian function given by [14; 15]:

\[\hat{L}_{\rho,k}(\theta,\mu,u)= F(\theta^{k};\xi^{k+1})+\partial F(\theta^{k};\xi^{k+1})^{T}( \theta-\theta^{k})+ \tag{14}\] \[C(\mu)+\frac{\rho}{2}\|\theta-\mu+u\|_{2}^{2}+\frac{\|\theta- \theta^{k}\|^{2}}{2\eta^{k}}.\]

Here, \(\xi^{k+1}\) is a random sample drawn from an unknown distribution, \(\rho\) is the penalty parameter, and \(\eta^{k+1}\) is the step size. The S-ADMM algorithm is particularly useful when dealing with complex and large-scale optimization problems involving noisy data.

## 4 Methodology

### Stochastic Augmented Lagrangian (SAL)

Augmented Lagrangian methods typically adopt a two-level nested loop structure. The inner problem (4) is solved using conventional unconstrained optimization methods, while the outer loop updates the Lagrangian multipliers and the penalty factor based on the constraint violation of the inner loop solution. In their work, the authors of [13] propose a Stochastic Augmented Lagrangian (Aug-Lag) method tailored for DNN training presented in Algorithm 2. The inner loop of Aug-Lag involves using SGD to solve the unconstrained optimization problem. Unlike traditional ALMs that rely on dynamic convergence tolerances, Aug-Lag iterates through the entire training dataset once per outer iteration. In the outer loop, Aug-Lag accepts updates to the Lagrange multipliers whenever the SGD solution achieves a sufficient decrease in the constraint violation. The multiplier update remains unchanged from the conventional Aug-Lag method. The penalty parameter increases with a fixed factor when the SGD solution fails to satisfy the sufficient decrease criteria. Lastly, Aug-Lag makes use of learning rate decay, but it is reset to its initial value in each outer iteration. This ensures independent decay rates for each constructed training subproblem, preventing the model from stagnating.
### Constraints in Neural Networks

Deep Neural networks are often subject to constraints in order to impose certain conditions on the parameters of the model or to ensure desirable properties. Constraints in neural networks can be broadly categorized into hard and soft constraints. Hard constraints are strict conditions that must be satisfied for the model to be valid. Violating hard constraints typically renders the model unsuitable for the task, this is of particular importance in white-box models where parameters are meaningful under specific domain values. On the other hand, soft constraints are more flexible and allow some level of violation. For instance, \(L_{1}\) regularization is a soft constraint since we do not want a model to strictly satisfy it, rather it helps the model traverse the loss landscape in a better direction

Several types of constraints can be imposed on neural networks, we focused on the following:

\(L_{1}\) **and \(L_{2}\) Norm Constraints:**\(L_{1}\) and \(L_{2}\) norm constraints are commonly used to control the magnitude of model parameters. \(L_{1}\) norm constraints, \(\|\mathbf{w}\|_{1}\), enforce sparsity by encouraging many parameters to be exactly zero. \(L_{2}\) norm constraints, \(\|\mathbf{w}\|_{2}^{2}\), promote weight decay, effectively penalizing large parameter values. Both norms help prevent overfitting and can lead to more robust models.

**Orthogonality Constraints:** Orthogonality constraints enforce that certain weight matrices are orthogonal [16]:

\[\|\mathbf{W}^{\top}\mathbf{W}-\mathbf{I}\|_{F}^{2} \tag{15}\]

where \(\mathbf{W}\) is the weight matrix and \(\mathbf{I}\) is the identity matrix. This property is often beneficial in tasks involving feature extraction or when interpretability is essential.

**Non-Negativity Constraints:** Non-negativity constraints ensure that parameters remain non-negative throughout the training process. This is particularly useful in scenarios where negative values are not meaningful, such as distribution parameters.

\[(-w_{i})_{+},\text{ for all }i, \tag{16}\]

where \((h)_{+}=\max(0,h)\) is the Rectified Linear Unit (ReLU) and \(w_{i}\) represents a model parameter.
### Constraint Violation Metric

We introduce a metric designed to gauge constraint violation within DNNs. The purpose of this metric is to quantitatively evaluate the degree to which the specified constraints are breached by the model. We focus on assessing the constraint violation at the inception of the DNN and then tracing its evolution as training progresses. Given a set of constraints \(\mathcal{C}\) imposed on the DNN, we present the constraint violation metric as follows: Let \(\mathbf{\theta}\) represent the trainable parameters of the DNN at the commencement of training, and \(C(\mathbf{\theta})\) indicate the vector of constraint violation values for each constraint within \(\mathcal{C}\). The constraint violation metric is defined as:

\[\text{Constraint Violation (CV)}=\left(\sum_{i=1}^{m}|C_{i}(\mathbf{\theta})|^{p} \right)^{\frac{1}{p}}, \tag{17}\]

where \(m\) is the number of constraints within \(\mathcal{C}\), and \(p\) stands as a hyperparameter controlling the metric's sensitivity. The works of [13] and [4] set \(p=2\) to assess constraint violation. While this is suitable for hard constraints, alternative \(L_{p}\) norms might be better suited for certain soft constraints. For instance, \(L_{1}\) regularization shouldn't be strictly satisfied by a DNN; therefore, \(L_{\infty}\) could be a more appropriate choice for such a constraint. This is especially pertinent when different architectures exhibit varying parameter counts, rendering the comparison of constraint violation measures unfeasible.

In order to measure the improvement of the constraint violation across different architectures and stages, we introduce an extension of the metric:

\[\text{CV(e)}=CV_{e}/CV_{0}, \tag{18}\]

where \(CV_{i}\) signifies the constraint violation at step/epoch \(i\), and as such, CV(\(i\)) denotes the progression of constraint violation relative to the initial CV prior to training. Thus, the constraint violation metric proposed here is designed to provide a summary of the adherence of DNNs to the imposed constraints. The \(L_{p}\) norm allows us to control the sensitivity of the metric, making it adaptable to different types of constraints and model architectures.

## 5 Experiments

### Experimental Setup

Our experimentation includes diverse neural network architectures, such as ResNet13 [17], VGG11 [18], CNN, and GENEOnet [19]. These architectures underwent evaluations under various constraints, including \(L_{1}\) and \(L_{2}\) regularization, orthogonality, and non-negativity. We examined datasets like CIFAR-10, CIFAR-100, and MNIST for image classification. The datasets were augmented through isomorphic transformations (i.e., translations, rotations, and inversions), with data normalized as a preprocessing step. The data fidelity function \(F\) used was the cross-entropy loss. Utilizing a batch size of 128, we explored learning rates ranging from 0.001 to 0.00001, employing Adam and SGD optimizers. Additionally, for white-box models, the L-BFGS optimizer was employed, capitalizing on their limited parameter count. Our methodology encompassed different training iterations, where we contrasted the fixed penalty method (FP) with different penalty coefficient combinations \(\rho_{i}\in[0.0001,0.1]\), against the stochastic augmented Lagrangian method (SAL) outlined in Algorithm 2 and stochastic ADMM (S-ADMM) [14]. The penalty parameter \(\rho\) in SAL and S-ADMM varied in each experiment, although this variation did not influence method performance, given the progressive adaptation of the penalty during training, corroborated by [13]. Finally, our code is openly accessible and seamlessly integratable with existing architectures. Both the SAL and S-ADMM implementations are programmed using PyTorch and work as wrappers for data fidelity losses. The formulation of constraints is straightforward, rendering the incorporation of SAL and S-ADMM into ongoing projects a straightforward process.

### Numerical Results

In this section, we conduct a comprehensive evaluation of different Deep Neural Networks (DNNs) across the MNIST, CIFAR10, and CIFAR100 datasets, employing different constraint enforcement methods. Regarding the MNIST dataset (Table 1), the Stochastic Augmented Lagrangian (SAL) and Stochastic Alternating Direction Method of Multipliers (S-ADMM) consistently outperform the Fixed Penalty (FP) method. SAL, specifically, demonstrates an optimal trade-off between data fidelity and constraint adherence in both the CNN and GENEOnet architectures. The selected models possess a modest number of parameters (approximately 100K each), and the exclusion of more intricate architectures from testing is due to their suitability for modeling the MNIST challenge. That is, the chosen architectures
emphasize the necessity for regularization methods to strike a balance between constraint application and data fidelity. For the white box model, GENEOnet, the enforcement of non-negativity on its parameters is essential for interpretability. All methods successfully enforce this constraint throughout training. Nonetheless, FP requires a substantial penalty coefficient \(\rho_{i}\) to achieve this, potentially diverting attention from the data fidelity term and leading to suboptimal performance compared to other methods. On the CIFAR100 dataset (Table 2) our evaluation encompasses larger networks with millions of parameters. This is crucial because soft constraints such as \(L_{1}\) are applied to a greater number of parameters, introducing a more delicate equilibrium between constraint enforcement and data fidelity within the loss function. Nonetheless, SAL consistently achieves the highest Accuracy for both ResNet13 and VGG11, surpassing FP and S-ADMM. In this context, SAL does not lead to significant improvements in constraint enforcement over the other methods, instead, it achieves a balanced trade-off between data-fidelity and constraint enforcement. Notably, the performance of S-ADMM consistently keeps pace with SAL, demonstrating competitive Accuracy and constraint enforcement. This underscores that, despite a slightly relaxed constraint adherence, S-ADMM effectively achieves competitive predictive performance. In scenarios involving parallel optimization paradigms, S-ADMM emerges as a strong contender and may indeed be the method of choice.

In the case of the CIFAR10 dataset (Table 3), SAL remains the most effective regularization method in terms of Accuracy, consistently outperforming FP and S-ADMM across all evaluated architectures. Notably, as we delve into larger networks, both SAL and S-ADMM demonstrate significant performance advantages over the FP method. It's worth highlighting that FP exhibits a higher standard deviation in Accuracy compared to the other methods, suggesting greater instability across different penalty configurations, underscoring the limitations of fixed penalty methods. SAL excels by securing the top CV\({}_{p=2}\) score in the VGG11 and CNN models, substantially outperforming FP and S-ADMM. This compelling result, combined with SAL's superior Accuracy, showcases its unique ability to strike an optimal balance between data fidelity and constraint enforcement. Moreover, while FP records the best average CV\({}_{p=2}\) score in the ResNet13 and GENEOnet architectures, this achievement comes at a noticeable cost in terms of Accuracy. This observation reinforces the notion that SAL excels in finding the optimal equilibrium between constraint imposition and data fidelity, making it a promising choice for various deep learning tasks and architectures.

\begin{table}
\begin{tabular}{l|l|l|l l l} \hline \hline DNN & Constraints & Method & CV(-1) & CV\({}_{p=2}\) & Accuracy \\ \hline \multirow{3}{*}{ResNet13} & \(L_{1}\); & FP & **0.11** (\(\sigma\) **0.04**) & **330.5** (\(\sigma\) **1.4**) & 0.72 (\(\sigma\) 0.02) \\  & Orthogonality; & SAL & 0.14 (\(\sigma\) 0.02) & 363.4 (\(\sigma\) 30.1) & **0.77** (\(\sigma\) **0.01**) \\  & & S-ADMM & 0.13 (\(\sigma\) 0.01) & 346 (\(\sigma\) 31.9) & 0.75 (\(\sigma\) 0.01) \\ \hline \multirow{3}{*}{VGG11} & \(L_{1}\); & FP & 0.467 (\(\sigma\) 0.002) & 295.3 (\(\sigma\) 1.4) & 0.72 (\(\sigma\) 0.03) \\  & Orthogonality; & SAL & **0.461** (\(\sigma\) **0.001**) & **294.2** (\(\sigma\) **0.85**) & **0.73** (\(\sigma\) **0.02**) \\ \cline{1-1}  & & S-ADMM & 0.47 (\(\sigma\) 0.01) & 296 (\(\sigma\) 0.92) & 0.72 (\(\sigma\) 0.01) \\ \hline \hline \end{tabular}
\end{table}
Table 2: Quantitative results of ResNet13 and VGG11 on CIFAR100, where CV(-1) indicates the constraint violation at the end of training relative to the initial network initialization, while CV\({}_{p=2}\) represents the constraint violation \(L_{2}\) norm at test time. Evaluation of other DNNs was omitted due to the problem’s simplicity for networks like VGG11 and ResNet13. The reported results are the averages across 100 runs for each method, encompassing different hyperparameter configurations.

\begin{table}
\begin{tabular}{l|l|l|l l l} \hline \hline DNN & Constraints & Method & CV(-1) & CV\({}_{p=2}\) & Accuracy \\ \hline \multirow{3}{*}{CNN} & \(L_{2}\); & FP & **0.72** (\(\sigma\) **0.06**) & 3.2K (\(\sigma\) 7.6) & 0.70 (\(\sigma\) 0.02) \\  & Orthogonality; & SAL & 0.73 (\(\sigma\) 0.03) & **2.0K** (\(\sigma\) **15.4**) & **0.75** (\(\sigma\) **0.01**) \\  & & S-ADMM & 0.73 (\(\sigma\) 0.04) & 2.7K (\(\sigma\) 115) & 0.74 (\(\sigma\) 0.01) \\ \hline \multirow{3}{*}{GENEOnet} & \(L_{2}\); & FP & 0.72 (\(\sigma\) 0.07) & 2.7K (\(\sigma\) 152) & 0.80 (\(\sigma\) 0.1) \\  & Non-Negativity; & SAL & 0.74 (\(\sigma\) 0.04) & 2.4K (\(\sigma\) 30.1) & 0.82 (\(\sigma\) 0.01) \\ \cline{1-1}  & Orthogonality; & S-ADMM & **0.72** (\(\sigma\) **0.03**) & **1.7K** (\(\sigma\) **103**) & **0.83** (\(\sigma\) **0.02**) \\ \hline \hline \end{tabular}
\end{table}
Table 1: Quantitative results of CNN and GENEOnet on the MNIST Dataset. CV(-1) indicates the constraint violation at the end of training relative to the initial network initialization, while CV\({}_{p=2}\) represents the constraint violation \(L_{2}\) norm at test time. Evaluation of other DNNs was omitted due to the problem’s simplicity for networks like VGG11 and ResNet13. The reported results are the averages across 100 runs for each method, encompassing different hyperparameter configurations.
## 6 Conclusions

This work addresses the imperative need for effective Deep Neural Network (DNN) regularization to counter overfitting and enhance generalization. Conventional fixed penalty methods exhibit limitations in adaptability and sensitivity to hyperparameters. To mitigate these issues, we present an innovative approach that formulates DNN training as a constrained optimization problem, prioritizing data fidelity minimization while treating regularization terms as essential constraints. This conceptual foundation paves the way for the application of the Stochastic Augmented Lagrangian (SAL) method, introducing a dynamic and efficient regularization strategy. Notably, our approach's benefits extend beyond black-box regularization, demonstrating substantial enhancements in white-box models subject to stringent weight constraints for interpretability. Empirical validation across diverse datasets, encompassing image classification benchmarks such as MNIST, CIFAR10, and CIFAR100, underscores the efficacy of our methodology. SAL consistently outperforms in terms of Accuracy and constraint enforcement, indicating its potential to optimize DNNs under constraint-driven scenarios. In essence, our study introduces a pragmatic alternative to fixed penalty methods, emphasizing SAL's adaptability and performance improvement. Empirical results support our approach, indicating improved performance and interpretability in DNNs.
