A Frobenius norm regularization method for convolutional kernels to avoid unstable gradient problem
Pei-Chang Guo
cs.LG stat.ML








A Frobenius norm regularization method for  convolutional kernels to avoid unstable gradient problem


    
Pei-Chang Guo  e-mail:peichang@cugb.edu.cn 

School of Science,
China University of Geosciences, Beijing, 100083, China


    
==============================================================================================================================


Convolutional neural network is a very important model of deep learning. It can help avoid the exploding/vanishing gradient problem and improve the generalizability of a neural network if the singular values of the Jacobian of a layer are bounded around 1 in the training process.  We propose a new  penalty function for a convolutional kernel to let the singular values of the  corresponding transformation matrix are  bounded around 1.  We show how to carry out the gradient type methods. The penalty is about the structured transformation matrix corresponding to a convolutional kernel. This provides a new regularization method about  the weights of convolutional layers.



 Keywords:  regularization, transformation matrix, convolutional kernel, generalizability, unstable gradient.




§ INTRODUCTION


 Convolution without the flip is an important arithmetic in the field of deep learning  <cit.>. Depending on different strides and padding patterns, there are many different forms of convolution arithmetic<cit.>. Without losing generality, in this paper we will consider the same convolution with unit strides.  Our objective is to penalize the kernel K to let the singular values of the corresponding  transformation matrix M  be bounded around 1.


 First we introduce the convolution arithmetic in deep learning,  which is different from the convolution in signal processing. When we refer to convolution in deep learning, only element-wise multiplication and addition are performed.  There is no reverse for the convolutional kernel in deep learning. We use * to denote the convolution arithmetic in deep learning and ⌜·⌝ is to round a number to the nearest integer greater than or equal to that number. If a convolutional kernel is a matrix K∈ℝ^k× k  and the input is a matrix  X∈ℝ^N× N,  each entry of the output Y∈ℝ^N × N  is produced by

    Y_r,s= (K*X)_r,s= ∑_p∈{1,⋯,k}∑_q∈{1,⋯,k} X_r-m+p,s-m+qK_p,q,

where m=⌜ k/2⌝, , and X_i,j=0 if i≤ 0 or i> N, or j≤ 0 or j> N.

In convolutional neural networks, usually there are multi-channels and a convolutional kernel is  represented by a 4 dimensional tensor.
If a convolutional kernel is a 4 dimensional tensor K∈ℝ^k× k× g  × h  and the input is 3 dimensional tensor X∈ℝ^N× N× g, each entry of  the output Y∈ℝ^N × N × h  is produced by

    Y_r,s,c= (K*X)_r,s,c=∑_d∈{1,⋯,g}∑_p∈{1,⋯,k}∑_q∈{1,⋯,k} X_r-m+p,s-m+q,dK_p,q,d,c,

where m=⌜ k/2⌝ and X_i,j,d=0 if i≤ 0 or i> N, or j≤ 0 or j> N.

Each convolutional kernel is corresponding to a linear transformation matrix, and  the output Y=K*X can be reshaped from the multiplication of the transformation matrix with the reshaped X.  We use vec(X) to denote the vectorization of X. If X is a matrix,  vec(X) is the column vector got  by  stacking the columns of  X on top of one another. If X is a tensor,  vec(X) is the column vector got  by  stacking the columns of the flattening of X along the first index (see <cit.> for more on flattening of a tensor).   Given a kernel K, assume M is the linear transformation matrix corresponding to the kernel K, we have
 
    vec(Y)=Mvec(X).

It is desirable to let K satisfy   that vec(Y)≈vec(X), where · denotes a certain vector norm,  when training deep convolutional  neural networks <cit.>.  This is to let the singular values of the linear transformation matrix M corresponding to the kernel K be bounded around 1.


Given a general kernel K whose size is k× k× g  × h and the input whose size is N× N× g, it is not known whether there exists an orthogonal hN^2× gN^2 transformation matrix  corresponding to the k× k× g  × h kernel. It's theoretically difficult to answer this question.  In this paper, we will propose a penalty function for a convolutional kernel to let the singular values of the corresponding transformation matrix be bounded around 1.
The goal is to minimize the following regularization term

    max{|σ_max(M)-1|,|σ_min(M)-1|}

where M is the linear transformation matrix corresponding to kernel K.
But the term (<ref>) is hard to minimize directly. For a general matrix A, people  let the singular values of A be bounded around 1 through penalizing the term  A^TA-I_2, where ·_2 denotes the spectral norm of a matrix, and I is the identity matrix <cit.>. In this paper, we will use M^TM-α I_F^2 as the penalty function, where ·_F denotes the Frobenius norm of a matrix,  to let   the singular values of M be bounded around 1.




As we know, given a matrix, the singular values/eigenvalues are continuous functions depending on the entries of the matrix. We can calculate the partial derivatives of a singular value with respect to the entries, and  let  the singular values of a matrix  be bounded through changing the entries. But the transformation matrix M corresponding to a convolutional kernel is structured, i.e., M has special matrix structure. When changing the entries of M, we should preserve the special structure of M such that the updated M can still correspond to the same size convolutional kernel.  In this paper we will show how to preserve the special structure of M when we minimize M^TM-α I_F^2. The modification on M is actually carried out on a special matrix manifold.

There have been papers devoted to enforcing  the orthogonality or spectral norm regularization on the weights of a neural network <cit.>. The difference between our paper and papers including <cit.> and the references therein  is about how to handle convolutions. They enforce the constraint directly on the h× (gkk)  matrix reshaped from the kernel K∈ℝ^k× k× g  × h,  while we enforce the the  constraint on the transformation matrix M corresponding to the convolution kernel K. In  <cit.>, the authors project a convolutional  layer onto the set of layers obeying a bound on the operator norm of the layer and use numerical results to show this is an effective regularizer. A drawback of the method in <cit.> is that projection can prevent the singular values of the transformation matrix being large but can't avoid the singular values to be too small. In <cit.> a 2-norm regularization method is proposed for convolutional kernels to constrain the singular values of the corresponding transformation matrices. In this paper we propose a Frobenius norm regularization method for convolutional kernels.


The rest of the paper is organized as follows. In subsection <ref>, we will introduce the origin of our problem in deep learning applications.
As we have mentioned, the input channels and the output channels maybe more than one so the kernel is usually represented by a tensor K∈ℝ^k× k× g  × h.
In Section <ref>,  we first consider the case that the numbers of input channels and the output channels are both 1. We propose the penalty function, calculate the partial derivatives and propose the gradient descent algorithm for this case. In Section <ref>, we propose the penalty function and calculate the partial derivatives for the case of multi-channel convolution.
In Section <ref>,  we present numerical results to show the method is feasible and effective. In Section <ref>, we will give some conclusions and point out some interesting work that could be done in future.



 §.§ Applications in deep learning


This problem (<ref>) has important applications in training deep convolutional neural networks.
Convolutional neural network is a very important model of deep learning. A typical convolutional neural network consists of convolutional layers, pooling layers, and fully connected layers. In recent years, deep convolutional neural networks have been applied successfully in many fields, such as face recognition, self-driving cars, natural language understanding and speech recognition. Training the neural networks can be seen as an optimization problem, which is seeking the optimal weights (parameters) by reaching the minimum of loss function on the training data.
This can be described as follows: given a labeled data set {(X_i,Y_i)}_i=1^N, where X_i is the input and Y_i is the output, and a given parametric family of functions 𝔽={f(Θ,X)}, where Θ denotes the parameters contained in the function, the goal of training the neural networks is to find the best parameters Θ such that Y_i≈ f(Θ,X_i) for i=1,⋯,N. The practice is to minimize the so called loss function, i.e., Σ_i=1^NY_i-f(Θ,X_i) in certain measure,  on the training data set.


Exploding and vanishing gradients are fundamental obstacles to effective training of deep neural networks <cit.>. The singular values of the Jacobian of a layer bound the factor by which it changes the norm of the backpropagated signal. If these singular values are all close to 1, then gradients neither explode nor vanish.

On the other hand, it can help improve the generalizability to let  the singular values of the transformation matrix corresponding to a kernel are bounded around 1. Although the training of neural networks can be seen as an optimization problem, but the goal of training is not merely to minimize the loss function on training data set. In fact, the performance of the trained model on new data is the ultimate concern. That is to say, after we find the weights or parameters Θ through minimizing the loss function on training data set, we will use the weights Θ to get a neural network to predict the output or label for the new input data. Sometimes, the minimum on the training model is reached while the performance on test data is not satisfactory. A concept,  generalizability, is used to describe this phenomenon.
 The generalizability can  be improved through reducing the sensitivity of a loss function against the input data perturbation <cit.>.
 


Therefore, to avoid the exploding/vanishing gradient problem and improve the generalizability of a neural network,  the singular values of the Jacobian of a layer are expected to be close to 1 in the training process, which can be formulated as a constrained optimization problem. We divide the  weights Θ of a convolutional neural network into two parts, one is the weights of convolutional layers  and another is the complement.  Assuming the number of convolutional layers is l, we use K_p to denote the kernel for the p-th convolutional layer and M_p to denote the linear transformation matrix corresponding to K_p, and use W_q, 1≤ q≤ m to denote other weight matrices that  belong to other layers. Then researchers in the field of deep learning are interested to consider the following constrained optimization problem:

    min_K_1,K_2,⋯, K_l,W_1,W_2,⋯,W_m1/NΣ_i=1^NY_i-f(K_1,K_2,⋯, K_l,W_1,W_2,⋯,W_m, X_i)


    M_p^TM_p≈ I,    1≤ p ≤ l,       W_q^TW_q≈ I,     1≤ q ≤ m,

where  I denotes the identity matrix.











§ PENALTY FUNCTION FOR ONE-CHANNEL CONVOLUTION


As a warm up, we first focus on the case that  the numbers of input channels and the output channels are both 1. In this case the weights of the kernel are a k× k matrix.
Without loss of generality, assuming the data matrix is N× N, we use a 3× 3 matrix as a convolution kernel to show the associated transformation matrix. Let K be the convolution kernel,

    K=([ k_11 k_12 k_13; k_21 k_22 k_23; k_31 k_32 k_33 ]).

Then the transformation matrix corresponding with the convolution arithmetic is

    M=(
      [  A_0 A_-1    0    0    ⋯    0;  A_1  A_0 A_-1    ⋱    ⋱    ⋮;    0  A_1  A_0    ⋱    ⋱    0;    0    ⋱    ⋱    ⋱ A_-1    0;    ⋮    ⋱    ⋱  A_1  A_0 A_-1;    0    ⋯    0    0  A_1  A_0;      ])

where

    A_0=(
      [ k_22 k_23    0    0    ⋯    0; k_21 k_22 k_23    ⋱    ⋱    ⋮;    0 k_21 k_22    ⋱    ⋱    0;    0    ⋱    ⋱    ⋱ k_23    0;    ⋮    ⋱    ⋱ k_21 k_22 k_23;    0    ⋯    0    0 k_21 k_22 ]),  
    A_-1=(
      [ k_32 k_33    0    0    ⋯    0; k_31 k_32 k_33    ⋱    ⋱    ⋮;    0 k_31 k_32    ⋱    ⋱    0;    0    ⋱    ⋱    ⋱ k_33    0;    ⋮    ⋱    ⋱ k_31 k_32 k_33;    0    ⋯    0    0 k_31 k_32 ]),


    A_1=(
      [ k_12 k_13    0    0    ⋯    0; k_11 k_12 k_13    ⋱    ⋱    ⋮;    0 k_11 k_12    ⋱    ⋱    0;    0    ⋱    ⋱    ⋱ k_13    0;    ⋮    ⋱    ⋱ k_11 k_12 k_13;    0    ⋯    0    0 k_11 k_12 ]).

In this case, the transformation matrix M corresponding to the convolutional kernel K  is a N^2× N^2 doubly  block banded Toeplitz matrix, i.e., a block banded Toeplitz matrix with its blocks are banded Toeplitz matrices. For the details about Toeplitz matrices, please see references  <cit.>. We will let n=N^2 and use 𝒯 to denote the set of all matrices like M in (<ref>), i.e., doubly  block banded Toeplitz matrices with the fixed bandth.

We will use M^TM-α I_F^2 as the penalty function to regularize the convolutional kernel K, and calculate ∂M^TM-α I_F^2/∂ K_p,q, i.e., the partial derivative of Frobenius norm of M^TM-α I versus  each entry K_p,q of the convolution kernel.  Our method provides a new method to calculate the gradient of the penalty function about transformation matrix versus the convolution kernel. People can construct other penalty function about M and get the gradient descent method when training their convolutional networks.
The following lemma is easy but useful in the following derivation.

The partial derivative of square of Frobenius norm of A∈ℝ^n × n with respect to each entry a_ij is ∂A_F^2 /∂ a_ij=2a_ij.


If an entry a_ij of the matrix A ∈ℝ^n× n changes, only the entries belonging to j-th row or j-th volume of the matrix A^TA are affected.
Actually, we have the following lemma.

If we use (A^TA)_s,t to denote the (s,t) entry of the matrix A^TA, then ∂(A^TA)_s,t/ ∂ a_ij is the (s,t) entry of the  matrix
D=A^T(e_ie_j^T)+(e_je_i^T)A, where

    D=( [       0       ⋯       ⋯       0    a_i1       0       ⋯       0;       ⋮       ⋮       ⋮       ⋮    a_i2       ⋮       ⋮       ⋮;       ⋮       ⋮       ⋮       ⋮       ⋮       ⋮       ⋮       ⋮;       0       ⋯       ⋯       0 a_i,j-1       0       ⋯       0;    a_i1    a_i2       ⋯ a_i,j-1  2 a_ij a_i,j+1       ⋯    a_in;       0       ⋯       ⋯       0 a_i,j+1       0       ⋯       0;       ⋮       ⋮       ⋮       ⋮       ⋮       ⋮       ⋮       ⋮;       0       ⋯       ⋯       0    a_in       0       ⋯       0 ]).


We have the following formula from lemma <ref> and lemma <ref>

    1/2∂M^TM-α I_F^2/∂ m_ij   =   1/2∑_s,t=1,⋯,nM^TM-α I_F^2/(M^TM-α I)_s,t∂(M^TM-α I)_s,t/∂ m_ij
       =   ∑_t=1,⋯,n(M^TM-α I)_j,t m_it+∑_s=1,⋯,n(M^TM-α I)_s,j m_is.

For a matrix M∈𝒯,  The value of K_p,q will appear in different  (i,j) indexes. We use Ω to denote this index set, i.e., for each (i,j)∈Ω , we have   m_ij=K_p,q. The chain rule formula about the derivative tells us that, if we want to calculate ∂M^TM-α I_F^2/∂ K_p,q, we should calculate  ∂M^TM-α I_F^2/∂ m_ij for all (i,j)∈Ω and take the sum, i.e.,


    1/2∂M^TM-α I_F^2/∂ K_p,q   =   1/2∑_(i,j)∈Ω∂M^TM-α I_F^2/∂ m_ij
       =   ∑_(i,j)∈Ω(∑_t=1,⋯,n(M^TM-α I)_j,tm_it+∑_s=1,⋯,n(M^TM-α I)_s,jm_is).



We summarize the above results as the following theorem. We can use the formula (<ref>) to carry out the gradient descent method for M^TM-α I_F^2.


Assume  M∈ℝ^n× n is the doubly  block banded Toeplitz matrix corresponding to the one channel convolution kernel K∈ℝ^k× k. If Ω is the set of all indexes (i,j) such that m_ij=K_p,q,  we have

    1/2∂M^TM-α I_F^2/∂ K_p,q= ∑_(i,j)∈Ω(∑_t=1,⋯,n(M^TM-α I)_j,tm_it+∑_s=1,⋯,n(M^TM-α I)_s,jm_is).



Theorem <ref> provides new insight about how to regularize a convolutional kernel K such that singular values of the corresponding transformation matrix are in a bounded interval. We can use the formula (<ref>) to carry out the gradient type methods.
In future, we can construct other penalty functions to let the transformation matrix corresponding to a convolutional kernel have some prescribed property,  and calculate the gradient of the penalty function with respect to the kernel  as we have done in this paper.




§ THE PENALTY FUNCTION AND THE GRADIENT FOR MULTI-CHANNEL CONVOLUTION

In this section we consider the case of multi-channel convolution. First we show the transformation matrix corresponding to multi-channel convolution.
At each convolutional layer,  we have convolution kernel K∈ℝ^k× k× g  × h and the input X∈ℝ^N× N× g; element X_i,j,d is the value of the input unit within channel d at row i and column j. Each entry of  the output Y∈ℝ^N × N × h  is produced by

    Y_r,s,c= (K*X)_r,s,c=∑_d∈{1,⋯,g}∑_p∈{1,⋯,k}∑_q∈{1,⋯,k} X_r-m+p,s-m+q,dK_p,q,d,c,

where X_i,j,d=0 if i≤ 0 or i> N, or j≤ 0 or j> N.
By inspection, vec(Y)=Mvec(X), where M is as follows

    M=([ M_(1)(1) M_(1)(2)        ⋯ M_(1)(g); M_(2)(1) M_(2)(2)        ⋯ M_(2)(g);        ⋮        ⋮        ⋯        ⋮; M_(h)(1) M_(h)(2)        ⋯ M_(h)(g) ]),

and each B_(c)(d)∈𝒯, i.e., B_(c)(d) is a N^2× N^2 doubly  block banded Toeplitz matrix corresponding to the portion K_:,:,d,c of K that concerns the effect of the d-th input channel on the c-th output channel.

Similar as the proof in Section <ref>,  we have the following theorem.

Assume  M  is the structured matrix corresponding to the multi-channel convolution kernel K∈ℝ^k× k× g × h as defined in (<ref>). Given (p,q,z,y), if Ω_p,q,z,y is the set of all indexes (i,j) such that m_ij=k_p,q,z,y,  we have

    1/2∂M^TM-α I_F^2/∂ K_p,q,z,y= ∑_(i,j)∈Ω(∑_t=1,⋯,g*N^2(M^TM-α I)_j,tm_it+∑_s=1,⋯,g*N^2(M^TM-α I)_s,jm_is).


Then the  gradient descent algorithm for the penalty function M^TM-α I_F^2 can be devised, where the number of channels maybe more than one.
We present the detailed gradient descent algorithm for the the penalty function M^TM-α I_F^2 as follows.







§ NUMERICAL EXPERIMENTS

The numerical  tests were performed on a laptop (3.0 Ghz and 16G Memory) with MATLAB R2016b.
We use M to denote the transformation matrix corresponding to the convolutional kernel. The largest singular value and smallest singular value of M (denoted as “σ_max(M) and σ_min(M)), the iteration steps (denoted as “iter") are demonstrated to show the effectiveness of our method. The efficiency is related with the step size λ. According to our experience, the norm of the matrix reshaped from the gradient tensor G∈ℝ^k× k× g  × h in Algorithm <ref> decreases as the number of iteration steps become larger.  Therefore, we can let the step size λ be a small number at first and gradually increase λ.  In our numerical experiments, for Algorithm <ref> we  use  the following dynamic adjustment of step size
   
    if (iter<10)

        λ=1e-5;

    elseif (iter<20)

        λ=1e-4;

    else

        λ=1e-3;

    end

Numerical experiments are implemented on extensive test problems. Our method is effective in letting σ_max(M) and σ_min(M) be approximate to 1.
We present the numerical results for some random generated multi-channel convolution kernels.

We start from a random kernel with each entry normally distributed on [0, 1], i.e., in MATLAB, K is generated by the following command

    rng(1);

    K= randn(k,k,g,h);

We consider kernels of different sizes with 3× 3 filters, namely K∈ℝ^3× 3× g  × h for various values of g,h.  For each kernel, we use the input data matrix of size   20× 20× g.  We then minimize ℛ_1 (K) = M^TM- I_F^2 using Algorithm <ref> and we demonstrate the beneficial effect of  decreasing σ_max(M) while increasing σ_min(M). We present in Figure <ref> the results of 3× 3× 3  × 1, 3× 3× 1  × 3, 3× 3× 3  × 6, and 3× 3× 6  × 3 kernels. In the figures, we have shown the convergence of σ_max(M) (blue  line) on the left axis scale and σ_min(M) (red line) on the right axis scale. From Figure <ref>, we see that at the first about 20 steps, σ_max(M) and σ_min(M) all decrease and then  σ_max(M) become very close to 1. Then in the following steps σ_max(M) become almost unchanged while σ_min(M) increase from smaller than 1 to be very close to 1.
If the standard of constraining the singular values is not very high, one can stop the gradient descent process after first few steps.



We would like to point out, we have used ℛ_1 (K) = M^TM- I_F^2  to do numerical experiments on other random generated examples, including  random kernels with each entry uniformly distributed on [0, 1].
The convergence figures of σ_max(M) and σ_min(M) are similar with the subfigures in  Figure <ref>.

We noticed that in <cit.> a 2-norm regularization method about convolutional kernels is proposed. The difference between 2-norm method in <cit.> and  Frobenius norm method in this paper is needed to investigate further. Now we know that  the computational cost at each iteration step  is the updating of two singular vectors of M^TM-α I for 2-norm method and the computation of the matrix M^TM-α I for Frobenius norm method. The two singular vectors are obtained from two step of power method while the computation of M^TM-α I is done using block matrix multiplication algorithms. We compared the elapsed CPU time of each iteration step for these two  methods and find the time difference  is  little.
The efficiency  of each method, i.e., the needed iteration steps to let σ_max(M) and σ_min(M) be bounded in a satisfying interval, is related with the step size λ. So we can't definitely tell which method is more efficient. But in our extensive numerical experiments,  for each method we test several  different λ, and choose the “most" efficient λ  from these  values of λ according to our experience. We use the “optimal" λ for each method to compare the efficiency. Frobenius method needs less iteration steps to let σ_max(M) and σ_min(M) be in a designated bounded interval than 2-norm method.








§ CONCLUSIONS

In this paper, we provide Frobenius norm method to  regularize the weights of convolutional layers in deep neural networks.
We regularize convolutional kernels to let the singular values of the structured transformation matrix corresponding to a convolutional kernel be close to 1. We give the penalty function and propose the gradient decent algorithm for the convolutional kernel. We see this method is effective and we will improve it in  future.

In future, we will  continue to  devise other forms of penalty functions to constrain the singular values of structured transformation matrices corresponding to convolutional kernels and apply this type of regularization method into the training of neural networks.
Besides, the details about convergence of the gradient descent method could be focused on. For example, how to choose the optimal parameter λ in the algorithm?



§ ACKNOWLEDGEMENTS

The author is grateful to Prof. Qiang Ye for his helpful discussions.
99
brock2017
Andrew Brock, Theodore Lim, James M Ritchie, and Nick Weston. Neural photo editing with introspective
adversarial networks. In ICLR, 2017.
chan2007
 R. Chan and X. Jin, An Introduction to Iterative Toeplitz Solvers, SIAM, Philadelphia, 2007.
 cisse
 Moustapha Cisse, Piotr Bojanowski, Edouard Grave, Yann Dauphin, Nicolas Usunier.
 Parseval Networks: Improving Robustness to Adversarial Examples. In ICML, 2017.
 dumoulin2018
 Vincent Dumoulin, Francesco Visin. A guide to convolution arithmetic for deep learning. ArXiv, 2018.
 golub2012
G.-H. Golub and  C.-F. Van Loan, Matrix computations, Johns Hopkins University Press, Baltimore, 2012.
goodfellow2013
 I. J. Goodfellow, J. Shlens, and C. Szegedy. Explaining and harnessing adversarial examples. In ICLR,
2015.
guo2019
P. Guo, Q. Ye. On Regularization of Convolutional Kernels in Neural Networks, ArXiv 2019.

hochreiter2001
S. Hochreiter, Y. Bengio, P. Frasconi, J. Schmidhuber, et al. Gradient flow in recurrent nets: the difficulty of
learning long-term dependencies, In Field Guide to Dynamical Recurrent Networks, IEEE Press, 2001.
jin2002
 X. Jin, Developments and Applications of Block Toeplitz Iterative Solvers, Science Press, Beijing, 2002.







kova2008
Kovac̆ević, Jelena and Chebira, Amina. An introduction to frames, Now Publishers Inc, Boston,
2008.
miyato2018
Takeru Miyato, Toshiki Kataoka, Masanori Koyama, Yuichi Yoshida. Spectral Normalization for Generative Adversarial Networks. In ICLR, 2018.
sedghi2018
Hanie Sedghi, Vineet Gupta and Philip M. Long. The Singular Values of Convolutional Layers. In ICLR, 2019.
stewart
G. W. Stewart. Matrix Algorithms: Volume II. Eigensystems, SIAM, 2001.
szegedy2014
C. Szegedy, W. Zaremba, I. Sutskever, J. Bruna, D. Erhan, I. J. Goodfellow, and R. Fergus. Intriguing
properties of neural networks. In ICLR, 2014.
tsuzuku2018
Y. Tsuzuku, I. Sato,  and M. Sugiyama.
Lipschitz-Margin Training: Scalable Certification of Perturbation Invariance for Deep Neural Networks.
In NIPS, 2018.
zhang
C. Zhang, S. Bengio, M. Hardt, B. Recht, and O. Vinyals. Understanding deep learning requires rethinking generalization. In ICLR, 2017.
yoshida
Yuichi Yoshida, Takeru Miyato. Spectral Norm Regularization for Improving the Generalizability of Deep Learning, ArXiv 2017.





