Kernel based regression with robust loss function via iteratively reweighted least squares
Hongwei Dong Liming Yang



















College of Science, China Agricultural University, Beijing, 100083, China
[cor1]Corresponding author
cauyanglm@163.com




Least squares kernel based methods have been widely used in regression problems due to the simple implementation and good generalization performance. Among them, least squares support vector regression (LS-SVR) and extreme learning machine (ELM) are  popular techniques. However, the noise sensitivity is a major bottleneck. To address this issue, a generalized loss function, called ℓ_s-loss, is proposed in this paper. With the support of novel loss function, two kernel based regressors are constructed by replacing the ℓ_2-loss in LS-SVR and ELM with the proposed ℓ_s-loss for better noise robustness. Important properties of ℓ_s-loss, including robustness, asymmetry and asymptotic approximation behaviors, are verified theoretically. Moreover, iteratively reweighted least squares (IRLS) is utilized to optimize and interpret the proposed methods from a weighted viewpoint. The convergence of the proposal are proved, and detailed analyses of robustness are given. Experiments on both artificial and benchmark datasets confirm the validity of the proposed methods.




Robust regression; Support vector machine; Extreme learning machine; Iteratively reweighted least squares









 




§ INTRODUCTION


Machine learning algorithms can be divided into pattern classification and regression according to the aims. For the task of pattern classification, some deep representation learning methods are quite mature and mainstream <cit.>. However, they have not shown dominance in regression problems. Least squares kernel based  regression is an effective way to this issue <cit.>, and least squares support vector regression (LS-SVR) <cit.> and extreme learning machine (ELM) <cit.> are two representatives of this kinds of method. LS-SVR is an improved regression version of support vector machine <cit.>. SVM is a powerful non-parametric method for classification, whose central idea is to construct two parallel hyperplanes that separate the two classes with maximum margin. ELM is a single-hidden layer feedforward neural network, whose model parameters of hidden nodes are randomly determined. Kernel technique induced strong nonlinear capability ensures their generalization ability. And the solution obtained by solving a set of linear equations gives them good adaptability for large-scale problems, which can be regarded as a key factor for their success.
For the prediction of dataset Z={(x_i,y_i)∈ℝ^n×ℝ}_i=1^N, to get a nonlinear mapping f(·) satisfied f(x)→ y, a classical paradigm of least squares kernel based methods is to pursue the structural risk minimization plus the ℓ_2-loss empirical risk <cit.> minimization (ERM), as follows: 

    min_fτ||f||^p+1/N∑_i=1^Nℓ_2(y_i-f(x_i))

where ℓ_2(y-f(x))=(y-f(x))^2. The first term is the structural risk term, which controls the model complexity to prevent overfitting. p=2, i.e., ridge regression, is the most widely used and effective choice. The second term is the empirical risk term, which determines the fidelity through the least square loss function. τ controls the trade-off. Both LS-SVR and ELM can be converted into the form of Eq.(<ref>), and the difference between them lies in the representation of f(·).
As the most widely used regressor, least squares regression focuses on minimizing the mean square error, i.e., the ERM term with ℓ_2-loss, which is also part of the goals of LS-SVR and ELM. The reason for the effectiveness lies in that, in all Gaussian distribution induced models <cit.>, the empirical mean has an optimal minimax mean square error among all mean estimators. Therefore, LS-SVR and ELM also have the optimal mean of estimation for most times. Not only that, the computational convenience for large-scale problems can be guaranteed because their solutions are obtained through solving a set of linear equations. 
However, x is often contaminated with noise because of erroneous samplings and measurements in practical applications, and the errant observations are usually called outliers <cit.>. Their existence will cause the data distribution change from Gaussian to heavy-tailed, and untraceable biases will be introduced. The reason for this phenomenon can be analyzed from two aspects: From a robust statistics point of view, a bounded gradient function is needed for robust loss functions <cit.> and this requirement is not met by ℓ_2-loss. From a ERM point of view, ℓ_2-loss changes rapidly. Therefore, the loss will be very huge when the value of residual is big, so the ERM may fail to approximate the expected risk <cit.>. 
To handle this problem, robust methods have been widely studied <cit.>. Although they are sub-optimal estimators in theory compared with the ℓ_2 based estimators <cit.>, good noise robustness is their advantage. Yao and Tong proposed a generalized asymmetric ℓ_2-loss to give different weight depending on whether the residual is positive or negative <cit.>. Chen et al. used ℓ_1-loss for SVR and designed a effective Split-Bregman method to get the optimal solution <cit.>. Some studies <cit.> seek robust methods based on M-estimator induced loss functions <cit.>. Steinwart and Christmann <cit.> made a systematic analysis about quantile and a generalized asymmetric loss was proposed to realize kernel based quantile regression. Omer <cit.> reviewed the commonly used losses and proposed a generalized one based on maximum likelihood. Entropy has also been studied and introduced into SVR or ELM for robust regression <cit.>. Recently, to handle more general problems, truncated loss functions have been proposed and proved to be effective <cit.>. The linear rising trend, even more slowly, is what these loss functions have in common, and it is also the reason why they have appealing performance. The above methods aim at modifying loss functions to acquire robustness. Besides, the improvements directly applied to ERM, i.e., truncated minimizations, have also been explored <cit.>. Although achieved promising results, the modified objectives of truncated minimizations are really complicated. Therefore, the optimization cost of these methods is considerable. 
Weighted least squares is also an effective way for robust learning <cit.>. Generally, a weight w∈ [0,1] is considered for all observations. The weights of errant ones should tend to be zero so as to eliminate their adverse impacts. Although the aforementioned techniques are different in implementation, they can be regarded as obtaining robustness in an indirect way, in which the modifications of loss function and objective play the role of proxy. Many studies have proved that there is an inseparable relationship between robust loss fuctions and weighted methods <cit.>. On one hand, robust loss functions can be regarded as reducing the adverse effects of outliers by giving them small weights <cit.>. On the other hand, weighted each sample can also be considered as a special loss function <cit.>. Moreover, typical methods for solving the non-convex robust loss functions, such as difference of convex algorithm <cit.>, concave-convex procedure <cit.> and half-quadratic optimization <cit.>, have been proved to be iterative variants of the weighted methods <cit.>. Therefore, it is possible to build a better robust loss function and solve a robust model from a weighted viewpoint <cit.>.
Inspired by previous works, a novel robust loss function, called ℓ_s-loss, is proposed in this paper. On basis of ℓ_s-loss, two alternatives to the ℓ_2 based methods are constructed for better noise robustness. The proposed methods are optimized and interpreted by iteratively reweighted least squares (IRLS) technique <cit.>. Specifically, the objective of the proposed methods can be solved by optimizing the weighted baselines iteratively, so the suitability for large-scale problems can be guaranteed because they still solve linear systems of equations. Moreover, the effectiveness of ℓ_s-loss can be interpreted from the analysis of the iteratively reweighted process. In addition, the convergence of the proposal are theoretically proved. Experiments on both artificial and benchmark datasets demonstrate the robustness of the proposed methods. In summary, in this paper we make the following contributions:

	
  - A novel ℓ_s loss function is proposed to achieve better performance under the interference of noise.
	
  - Robust methods are constructed by combining ℓ_s-loss and two kernel based regressors.
	
  - IRLS technique is used to optimize and interpret the proposed regressors.
	
  - The validity of proposed methods is demonstrated on artificial and benchmark datasets.

The rest of this paper is organized as follows: Relevant literatures are reviewed in Section <ref>. The proposed methods are introduced in Section <ref>. Section <ref> discusses the robustness. In Section <ref>, experimental results are exhibited. Conclusion and future directions are given in Section <ref>.


§ BACKGROUND
 
We start this section by introducing the least squares kernel based regression, followed by its extension, LS-SVR, ELM, and more generalized IRLS. 


 §.§ Least squares kernel based regression

 
In this part, we concisely review the principles of least squares kernel based regression. Given the training set Z={(x_i,y_i)∈ℝ^n×ℝ}_i=1^N. The commonly used ℓ_2 based methods can be written as the following optimization problem:

    min_f   1/2‖ f‖_2^2+C/2∑_i=1^N ℓ_2(y_i-f(x_i))

where C>0 is a regularization hyperparameter which balances the structural risk and empirical risk. For LS-SVR, the prediction function is f(x)=w^Tϕ(x)+b, where ϕ(·):ℝ^n→ℝ^m is the kernel function which maps the input space into a higher dimensional space, w∈ℝ^m is the weight vector, and b∈ℝ represents the bias. The optimization problem of LS-SVR is usually converted into its dual problem by introducing Lagrangian multiplier α. Eliminating the original variables to obtain the solution by solving the following linear equations: 

    [[      0    1^T;      1 K+1/CE ]]
    [[ b; α ]]=
    [[ 0; Y ]]

where Y=[y_1,y_2,⋯,y_N]^T, 1=(1,1,⋯,1)^T∈ℝ^N, E denotes the N× N identity matrix,  K is the kernel matrix with K_ij=ϕ(x_i)^Tϕ(x_j). Finally, the prediction function of LS-SVR can be written as:

    f(x)=∑_i=1^Nα^*_iK(x,x_i)+b^*.

For Tikhonov regularized ELM <cit.>, f is a single-hidden layer feedforward neural network. It can be written as: f(x)=h(x)β, where h(x)=(h_1(x),h_2(x),⋯,h_L(x)), h_i(x) is the hidden layer function g(α_i,b_i,x) between the input layer and the ith hidden node (α_i, b_i are randomly chosen). β=(β_1,β_2,⋯,β_L)^T is the output weight between hidden and output nodes, and its solution can be expressed as:

    β^*={[ H^T(1/C+HH^T)^-1 Y,                 N<L;  (1/C+H^TH)^-1H^TY,                N≥ L ].

where the two forms are equivalent based on Woodbury identity <cit.>.


 §.§ Iteratively reweighted least squares

 
IRLS is a generalized robust learning paradigm, whose idea is to consider implementing multiple weighted process for better performance. For any loss function ℓ(·), its gradient function ψ(·) and weight function v(·) are defined as:

    ψ(u)=∂ℓ(u)/∂ u,


    v(u)={[ ψ(u)/2u,      u≠0;   ψ'(0),     u=0. ].

Similar to M-estimators, a robust scale estimation can also be considered. However, improper setting of the scale may result in inability to convergence <cit.>. IRLS can be expressed by a sequence of successive minimizers of weighted ℓ_2 based theoretical regularized risk, as follows:

    f_k+1=min_f τ||f||_2^2+E_(X,Y)∼ P[v(Y-f_k(X))(Y-f(X))^2].

To let the sequence {f_k} converge, the following conditions have been proved to be necessary in <cit.>:

  
  v1 v(x) is a non-negative bounded Borel measurable function.
  
  v2 v(x) is an even function.
  
  v3 v(x) is continuous and differentiable, with v'(x)≤ 0 for x>0.

The sequence {f_k} can be used as the solution of the ℓ based theoretical regularized risk. Specifically, the global minimum can be obtained as the limit of the sequence {f_k} with arbitrary initialization if ℓ(·) is convex. Otherwise, the f_k (k→∞), should be a local minimum depending on the initial start. Moreover, if ℓ(·) is a robust loss function, the following requirements <cit.> should be met:

  
  c1 ψ(x) is a measurable, real, odd function.
  
  c2 ψ(x) is continuous and differentiable.
  
  c3 ψ(x) is bounded.
  
  c4 ψ(x) is increasing or strictly increasing.

However, in practical applications, the prior information of distribution P is not available, that is, the expected risk can not be solved. Therefore, empirical risk is used to approximate the expected risk in Eq.(<ref>). Taking LS-SVR as an example, a sequence of successive minimizers of weighted LS-SVR can be expressed as:

    (w_k+1,b_k+1)=min_w,b1/2‖w‖_2^2+C/2∑_i=1^Nv(y_i-(w_k^Tϕ(x_i)+b_k))
    (y_i-(w^Tϕ(x_i)+b))^2.

The convergence and approximability of the sequence {w_k, b_k} will be proved latter. As described above, IRLS can be used to optimize any loss function based regressors, and it can also be used to interpret the robustness of arbitrary loss functions.


§ PROPOSED METHODS

In this section, we firstly introduce the proposed ℓ_s-loss. Then two robust regressors are constructed on basis of ℓ_s-loss, and optimized by IRLS technique. Finally, we prove the convergence of the proposed methods, and show the approximability between the utilized optimization technique and the original problem.


 §.§ The proposed ℓ_s-loss function

As stated before, the robustness of loss functions and regressors can be interpreted from the viewpoint of IRLS. In this subsection, a novel loss function, i.e., ℓ_s-loss, is proposed for better noise robustness. Based on the observation that the sigmoid function (1/(1+exp(-u))) can be adjusted slightly to satisfy the requirement for the gradient function of a robust loss, i.e., c1-c4. Moreover, corresponding weight function meets the convergence conditions v1-v3. Therefore, a novel robust function can be obtained, which is induced from sigmoid function and defined as follows:

    ℓ_s(u)=ln(1+e^λ u)-λ/2u+l_0

where λ, l_0∈ℝ. λ is a hyperparameter used to control the amplitude, and l_0 is a constant to guarantee the loss function through the origin.
 

Remarks. Properties of ℓ_s-loss are highlighted as follows:

  
  - ℓ_s-loss is a convex, continuous and differentiable loss function, due to the smoothness and convexity, it can be optimized efficiently.
  
  - The gradient function of ℓ_s-loss is a bounded, continuous, differentiable and strictly increasing odd function. Bounded gradient function theoretically leads to noise robustness.
  
  - The weight function of ℓ_s-loss is a non-negative even function, and when its independent variable is greater than 0, its derivative is less than 0. This means that the samples with large residuals will be given smalle weights.
  
  - ℓ_s-loss treats the samples with large residuals like ℓ_1-loss, and both are more robust than ℓ_2-loss, because when u→∞, the following limits hold:
  
    lim_u→∞ℓ_s(u)/ℓ_1(u)=λ/2,    lim_u→∞ℓ_s(u)/ℓ_2(u)=0.
 
  
  - ℓ_s-loss can be regarded as a generalized convex loss because of the asymptotic property between ℓ_1 and ℓ_2.


As shown in Fig. <ref>, the height of ℓ_s-loss will rise with the increase of λ. Moreover, the asymptotic property of ℓ_s between ℓ_1 and ℓ_2 can be seen from the right figure. When λ=1 (black solid line), the  shape of ℓ_s-loss is relatively smooth, and the height is lower than ℓ_1-loss. When λ=3 (blue solid line), ℓ_s-loss approximate to ℓ_1-loss, and when λ=5 (magenta solid line), ℓ_s-loss is located between ℓ_1-loss and ℓ_2-loss. The larger value of λ, the sharper shape and the higher amplitude of the ℓ_s-loss.


 §.§ Iteratively reweighted algorithm for ℓ_s based LS-SVR

 
A robust kernel based regressor can be constructed by replacing the ℓ_2-loss of LS-SVR with ℓ_s-loss. Therefore, Eq.(<ref>) is changed to:

    min_w,b   1/2‖w‖_2^2+C/2∑_i=1^Nℓ_s(y_i-(w^Tϕ(x_i)+b)).

As stated above, we solve Eq.(<ref>) by IRLS. Noting that the method based on ℓ_s-loss and solved by IRLS technique is called IRLS-SVR in this paper. For a sequence of minimizers of ℓ_s based weighted LS-SVR, the k+1th iteration problem can be written as:

    min_w,b   1/2‖w‖_2^2+C/2∑_i=1^N v_s(y_i-(w_k^Tϕ(x_i)+b_k))
    (y_i-(w^Tϕ(x_i)+b))^2

where v_s(y_i-(w_k^Tϕ(x_i)+b_k)) represents the value of ℓ_s-loss induced weight function which is computed by the ith sample and results of the kth iteration. Noting v_s(y_i-(w_k^Tϕ(x_i)+b_k)) as v_s(w_k,b_k) and introducing the residual variable ξ, Eq. (<ref>) can be rewritten as follows:

    (w_k+1, b_k+1)=min_w,b      1/2‖w‖_2^2+C/2∑_i=1^Nv_s(w_k, b_k)ξ_i^2 
    
    s.t.       y_i=w^Tϕ(x_i)+b+ξ_i,    i=1,2,⋯,N.

The Lagrangian function can be written as: 

    ℒ(w,b,ξ,α)=1/2‖w‖_2^2+
    C/2∑_i=1^Nv_s(w_k, b_k)ξ_i^2-∑_i=1^Nα_i(w^Tϕ(x_i)
    +b+ξ_i-y_i)

where α>0 is the Lagrangian multiplier. Setting the derivatives of Eq. (<ref>) to zero shows that:

    ∂ℒ/∂w= w-∑_i=1^Nα_i ϕ(x_i)=0,


    ∂ℒ/∂ b= -∑_i=1^Nα_i =0,


    ∂ℒ/∂ξ_i=Cv_s(w_k, b_k)ξ_i-α_i=0,


    ∂ℒ/∂α_i=w^Tϕ(x_i)+b+ξ_i-y_i=0.

Eliminating the variable w and ξ, Eq. (<ref>) to Eq. (<ref>) can be transformed as:

    [[   0 1^T;   1 K+V ]]
    [[ b; α ]]=
    [[ 0; Y ]]

where a weight matrix V replaces the E/C in Eq.(<ref>), which can be defined as:

    V=diag(1/Cv_s,1(w_k, b_k),1/Cv_s,2(w_k, b_k),⋯,1/Cv_s,N(w_k, b_k)),

and initialized to the identity matrix. The convergence solution of Eq.(<ref>) will be used in Eq.(<ref>) for prediction. The iterative optimization process of IRLS-SVR is outlined as Algorithm <ref>.



 §.§ Iteratively reweighted algorithm for ℓ_s based ELM

In this part, the idea of IRLS-SVR is extended to ELM, and another robust method is constructed. Ordinary ELM <cit.> can easily lead to overfitting, so the Tikhonov regularization is used as supplementary. A robust regularized ELM, i.e., IRLS-ELM, can be obtained by replacing the ℓ_2 with ℓ_s, as follows:

    min_β   1/2‖β‖_2^2+C/2∑_i=1^Nℓ_s(y_i-h(x_i)β).

Eq.(<ref>) can also be solved by IRLS. For a sequence of minimizers of ℓ_s based weighted ELM, the k+1th iteration
problem can be written as: 

    β_k+1=min_β      1/2‖β‖_2^2+C/2∑_i=1^Nv_s(β_k)ξ_i^2
    
    s.t.       y_i=h(x_i)β+ξ_i,    i=1,2,⋯,N

where v_s(β_k) can be computed by v_s(y_i-h(x_i)β_k). By introducing the Lagrange multiplier α, the Lagrange function of Eq. (<ref>) can be expressed as:

    ℒ(β,ξ,α)=1/2‖β‖_2^2+C/2∑_i=1^N v_s(β_k)ξ_i^2-∑_i=1^Nα_i(h(x_i)β+ξ_i-y_i).

Then setting the derivatives of Eq. (<ref>) to zero shows that:

    ∂ℒ/∂β= β^T-∑_i=1^Nα_i h(x_i)=0,


    ∂ℒ/∂ξ_i=Cv_s(β_k)ξ_i-α_i=0,


    ∂ℒ/∂α_i=h(x_i)β+ξ_i-y_i=0.

According to <cit.>, the solution of Eq. (<ref>) is

    β_k={[  H^T(1/C+V HH^T)^-1V Y,                     N<L; (1/C+H^T V H)^-1H^TV Y,                    N≥ L ].

where the two forms of β are equivalent based on Woodbury identity, and V is the weight matrix indunced by ℓ_s-loss, which can be initialized to the identity matrix and defined as:

    V=diag(v_s,1((β_k),v_s,2((β_k),⋯,v_s,N((β_k))

The convergence solution of Eq.(<ref>) will be used for prediction as:

    f(x)=h(x)β^*.

The iterative optimization process of IRLS-ELM is outlined as Algorithm <ref>.




 §.§ Convergence and approximability


In this subsection, the convergence and approximability of the proposed methods are discussed. It is worthy noting that related conclusion was mentioned in <cit.> on basis of the regularized theoretical risk. According to the statements of <cit.>, for the ℓ_s based regularized theoretical risk, there exists a f_∞ such that f_k→ f_∞ as k→∞ since the ℓ_s induced weight function satisfies v1-v3. Besides, the convergence solution f^* can approximate the optimal solution of the following objective:

    min_f  τ‖ f‖_2^2+E_(X,Y)∼ P(ℓ_s(Y-f(X))).

We prove that this conclusion still holds when the regularized empirical risk of arbitrary convex loss fucntion ℓ(·) is minimized. Taking LS-SVR with ℓ-loss as an example, Eq.(<ref>) should be modified as:

    min_w  τ‖w‖_2^2+1/N∑_i=1^Nℓ(y_i-w^Tϕ(x_i))

where the bias is deliberately omitted here for convenient. To be consistent, the k+1th segment of IRLS is given:

    w_k+1=min_w  τ‖w‖_2^2+1/N∑_i=1^Nv(y_i-w_k^Tϕ(x_i))(y_i-w^Tϕ(x_i))^2.

Before the proof of convergence, the following representation theorem is given firstly:
 

Lemma 1. Note the map h(x,y) as h_k(x,y)=y-w_k^Tϕ(x). For the result of k+1th iteration, it holds that:

    w_k+1=1/τ1/N∑_i=1^N[v(y_i-w_k^Tϕ(x_i))
    h_k+1(x_i,y_i)ϕ(x_i)].


 

Proof. For the optimization problem of Eq. (<ref>), Fermat Lemma gives the necessary conditions for the objective function to be extreme at some point. Due to w_k+1 is the optimal solution of the kth iteration, the following equation holds:

    ∂/∂w_k+1 [τ‖w_k+1‖_2^2+1/N∑_i=1^Nv(y_i-w_k^Tϕ(x_i))(y_i-w_k+1^Tϕ(x_i))^2]=0.

Expanding the derivative, we have:

    2τw_k+1-2/N∑_i=1^Nv(y_i-w_k^Tϕ(x_i))(y_i-w_k+1^Tϕ(x_i))ϕ(x_i)=0.

The theorem can be proved by replacing y_i-w_k+1^Tϕ(x_i) with h_k+1(x_i,y_i). 

The sequence {w_k} can be proved to be converge with the help of the above representation theorem. Noting the objective of Eq.(<ref>) as R(w), it can be seen that R≥ 0 is obviously true. Therefore, we mainly focus on whether {R(w_k)} is strictly decreasing with the increase of k.
 

Theorem 1. Initialize w_0∈ℝ^m randomly. If the weight function v(·) induced by ℓ(·)-loss satisfying v1-v3, the solution of Eq.(<ref>) holds that w_k→w_∞, as k→∞.
 

Proof. Define a real function U(·), which satisfies U'(z)=ψ(z)=2zv(z). And a real function g(·), which satisfies g(z^2)=U(z). We have 2zv(z)=(g(z^2))'=g'(z^2)· 2z, and g'(z^2)=v(z). Because of v1 and v3, it holds that U'(z)≥ 0 for z≥0, and U'(z)≤ 0 for z<0. Due to v3, the weight function v(·) is decreasing, so the function g(·) is concave. So the inequality g(a)-g(b)≤(a-b)g'(b) holds. Therefore, the difference of objective function between two adjacent iterations can be written as:

    R(w_k+1)-R(w_k)   =τ||w_k+1||_2^2-τ||w_k||_2^2+1/N∑_i=1^N[U(y_i-w_k+1^Tϕ(x_i))-U(y_i-w_k^Tϕ(x_i))] 
       ≤1/N∑_i=1^N[((y_i-w_k+1^Tϕ(x_i))^2-(y_i-w_k^Tϕ(x_i))^2)
    g'((y_i-w_k^Tϕ(x_i))^2)]_R_1+τ||w_k+1||_2^2-τ||w_k||_2^2_R_2.

Based on g'(z^2)=v(z) and difference of two squares, R_1 can be written as:

    1/N∑_i=1^N[v(y_i-w_k^Tϕ(x_i))
    (2y_i-w_k+1^Tϕ(x_i)-w_k^Tϕ(x_i))(w_k^Tϕ(x_i)
    -w_k+1^Tϕ(x_i))].

Substituting (2y_i-2w_k+1^Tϕ(x_i))+(w_k+1^Tϕ(x_i)-w_k^Tϕ(x_i)) for (2y_i-w_k+1^Tϕ(x_i)-w_k^Tϕ(x_i)), and replacing y_i-w_k+1^Tϕ(x_i) with h_k+1(x_i,y_i), R_1 can be written in two parts, i.e., R_11 and R_12, as follows:

    R_11=-1/N∑_i=1^N[v(y_i-w_k^Tϕ(x_i))
    (w_k^Tϕ(x_i)-w_k+1^Tϕ(x_i))^2],


    R_12   =1/N∑_i=1^N[v(y_i-w_k^Tϕ(x_i))
    (w_k^Tϕ(x_i)-w_k+1^Tϕ(x_i))2h_k+1(x_i,y_i)]
       =(w_k-w_k+1)^T1/N∑_i=1^N[v(y_i-w_k^Tϕ(x_i))
    2h_k+1(x_i,y_i)ϕ(x_i)].

Using Lemma 1, Eq.(<ref>) can be transformed as:

    R_12=(w_k-w_k+1)^T2τw_k+1=-2τ||w_k+1||_2^2+2τw_k+1^Tw_k.

Therefore, the following formula holds:

    R(w_k+1)-R(w_k)=R_11+R_12+R_2=R_11-τ||w_k+1||_2^2+2τw_k+1^Tw_k
    -τ||w_k||_2^2=R_11-τ||w_k+1-w_k||_2^2

It is obviously that R_11 is negative. So the set {R(w_k)} is strictly decreasing with the increase of iteration time k. Therefore, w_k→w_∞, as k→∞, and the theorem is proved. 

It can be inferred from Theorem 1 that the proposed methods, i.e., IRLS-SVR and IRLS-ELM, can converge in theory. Next, we focus on the approximability of IRLS, i.e., the degree of approximation between the convergence solution of Eq.(<ref>) and the optimal solution of Eq.(<ref>). Without loss of generality, we still take LS-SVR as an example. The following proposition can be obtained:
 

Proposition 1. The optimal solution of ℓ based LS-SVR can be obtained by solving a sequence of weighted LS-SVR with the weight function v(·) induced by ℓ-loss under arbitrary initialization.
 

Proof. For the set {w}, which satisfies the requirements of convergence and obtained by solving a sequence of Eq.(<ref>), the limit w_∞ must meet the following condition based on Lemma 1,

    w_∞=1/τ1/N∑_i=1^N[v(y_i-w_∞^Tϕ(x_i))
    (y_i-w_∞^Tϕ(x_i))ϕ(x_i)].

A quantitative representation theorem for Eq. (<ref>) with arbitrary convex loss function was proposed in <cit.>, as follows:

    w=-1/2τ1/N∑_i=1^Nδ_iϕ(x_i)

where δ_i=ℓ'(y_i-w^Tϕ(x_i)), ℓ' denotes the derivative with respect to x. Due to ℓ'(u)=ψ(u)=2uv(u), Eq. (<ref>) can be written as:

    w=-1/2τ1/N∑_i=1^N(ψ(y_i-w^Tϕ(x_i))(-1)ϕ(x_i))=1/2τ1/N∑_i=1^N(v(y_i-w^Tϕ(x_i))2(y_i-w^Tϕ(x_i))ϕ(x_i)).

Compare Eq. (<ref>) and Eq. (<ref>), it can be found that the IRLS solution w_∞ satisfies the quantitative representation theorem for Eq. (<ref>). The proposition is proved due to the local optimal solution is certainly the global optimal one of convex optimization, and ℓ(·) is a convex, continuous and differentiable loss function. 

Without loss of generality, LS-SVR acts as an example in the proof, and these conclusions can be directly generalized to ELM. So far we theoretically prove the convergence and approximability of the proposed methods. Convergence is the basic condition of the algorithm. Approximability can help us interpret the robustness of the proposed methods.


§ ROBUST ANALYSIS


In this section, we illustrate the robustness of the proposal. By directly observing the solution process of IRLS to the proposed methods, we can find that the samples which are difficult to predict will be gradually ignored as the iteration proceeds. 
The reason for this phenomenon is that the ℓ_s induced weight function v_s(·), which is defined as: 

    v_s(u)=λ/2u(1/1+exp(-λ u)-1/2),

is a non-negative even function. Therefore, derivative of v_s(·) is negative, which means that for samples with large residuals, their weights will decrease with the increase of the residual and eventually tend to zero. 
In addition to the interpretion of robustness with the help of IRLS, it is also analyzed from theoretical and numerical aspects. Influence function and sensitivity curve are used as the tools of theoretical and numerical analysis, respectively. The results can reflect the robustness of the proposed methods to some extent.


 §.§ Theoretical perspective


As known, the derivative of a function describes its change rate at certain points. It is easy to conclude that the change rate of ℓ_2-loss is very fast because its gradient is linear, which leads to a sharp form and noise sensitive. For robust loss functions, bounded derivatives are necessary <cit.>. From a theoretical point of view, bounded influence function (IF) means that the change of function value caused by noise has an upper limit <cit.>. Influence function of estimator T can be defined as:

    IF=lim_ε→ 0T((1-ε)F+ε H)-T(F)/ε

where F is the main distribution, H is the pollution distribution and ε is the pollution rate. Specifically, for loss function ℓ(·), the above formula can be written as:

    IF=M^-1ℓ'(z_y-f(z_x))z_x

where z=(z_x,z_y) is the polluted point, f is the prediction function, and M is the mean of ∑_iℓ”(y_i-f(x_i))x_i^Tx_i.
Another related conclusion were presented in <cit.>, as follows:

    IF=S^-1(E_F(ℓ'(Y,f(X))ψ(X)))-ℓ'(z_y,f(z_x))S^-1ψ(z_x)

where S(f)=4f/C+E_F((ℓ”(Y,f(X))<ψ(X),f>ψ(X))). By comparing the above two formulas, it can be seen that bounded gradient and kernels (such as radial basis function, RBF) is the foundation of a bounded influence function. According to the representation of ℓ_s-loss, its gradient function can be written as follows:

    ψ_s(u)=λ/1+exp(-λ u)-λ/2.

As stated before, the requirements of c1-c4 should be met if it is a robust loss function. Therefore, one can easily prove that the proposed ℓ_s-loss is a robust loss function from the expression of Eq. (<ref>).



 §.§ Numerical perspective


Even with the above analysis, it is still not easy to measure the robustness of the algorithm. In this part, we use sensitivity curve (SC) to show the robustness visually, which can be seen as finite version of the influence function. From <cit.>, SC at an additional point z_i(x_i,y_i) can be defined as:

    SC(z_i;f)=(f(P)-f(P^i))/1/N

where P is the training set with N samples and P^i=P\ z_i. f(P) and f(P^i) represent the constructed decision functions with and without z_i, respectively. Generally, the additional point z_i may be set as an errant point, i.e., outlier, so that we can see how it affects the model. It can be seen from the definition that SC reflects the impact of outliers  on decision-making, and a smaller value of SC undoubtedly means better robustness. 
Artificial data points with several deliberately added outliers are generated in two different settings. The first data set is obtained from y=sin(u)cos(u^2), u∈[-1,1], and two outliers are z_1(-0.8,-5), z_2(0.8,5). The second data set is obtained from y=15(u^2-1)^2u^4exp(-u), u∈[-1,1], and four outliers are z_1(0,5), z_2(0.1,5), z_3(0.7,5), z_4(0.8,5). Using these data, 2D SC of LS-SVR and IRLS-SVR are drawn to show their robustness visually. According to Eq.(<ref>), under different settings, the SC of each outlier is shown in Figure <ref>. It can be seen that the SC values of the proposal is significantly lower than the one of LS-SVR, which means that the robustness is actually improved.


§ EXPERIMENTS



Experiments on artificial and benchmark datasets are carried out in order to evaluate the effectiveness of proposed methods. Specifically, classical and advanced regressors are chosen for comparison. All the experiments are implemented on a personal computer with Intel(R) Core(TM) i5-3230M CPU with 4 GB RAM, and MATLAB R2014a environment. Root mean square error (RMSE), mean absolute error (MAE) and mean relative error (MRE) are selected as evaluation criterion.


 §.§ Simulation on synthetic data


For synthetic data, we generate the training data from sinc function. Moreover, noise subject to three kinds of  distributions, including Gaussian N(0,0.3^2), Laplacian L(0,1) and χ^2 with 4 degree of freedom, is added to verify the robustness. IRLS-SVR is used in this part to compare with LS-SVR and weighted LS-SVR (WLS-SVR). There are totally 500 training and 300 testing samples, and the noise is only added to the training set. The process are repeated 500 times to reduce randomness. Regression curves are drawn in Figure <ref>.

It can be obviously seen from Figure <ref> that the approximation degree of regression and real curves is relatively high under Gaussian noise. But ℓ_2 based regressor and one-shot weighted are difficult to handle more complicated asymmetric distributed noises. The proposal has better performance because of the stronger capacity of ℓ_s-loss as well as the IRLS solution. 






 §.§ Simulation on benchmark data


We test the proposed methods with several classical regressors on six UCI benchmark datasets <cit.> to illustrate their effectiveness. For IRLS-SVR, LS-SVR <cit.> and weighted LS-SVR (WLS-SVR) <cit.> are chosen for comparison. ELM <cit.> and weighted ELM (W-ELM) <cit.> are used to compare with IRLS-ELM. Information about the used datasets can be found from Table <ref>. All samples are scaled to let the features locate in [0,1] so as to improve the performance. Testing accuracy is obtained by using 10-fold cross validation <cit.>. For LS-SVR and its variants, the hyperparameter C is selected from {2^i|i=-4,⋯,8}, and the rest ones are selected from {2^i|i=-3,⋯,3} by grid search. For ELMs, the selection of C,λ is the same as that of LS-SVR, and the optimal number of hidden nodes L is chosen from {r· N|r=5%,10%,20%,⋯,50%} where N is the number of training samples.  Noise-free and noisy experiments are carried out. For the noisy experiment, 20% training samples are randomly selected to simulate outliers by amplifying their regression values ten times. Besides, experiments are implemented five times and the mean and variance are recorded in order to reduce the randomness. Noise-free experimental results are summarized in Table <ref> for LS-SVRs and Table <ref> for ELMs.
From Tables <ref> and <ref>, one can see that when training without outliers, the accuracy of the proposal is comparable to that of comparing methods. As shown in Tables <ref> and <ref>, when the training set is contaminated by outliers, results of ℓ_2 based methods are unsatisfactory in all datasets which reflects the sensitivity of ℓ_2-loss to outliers. The difference between multiple and one-shot weighted processes are the design of weight functions and the frequency of weighted operations. These two points results in the difference in their performance. Compared with ℓ_2 based methods, the improved ones have more superior performance, and the proposed IRLS-SVR and IRLS-ELM achieve better results in more datasets. It is worhty noting that the value of hyperparameters is not re-selected for noisy experiment, but the value obtained from the previous noise-free experiment is reused. Therefore, such results are acceptable and sufficient to verify the robustness of the proposed methods.



 §.§ Comparison with advanced methods


In order to further verify the performance of the proposed ℓ_s-loss, various advanced LS-SVR based methods, including pinball loss SVR (PSVR) <cit.>, least absolute deviation SVR (LADSVR) <cit.>, robust non-convex LS-SVR (RLSSVR) <cit.>, robust generalized quantile loss based SVR (RQSVR) <cit.> are selected to implement comparative experiments. We select two benchmark datasets, i.e., AutoMPG and Boston Housing, for verification. The experimental results are drawn in Figure <ref>.
In the experiment, we randomly select 5% of the labeled samples from each dataset to choose the optimal hyperparameters. Then 20% labeled samples are used for training and the rest for testing. Similar to before, artificial outliers are added to the training set. The whole process is repeated 50 times to reduce the randomness. As shown in Figure <ref>, the experiments on two datasets show the same conclusion that the proposed method not only has good prediction accuracy, but also has good speed. It is worth noting that compared with the comparing methods, the proposal is the easiest to be implemented. It doesn't need to solve the time-consuming quadratic programming problem or involve advanced optimization methods, such as CCCP for RLSSVR and Split-Bregman iteration for LADSVR.
From the above experimental results including LS-SVR and ELM based regressions, we make the following summary:

	
  - In noise-free experiments, performance of the proposal is close to comparing methods. The reason for such a phenomenon is that the goal of this work is to improve the robustness, and the dataset polluted by noise is the hypothetical application scene. 
	
  - Compared with classical methods, the proposed methods have better performance in most cases although the hyperparameters are not optimized. This shows that the proposed ℓ_s-loss can improve the noise robustness of baselines in a plug-and-play manner.	
	
  - Compared with advanced kernel based regressors, the proposal has at least comparable robustness. We believe that the robustness comes from the proposed ℓ_s-loss, which meets the theoretical requirements of robust loss function. Besides, the running speed of the proposed methods is satisfactory, which is closely related to the good properties of ℓ_s-loss and the efficient solution brought by IRLS.



 §.§ Effect of hyperparameters


In this part, experiments are carried out to reveal the influence of hyperparameters on the proposed methods. Specifically, AutoMPG dataset is used to obtain the RMSE under different hyperparameter combinations. Results of IRLS-SVR are shown in Figure <ref> as 3D histograms, and results of IRLS-ELM can be seen from Figure <ref>.
Figure <ref> shows the experimental results of the proposed IRLS-SVR. It can be seen from the figure that each hyperparameter has a certain effect on the model. When C is fixed, the change of kernel parameter γ has little effect on the model, while a larger value of λ will make the model performance better. When γ is fixed, the larger values of  C and λ are more efficient. When λ is fixed, a medium size of C and a small value of γ are slightly better. In summary, the chosen of λ should be larger, and γ should be smaller, which will make the model performs better.


Figure <ref> shows the experimental results of the proposed IRLS-ELM. It can be concluded that the number of hidden nodes L has a great impact on the performance, and the influence of the remaining ones is relatively small. Specifically, when the value of C is fixed, the change in λ has little effect on RMSE, but a smaller L will significantly enhance the performance. Similarly, the same phenomena can be observed when fixing the value of λ. Besides, the value of C and λ have little effect when the optimal value of L is selected. Therefore, more attention should be paid to search for a applicable number of hidden nodes when determining the value of hyperparameters. However, the number of L is not as large as possible because more hidden nodes will reduce the calculation speed.


§ CONCLUSION


In this paper, we propose a robust ℓ_s loss function, which is continuously and derivable, and its gradient function is bounded and strictly increasing. Simultaneously, devoting the proposed ℓ_s-loss into least squares kernel based regressors and relacing the ℓ_2-loss, two models, i.e., IRLS-SVR and IRLS-ELM, are constructed for better noise robustness. Moreover, IRLS technique is utilized to optimize and interpret the proposed methods. This work starts with the improvement of ℓ_2 loss function, and the effectiveness can be observed by analyzing the weighted process of the model solution. In addition, convergence of the proposal has been proved theoretically. Global optimum can be achieved due to the convexity of the proposed ℓ_s loss function. Thanks to the novel loss function, the proposed methods achieve good results in the experiments on both artificial and benchmark datasets. The effect of hyperparameters is also discussed. 
Although achieves promising results, the difficulty of hyperparameter selection will result in huge training costs. In the future work, we will consider how to combine the proposed methods with hyperparameter optimization to address this issue.
  

Acknowledgements

This work is supported by National Nature Science Foundation of China (Nos. 11471010, 11271367).


Deep
Krizhevsky A, Sutskever I, Hinton GE (2012) ImageNet classification with deep convolutional neural networks.
In: Proceedings of Advances in Neural Information Processing Systems, pp 1097–1105

Lecun2014Backpropagation
Lecun Y, Boser B, Denker JS, Henderson D, Howard RE, Hubbard W, Jackel LD (1989) Backpropagation applied to handwritten zip code recognition. Neural Comput 1(4):541–551
http://dx.doi.org/10.1162/neco.1989.1.4.541
doi:10.1162/neco.1989.1.4.541

Audibert2011Robust
Audibert JY, Catoni O (2011) Robust linear least squares regression. Ann Stat 39(5):2766–2794
http://dx.doi.org/10.1214/11-AOS918
doi:10.1214/11-AOS918

add4
Cheung YM, Zeng H (2009) Local kernel regression score for selecting features of high-dimensional data. IEEE Trans Knowl Data Eng 21(12):1798-1802 
http://dx.doi.org/10.1109/TKDE.2009.23
doi:10.1109/TKDE.2009.23

Suykens2002Least
Suykens JAK, Gestel TV, Brabanter JD, Moor BD, Vandewalle J (2002) Least squares support vector machines. Int J Circuit Theory Appl 27(6):605–615
http://dx.doi.org/10.1002/(SICI)1097-007X(199911/12)27:6<605::AID-CTA86>3.0.CO;2-Z
doi:10.1002/(SICI)1097-007X(199911/12)27:6<605::AID-CTA86>3.0.CO;2-Z

Huang2006Extreme
Huang GB, Zhu QY, Siew CK (2006) Extreme learning machine: Theory and
applications. Neurocomputing 70(1):489–501
http://dx.doi.org/10.1016/j.neucom.2005.12.126
doi:10.1016/j.neucom.2005.12.126

vapniksvm
Vapnik VN (2008) Statistical learning theory. Wiley, New York

Ref16
Bartlett P, Mendelson S (2006) Empirical minimization. Probab Theory Relat Field 135(3):311–334
http://dx.doi.org/10.1007/s00440-005-0462-3
doi:10.1007/s00440-005-0462-3

Ref17
Fama F, MacBeth D, Jackel LD (1973) Risk, return, and equilibrium: Empirical
tests. J Polit Econ 81(3):607–636
http://dx.doi.org/10.1086/260061 doi:10.1086/260061


Catoni2010Challenging
Catoni O (2010) Challenging the empirical mean and empirical variance: A deviation
study. Ann Inst Henri Poincare-Probab Stat 48(4):1148–1185
http://dx.doi.org/10.1214/11-AIHP454
doi:10.1214/11-AIHP454

add1
Kallummil S, Kalyani S (2019) Noise statistics bblivious GARD for robust regression with sparse outliers. IEEE Trans Signal Process 67(2):383-398 
http://dx.doi.org/10.1109/TSP.2018.2883025
doi:10.1109/TSP.2018.2883025

Christmann2007Consistency
Christmann A, Steinwart I (2007) Consistency and robustness of kernel based
regression. Bernoulli 13(3):799–819
http://dx.doi.org/10.3150/07-BEJ5102
doi:10.3150/07-BEJ5102

add3
Huang D, Cabral R, Torre FDl (2016) Robust regression. IEEE Trans Pattern Anal Mach Intell 38(2):363-375 
http://dx.doi.org/10.1109/TPAMI.2015.2448091
doi:10.1109/TPAMI.2015.2448091

Zhang2018
Zhang L, Zhou ZH (2018) ℓ_1-regression with heavy-tailed distributions. in: Proceedings of Advances in Neural Information Processing Systems 

Yao1996Asymmetric
Yao Q, Tong H (2007) Asymmetric least squares regression estimation: A
nonparametric approach. J Nonparametr Stat 6(4):273–292
http://dx.doi.org/10.1080/10485259608832675]
doi:10.1080/10485259608832675]

Chen2017Least
Chen C, Li Y, Yan C, Guo J, Liu G (2017) Least absolute deviation-based robust
support vector regression. Knowledge-Based Syst 131(1):183–194
http://dx.doi.org/10.1016/j.knosys.2017.06.009
doi:10.1016/j.knosys.2017.06.009

Chen2015A
Chen C, Yan C, Li Y (2015) A robust weighted least squares support vector
regression based on least trimmed squares. Neurocomputing 168(30):941–946
http://dx.doi.org/10.1016/j.neucom.2015.05.031
doi:10.1016/j.neucom.2015.05.031

Mangasarian2002Robust
Mangasarian OL, Musicant DR (2002) Robust linear and support vector regression.
IEEE Trans Pattern Anal Mach Intell 22(9):950–955
http://dx.doi.org/10.1109/34.877518
doi:10.1109/34.877518

Huber1
Huber PJ (2014) Robust statistics. Springer, New York
http://dx.doi.org/10.1007/978-3-642-04898-2_594
doi:10.1007/978-3-642-04898-2_594

Huber1964Robust
Huber PJ (1964) Robust estimation of a location parameter. Ann Math Statist 35(1):73–101
http://dx.doi.org/10.1214/aoms/1177703732
doi:10.1214/aoms/1177703732

nipssvm
Christmann A, Steinwart I (2007) How svms can estimate quantiles and the median.
In: Proceedings of Advances in Neural Information Processing Systems, pp 305–312

Omer2017Maximum
Karal O (2017) Maximum likelihood optimal and robust support vector regression with
lncosh loss function. Neural Netw 94(10):1–12
http://dx.doi.org/10.1016/j.neunet.2017.06.008
doi:10.1016/j.neunet.2017.06.008

Ren2018Correntropy
Ren Z, Yang Y (2018) Correntropy-based robust extreme learning machine for
classification. Neurocomputing 313(11):74–84
http://dx.doi.org/10.1016/j.neucom.2018.05.100
doi:10.1016/j.neucom.2018.05.100

Kai2015Outlier
Kai Z, Luo M (2015) Outlier-robust extreme learning machine for regression
problems. Neurocomputing 151(3):1519–1527
http://dx.doi.org/10.1016/j.neucom.2014.09.022
doi:10.1016/j.neucom.2014.09.022

Yang2018
Yang L, Dong H (2018) Support vector machine with truncated pinball loss and its
application in pattern recognition. Chemometrics Intell Lab Syst 177(6):89–99
http://dx.doi.org/10.1016/j.chemolab.2018.04.003
doi:10.1016/j.chemolab.2018.04.003

Yang2020
Yang L, Dong H (2019) Robust support vector machine with generalized quantile loss for classification and regression. Appl Soft Comput 81(8):105483
http://dx.doi.org/10.1016/j.asoc.2019.105483
doi:10.1016/j.asoc.2019.105483

Holland2017Robust
Holland MJ, Ikeda K (2017) Robust regression using biased objectives. Mach Learn 106(4):1–37
http://dx.doi.org/10.1007/s10994-017-5653-5
doi:10.1007/s10994-017-5653-5

Lugosi2016Risk
Lugosi G, Mendelson S (2020) Risk minimization by median-of-means tournaments. J Eur Math Soc 22(3):925-965
http://dx.doi.org/10.4171/JEMS/937
doi:10.4171/JEMS/937

catoni2009
Catoni O (2009) High confidence estimates of the mean of heavy-tailed real random
variables. arXiv:0909.5366

Suykens2002Weighted
Suykens JAK, Brabanter JD, Lukas L, Vandewalle J (2002) Weighted least
squares support vector machines: Robustness and sparse approximation. Neurocomputing 48(10):85–105
http://dx.doi.org/10.1016/s0925-2312(01)00644-0
doi:10.1016/s0925-2312(01)00644-0


Wang2014Robust
Wang K, Zhong P (2014) Robust non-convex least squares loss function for regression
with outliers. Knowledge-Based Syst 71:290–302
http://dx.doi.org/10.1016/j.knosys.2014.08.003
doi:10.1016/j.knosys.2014.08.003

Zhao2010Robust
Zhao Y, Sun J (2010) Robust truncated support vector regression. Expert Syst Appl 37(7):5126–5133
http://dx.doi.org/10.1016/j.eswa.2009.12.082
doi:10.1016/j.eswa.2009.12.082


chen2017neurocom
Chen K, Lv Q, Lu Y, Dou Y (2016) Robust regularized extreme learning machine for regression using iteratively reweighted least squares. Neurocomputing 230(12):345–358
http://dx.doi.org/10.1016/j.neucom.2016.12.029
doi:10.1016/j.neucom.2016.12.029


Akoa2008Combining
Dinh DP, Thi HAL, Akoa F (2008) Combining DCA (DC Algorithms) and
interior point techniques for large-scale nonconvex quadratic programming.
Optim Methods Softw 23(4):609–629
http://dx.doi.org/10.1080/10556780802263990
doi:10.1080/10556780802263990

Yang2016A
Yang L, Qian Y (2016) A sparse logistic regression framework by difference of
convex functions programming. Appl Intell 45(2):241–254
http://dx.doi.org/10.1007/s10489-016-0758-2
doi:10.1007/s10489-016-0758-2

YUILLE2002CCCP
Yuille AL (2002) CCCP algorithms to minimize the Bethe and Kikuchi free
energies: Convergent alternatives to belief propagation. Neural Comput 14(7):1691–1722
http://dx.doi.org/10.1162/08997660260028674
doi:10.1162/08997660260028674

Zhang2013robust
Zhang Y, Sun Y, He R, Tan T (2013) Robust subspace clustering via half-quadratic
minimization. In: Proceedings of IEEE International Conference on Computer Vision, pp 3096–3103

Ran2014Half
He R, Zheng W, Tan T, Sun Z (2014) Half-quadratic-based iterative minimization
for robust sparse representation. IEEE Trans Pattern Anal Mach Intell 36(2):261–275
http://dx.doi.org/10.1109/TPAMI.2013.102
doi:10.1109/TPAMI.2013.102

Feng2016Robust
Feng Y, Yang Y, Huang X, Mehrkanoon S, Suykens JAK (2016) Robust support
vector machines for classification with nonconvex and smooth losses. Neural
Comput 28(6):1217–1247
http://dx.doi.org/10.1162/NECO_a_00837
doi:10.1162/NECO_a_00837

Li2017Sparse
Li C, Zhou S (2017) Sparse algorithm for robust LSSVM in primal space.
Neurocomputing 275:2880–2891
http://dx.doi.org/10.1016/j.neucom.2017.10.011
doi:10.1016/j.neucom.2017.10.011

Xu2016Robust
Xu G, Hu B, Principe JC (2016) Robust C-loss kernel classifiers. IEEE Trans Neural Netw Learn Syst 29(3):510–522
http://dx.doi.org/10.1109/TNNLS.2016.2637351
doi:10.1109/TNNLS.2016.2637351

Lai2013Improved
Lai MJ, Xu Y, Yin W (2013) Improved iteratively reweighted least squares for
unconstrained smoothed ℓq minimization. SIAM J Numer Anal 51(2):927–957
http://dx.doi.org/10.1137/110840364
doi:10.1137/110840364

Green1984Iteratively
Green PJ (1984) Iteratively reweighted least squares for maximum likelihood
estimation, and some robust and resistant alternatives. J R Stat Soc Ser B-Stat Methodol 46(2):149–192
http://dx.doi.org/10.1111/j.2517-6161.1984.tb01288.x
doi:10.1111/j.2517-6161.1984.tb01288.x

irlskbr
Debruyne M, Christmann A, Hubert M, Suykens JAK (2010) Robustness of
reweighted least squares kernel based regression. J Multivar Anal 101:447–463
http://dx.doi.org/10.1016/j.jmva.2009.09.007
doi:10.1016/j.jmva.2009.09.007

add2
Yi S, He Z, Cheung YM, Chen WS (2018) Unified sparse subspace learning via self-contained regression. IEEE Trans Circuits Syst Video Technol 28(10):2537-2550 
http://dx.doi.org/10.1109/TCSVT.2017.2721541
doi:10.1109/TCSVT.2017.2721541


4938676
Deng W, Zheng Q, Chen L (2009) Regularized extreme learning machine. In: Proceedings of IEEE
Symposium on Computational Intelligence and Data Mining, pp 389–395
http://dx.doi.org/10.1109/CIDM.2009.4938676
doi:10.1109/CIDM.2009.4938676

Christmann2008Support
Christmann A, Steinwart I (2008) Support vector machines. Springer, New York
http://dx.doi.org/10.1007/978-0-387-77242-4
doi:10.1007/978-0-387-77242-4

Ref1006
Hampel FR, Ronchetti EM, Rousseeuw PJ, Stahel WA (1986) Robust
statistics: The approach based on influence functions. Wiley, New York
http://dx.doi.org/10.2307/1269782
doi:10.2307/1269782

pinsvm
Huang X, Shi L, Suykens JAK (2014) Support vector machine classifier with pinball loss. IEEE Trans Pattern Anal Mach Intell 36(5):984–997
http://dx.doi.org/10.1109/TPAMI.2013.1786
doi:10.1109/TPAMI.2013.178

Blake1998UCI
Dua D, Graff C (2019) UCI machine learning repository.
(<http://archive.ics.uci.edu/ml>)

Ref1010
Deng N, Tian Y, Zhang C (2012) Support vector machines: Optimization based theory, algorithms and extensions. CRC Press, Boca Raton

Crambes2013
Crambes C, Gannoun A, Henchiri Y (2013) Support vector machine quantile regression approach for functional data: Simulation and application studies. J Multivar Anal 121(11):50–68
http://dx.doi.org/10.1016/j.jmva.2013.06.004
doi:10.1016/j.jmva.2013.06.004


