Skip to content

# Least Squares SVM

Least Squares Support Vector Machines are a modification of the classical Support Vector Machine, please see Suykens et. al for a complete background.

## LSSVM Regression¶

In case of LSSVM regression one solves (by applying the KKT conditions) the following constrained optimization problem.

\begin{align} & \min_{w,b,e} \ \mathcal{J}_P(w,e) = \frac{1}{2}w^\intercal w + \gamma \frac{1}{2} \sum_{k = 1}^{N} e^2_k \\ & y_k = w^\intercal \varphi(x) + b + e_k, \ k =1, \cdots, N \end{align}

Leading to a predictive model of the form.

$$y(x) = \sum_{k = 1}^{N}\alpha_k K(x, x_k) + b$$

Where the values $\alpha \ \& \ b$ are the solution of

$$\left[\begin{array}{c|c} 0 & 1^\intercal_v \\ \hline 1_v & K + \gamma^{-1} \mathit{I} \end{array}\right] \left[\begin{array}{c} b \\ \hline \alpha \end{array}\right] = \left[\begin{array}{c} 0 \\ \hline y \end{array}\right]$$

Here K is the $N \times N$ kernel matrix whose entries are given by $K_{kl} = \varphi(x_k)^\intercal\varphi(x_l), \ \ k,l = 1, \cdots, N$ and $I$ is the identity matrix of order $N$.

## LSSVM Classification¶

In case of LSSVM for binary classification one solves (by applying the KKT conditions) the following constrained optimization problem.

\begin{align} & \min_{w,b,e} \ \mathcal{J}_P(w,e) = \frac{1}{2}w^\intercal w + \gamma \frac{1}{2} \sum_{k = 1}^{N} e^2_k \\ & y_k[w^\intercal \varphi(x) + b] = 1 - e_k, \ k =1, \cdots, N \end{align}

Leading to a classifier of the form.

$$y(x) = sign \left[ \sum_{k = 1}^{N}\alpha_k K(x, x_k) + b \right]$$

Where the values $\alpha \ \& \ b$ are the solution of

$$\left[\begin{array}{c|c} 0 & y^\intercal \\ \hline y & \Omega + \gamma^{-1} \mathit{I} \end{array}\right] \left[\begin{array}{c} b \\ \hline \alpha \end{array}\right] = \left[\begin{array}{c} 0 \\ \hline 1_v \end{array}\right]$$

Here $\Omega$ is the $N \times N$ matrix whose entries are given by

\begin{align} \Omega_{kl} & = y_{k} y_{l} \varphi(x_k)^\intercal\varphi(x_l), \ \ k,l = 1, \cdots, N \\ & = y_{k} y_{l} K(x_k, x_l) \end{align}

and $I$ is the identity matrix of order $N$.

  1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 // Create the training data set val data: Stream[(DenseVector[Double], Double)] = ... val numPoints = data.length val num_features = data.head._1.length // Create an implicit vector field for the creation of the stationary // radial basis function kernel implicit val field = VectorField(num_features) val kern = new RBFKernel(2.0) //Create the model val lssvmModel = new DLSSVM(data, numPoints, kern, modelTask = "regression") //Set the regularization parameter and learn the model model.setRegParam(1.5).learn()