Home > A Brief Introduction to Scientific Data Mining:

A Brief Introduction to Scientific Data Mining:

Direct Kernel Least-Squares Support Vector Machines with Heuristic Regularization 

Mark J. Embrechts

Department of Decision Sciences and Engineering Systems

Rensselaer Polytechnic Institute, Troy, NY 12180


AbstractThis paper introduces least squares support vector machines as a direct kernel method, where the kernel is considered as a data pre-processing step. A heuristic formula for the regularization parameter is proposed based on preliminary scaling experiments. 



A. One-Layered Neural Networks for Regression


      A standard (predictive) data mining problem is defined as a regression problem for predicting the response from descriptive features. In order to do so, we will first build a predictive model based on training data, evaluate the performance of this predictive model based on validation data, and finally use this predictive model to make actual predictions on a test data for which we generally do not know (or pretend not to know) the response value.

      It is customary to denote the data matrix asand the response vector as. In this case, there are n data points and m descriptive features in the dataset. We would like to inferfromby induction, denoted as, in such a way that our inference model works not only for the training data, but also does a good job on the out-of-sample data (i.e., validation data and test data).  In other words, we aim to build a linear predictive model of the type:


      The hat symbol indicates that we are making predictions that are not perfect (especially for the validation and test data). Equation (1) is the answer to the question “wouldn’t it be nice if we could apply wisdom to the data, and pop comes out the answer?” The vectoris that wisdom vector and is usually called the weight vector in machine learning.

      There are many different ways to build such predictive regression models. Just to mention a few possibilities here, the regression model could be a linear statistical model, a Neural Network based model (NN), or a Support Vector Machine (SVM)[1-3] based model. Examples for linear statistical models are Principal Component Regression models (PCR) and Partial-Least Squares models (PLS). Popular examples of neural network-based models include feedforward neural networks (trained with one of the many popular learning methods), Sef-Organizing Maps (SOMs), and Radial Basis Function Networks (RBFN). Examples of Support Vector Machine algorithms include the perceptron-like support vector machines (SVMs), and Least-Squares Support Vector Machines (LS-SVM), also known as kernel ridge regression. A straightforward way to estimate the weights is outlined in Equation (2).


Predictions for the training set can now be made forby substituting (2) in (1):


      Before applying this formula for a general prediction proper data preprocessing is required. A common procedure in data mining to center all the descriptors and to bring them to a unity variance. The same process is then applied to the response. This procedure of centering and variance normalization is known as Mahalanobis scaling. While Mahalanobis scaling is not the only way to pre-process the data, it is probably the most general and the most robust way to do pre-processing that applies well across the board. If we represent a feature vector as, Mahalanobis scaling will result in a rescaled feature vectorand can be summarized as:


whererepresents the average value andrepresents the standard deviation for attribute.

      Making a test model proceeds in a very similar way as for training: the “wisdom vector” or the weight vector will now be applied to the test data to make predictions according to:


      In the above expression it was assumed that there are k test data, and the superscript ‘test” is used to explicitly indicate that the weight vector will be applied to a set of k test data with m attributes or descriptors. If one considers testing for one sample data point at a time, Eq. (5) can be represented as a simple neural network with an input layer and just a single neuron, as shown in Fig. 1. The neuron produces the weighted sum of the average input features. Note that the transfer function, commonly found in neural networks, is not present here. Note also that that the number of weights for this one-layer neural networks equals the number of input descriptors or attributes. 

Fig. 1. Neural network representation for regression 

B. The Machine Learning Dilemma 

      Equations (2) and (3) contain the inverse of the feature kernel,, defined as:


      The feature kernel is a symmetric matrix where each entry represents the similarity between features. Obviously, if there were two features that would be completely redundant the feature matrix would contain two columns and two rows that are (exactly) identical, and the inverse does not exist. One can argue that all is still well, and that in order to make the simple regression method work one would just make sure that the same descriptor or attribute is not included twice. By the same argument, highly correlated descriptors (i.e., “cousin features” in data mining lingo) should be eliminated as well. While this argument sounds plausible, the truth of the matter is more subtle. Let us repeat Eq. (2) again and go just one step further as shown below. 


      Eq. (10) is the derivation of an equivalent linear formulation to Eq. (2), based on the so-called right-hand pseudo-inverse or Penrose inverse, rather than using the more common left-hand pseudo-inverse. It was not shown here how that last line followed from the previous equation, but the proof is straightforward and left as an exercise to the reader. Note that now the inverse is needed for a different entity matrix, which now has an dimensionality, and is called the data kernel, , as defined by:


      The right-hand pseudo-inverse formulation is less frequently cited in the literature, because it can only be non-rank deficient when there are more descriptive attributes than data points, which is not the usual case for data mining problems (except for data strip mining[17] cases). The data kernel matrix is a symmetrical matrix that contains entries representing similarities between data points. The solution to this problem seems to be straightforward. We will first try to explain here what seems to be an obvious solution, and then actually show why this won’t work. Looking at Eqs. (10) and (11) it can be concluded that, except for rare cases where there are as many data records as there are features, either the feature kernel is rank deficient (in case that , i.e., there are more attributes than data), or the data kernel is rank deficient (in case that , i.e., there are more data than attributes). It can be now argued that for the case one can proceed with the usual left-hand pseudo-inverse method of Eq. (2), and that for the case one should proceed with the right-hand pseudo inverse, or Penrose inverse following Eq. (10).

      While the approach just proposed here seems to be reasonable, it will not work well in practice. Learning occurs by discovering patterns in data through redundancies present in the data. Data redundancies imply that there are data present that seem to be very similar to each other (and that have similar values for the response as well). An extreme example for data redundancy would be a dataset that contains the same data point twice. Obviously, in that case, the data matrix is ill-conditioned and the inverse does not exist. This type of redundancy, where data repeat themselves, will be called here a “hard redundancy.” However, for any dataset that one can possibly learn from, there have to be many “soft redundancies” as well. While these soft redundancies will not necessarily make the data matrix ill-conditioned, in the sense that the inverse does not exist because the determinant of the data kernel is zero, in practice this determinant will be very small. In other words, regardless whether one proceeds with a left-hand or a right-hand inverse, if data contain information that can be learnt from, there have to be soft or hard redundancies in the data. Unfortunately, Eqs. (2) and (10) can’t be solved for the weight vector in that case, because the kernel will either be rank deficient (i.e., ill-conditioned), or poor-conditioned, i.e., calculating the inverse will be numerically unstable. We call this phenomenon “the machine learning dilemma:” (i) machine learning from data can only occur when data contain redundancies; (ii) but, in that case the kernel inverse in Eq. (2) or Eq. (10) is either not defined or numerically unstable because of poor conditioning. Taking the inverse of a poor-conditioned matrix is possible, but the inverse is not “sharply defined” and most numerical methods, with the exception of methods based on single value decomposition (SVD), will run into numerical instabilities. The data mining dilemma seems to have some similarity with the uncertainty principle in physics, but we will not try to draw that parallel too far.

      Statisticians have been aware of the data mining dilemma for a long time, and have devised various methods around this paradox. In the next sections, we will propose several methods to deal with the data mining dilemma, and obtain efficient and robust prediction models in the process. 

C. Regression Models Based on the Data Kernel 

      Reconsider the data kernel formulation of Eq. (10) for predictive modeling. There are several well-known methods for dealing with the data mining dilemma by using techniques that ensure that the kernel matrix will not be rank deficient anymore. Two well-known methods are principal component regression and ridge regression.[5] In order to keep the mathematical diversions to its bare minimum, only ridge regression will be discussed.

      Ridge regression is a very straightforward way to ensure that the kernel matrix is positive definite (or well-conditioned), before inverting the data kernel. In ridge regression, a small positive value, , is added to each element on the main diagonal of the data matrix. Usually the same value for  is used for each entry. Obviously, we are not solving the same problem anymore. In order to not deviate too much from the original problem, the value for  will be kept as small as we reasonably can tolerate. A good choice for  is a small value that will make the newly defined data kernel matrix barely positive definite, so that the inverse exists and is mathematically stable. In data kernel space, the solution for the weight vector that will be used in the ridge regression prediction model now becomes:


and predictions for can now be made according to:


where a very different weight vector was introduced: . This weight vector is applied directly to the data kernel matrix (rather than the training data matrix) and has the same dimensionality as the number of training data. To make a prediction on the test set, one proceeds in a similar way, but applies the weight vector on the data kernel for the test data, which is generally a rectangular matrix, and projects the test data on the training data according to:


where it is assumed that there are data points in the test set.




      The kernel transformation is an elegant way to make a regression model nonlinear. The kernel transformation goes back at least to the early 1900’s, when Hilbert addressed kernels in the mathematical literature. A kernel is a matrix containing similarity measures for a dataset: either between the data of the dataset itself, or with other data (e.g., support vectors[1,3]). A classical use of a kernel is the correlation matrix used for determining the principal components in principal component analysis, where the feature kernel contains linear similarity measures between (centered) attributes. In support vector machines, the kernel entries are similarity measures between data rather than features and these similarity measures are usually nonlinear, unlike the dot product similarity measure that we used before to define a kernel. There are many possible nonlinear similarity measures, but in order to be mathematically tractable the kernel has to satisfy certain conditions, the so-called Mercer conditions. [1]


      The expression above, introduces the general structure for the data kernel matrix,, for data. The kernel matrix is a symmetrical matrix where each entry contains a (linear or nonlinear) similarity between two data vectors. There are many different possibilities for defining similarity metrics such as the dot product, which is a linear similarity measure and the Radial Basis Function kernel or RBF kernel, which is a nonlinear similarity measure. The RBF kernel is the most widely used nonlinear kernel and the kernel entries are defined by


      Note that in the kernel definition above, the kernel entry contains the square of the Euclidean distance (or two-norm) between data points, which is a dissimilarity measure (rather than a similarity), in a negative exponential. The negative exponential also contains a free parameter, , which is the Parzen window width for the RBF kernel. The proper choice for selecting the Parzen window is usually determined by an additional tuning, also called hyper-tuning, on an external validation set. The precise choice for  is not crucial, there usually is a relatively broad range for the choice for  for which the model quality should be stable.

      Different learning methods distinguish themselves in the way by which the weights are determined. Obviously, the model in Eqs. (12 - 14) to produce estimates or predictions foris linear. Such a linear model has a handicap in the sense that it cannot capture inherent nonlinearities in the data. This handicap can easily be overcome by applying the kernel transformation directly as a data transformation. We will therefore not operate directly on the data, but on a nonlinear transform of the data, in this case the nonlinear data kernel. This is very similar to what is done in principal component analysis, where the data are substituted by their principal components before building a model. A similar procedure will be applied here, but rather than substituting data by their principal components, the data will be substituted by their kernel transform (either linear or nonlinear) before building a predictive model.

      The kernel transformation is applied here as a data transformation in a separate pre-processing stage. We actually replace the data by a nonlinear data kernel and apply a traditional linear predictive model. Methods where a traditional linear algorithm is used on a nonlinear kernel transform of the data are introduced here as “direct kernel methods.” The elegance and advantage of such a direct kernel method is that the nonlinear aspects of the problem are captured entirely in the kernel and are transparent to the applied algorithm. If a linear algorithm was used before introducing the kernel transformation, the required mathematical operations remain linear. It is now clear how linear methods such as principal component regression, ridge regression, and partial least squares can be turned into nonlinear direct kernel methods, by using exactly the same algorithm and code: only the data are different, and we operate on the kernel transformation of the data rather than the data themselves.

      In order to make out-of-sample predictions on true test data, a similar kernel transformation needs to be applied to the test data, as shown in Eq. (14). The idea of direct kernel methods is illustrated in Fig. 2, by showing how any regression model can be applied to kernel-transformed data. One could also represent the kernel transformation in a neural network type of flow diagram and the first hidden layer would now yield the kernel-transformed data, and the weights in the first layer would be just the descriptors of the training data. The second layer contains the weights that can be calculated with a hard computing method, such as kernel ridge regression. When a radial basis function kernel is used, this type of neural network would look very similar to a radial basis function neural network, except that the weights in the second layer are calculated differently.


Fig. 2. Direct kernels as a data pre-processing step

A. Dealing with Bias: Centering the Kernel 

      There is still one important detail that was overlooked so far, and that is necessary to make direct kernel methods work. Looking at the prediction equations in which the weight vector is applied to data as in Eq. (1), there is no constant offset term or bias. It turns out that for data that are centered this offset term is always zero and does not have to be included explicitly. In machine learning lingo the proper name for this offset term is the bias, and rather than applying Eq. (1), a more general predictive model that includes this bias can be written as:


where is the bias term. Because we made it a practice in data mining to center the data first by Mahalanobis scaling, this bias term is zero and can be ignored.

      When dealing with kernels, the situation is more complex, as they need some type of bias as well. We will give only a recipe here, that works well in practice, and refer the reader to the literature for a more detailed explanation.[3, 6] Even when the data were Mahalanobis-scaled, before applying a kernel transform, the kernel still needs some type of centering to be able to omit the bias term in the prediction model. A straightforward way for kernel centering is to subtract the average from each column of the training data kernel, and store this average for later recall, when centering the test kernel. A second step for centering the kernel is going through the newly obtained vertically centered kernel again, this time row by row, and subtracting the row average form each horizontal row.

      The kernel of the test data needs to be centered in a consistent way, following a similar procedure. In this case, the stored column centers from the kernel of the training data will be used for the vertical centering of the kernel of the test data. This vertically centered test kernel is then centered horizontally, i.e., for each row, the average of the vertically centered test kernel is calculated, and each horizontal entry of the vertically centered test kernel is substituted by that entry minus the row average.

      Mathematical formulations for centering square kernels are explained in the literature.[3, 6] The advantage of the kernel-centering algorithm introduced (and described above in words) in this section is that it also applies to rectangular data kernels. The flow chart for pre-processing the data, applying a kernel transform on this data, and centering the kernel for the training data, validation data, and test data is shown in Fig. 3. 

Fig. 3. Data pre-processing with kernel centering 

B. Direct Kernel Ridge Regression 

      So far, the argument was made that by applying the kernel transformation in Eqs. (13) and (14), many traditional linear regression models can be transformed into a nonlinear direct kernel method. The kernel transformation and kernel centering proceed as data pre-processing steps (Fig. 2). In order to make the predictive model inherently nonlinear, the radial basis function kernel will be applied, rather than the (linear) dot product kernel, used in Eqs. (2) and (10). There are actually several alternate choices for the kernel,[1-3] but the RBF kernel is the most widely applied kernel. In order to overcome the machine learning dilemma, a ridge can be applied to the main diagonal of the data kernel matrix. Since the kernel transformation is applied directly on the data, before applying ridge regression, this method is called direct-kernel ridge regression.

      Kernel ridge regression and (direct) kernel ridge regression are not new. The roots for ridge regression can be traced back to the statistics literature.[5] Methods equivalent to kernel ridge regression were recently introduced under different names in the machine learning literature (e.g., proximal SVMs were introduced by Mangasarian et al.[7] kernel ridge regression was introduced by Poggio et al.[8] and Least-Squares Support Vector Machines were introduced by Suykens et al.[9-10]). In these works, Kerned Ridge Regression is usually introduced as a regularization method that solves a convex optimization problem in a Langrangian formulation for the dual problem that is very similar to traditional SVMs. The equivalency with ridge regression techniques then appears after a series of mathematical manipulations. By contrast, we introduced kernel ridge regression with few mathematical diversions in the context of the machine learning dilemma and direct kernel methods. For all practical purposes, kernel ride regression is similar to support vector machines, works in the same feature space as support vector machines, and was therefore named least-squares support vector machines by Suykens et al.

      Note that kernel ridge regression still requires the computation of an inverse for a matrix, which can be quite large. This task is computationally demanding for large datasets, as is the case in a typical data mining problem. Since the kernel matrix now scales with the number of data squared, this method can also become prohibitive from a practical computer implementation point of view, because both memory and processing requirements can be very demanding. Krylov space-based methods[10] and conjugate gradient methods[1, 10] are relatively efficient ways to speed up the matrix inverse transformation of large matrices, where the computation time now scales as n2, rather than n3. The Analyze/Stripminer code[12] developed by the author applies M�ller’s scaled conjugate gradient method to calculate the matrix inverse.[13]

      The issue of dealing with large datasets is even more profound. There are several potential solutions that will not be discussed in detail. One approach would be to use a rectangular kernel, were not all the data are used as bases to calculate the kernel, but a good subset of “support vectors” is estimated by chunking[1] or other techniques such as sensitivity analysis. More efficient ways for inverting large matrices are based on piece-wise inversion. Alternatively, the matrix inversion may be avoided altogether by adhering to the support vector machine formulation of kernel ridge regression and solving the dual Lagrangian optimization problem and applying the sequential minimum optimization or SMO.[16] 


      It has been shown that kernel ridge regression can be expressed as an optimization method,[10-15] where rather than minimizing the residual error on the training set, according to:


we now minimize:


      The above equation is a form of Tikhonov regularization[14] that has been explained in detail by Cherkassky and Mulier[4] in the context of empirical versus structural risk minimization. Minimizing the norm of the weight vector is in a sense similar to an error penalization for prediction models with a large number of free parameters. An obvious question in this context relates to the proper choice for the regularization parameter or ridge parameter .

      In the machine learning, it is common to tune the hyper-parameter  using a tuning/validation set. This tuning procedure can be quite time consuming for large datasets, especially in consideration that a simultaneous tuning for the RBF kernel width must proceed in a similar manner. We therefore propose a heuristic formula for the proper choice for the ridge parameter, that has proven to be close to optimal in numerous practical cases [36]. If the data were originally Mahalanobis scaled, it was found by scaling experiments that a near optimal choice for  is


where n is the number of data in the training set.

      Note that in order to apply the above heuristic the data have to be Mahalanobis scaled first. Eq. (20) was validated on a variety of standard benchmark datasets from the UCI data repository, and provided results that are nearly identical to an optimally tuned  on a tuning/validation set. In any case, the heuristic formula for  should be an excellent starting choice for the tuning process for . The above formula proved to be also useful for the initial choice for the regularization parameter C of SVMs, where C is now taken as 1/.




The author acknowledges the National Science Foundation support of this work (IIS-9979860). The discussions with Robert Bress, Kristin Bennett, Karsten Sternickel, Boleslaw Szymanski and Seppo Ovaska were extremely helpful to prepare this paper. 


    [1] Nello Cristianini and John Shawe-Taylor [2000] Support Vector Machines and Other Kernel-Based Learning Methods, Cambridge University Press.

    [2] Vladimir Vapnik [1998] Statistical Learning Theory, John Wiley & Sons.

    [3] Bernhard Sch�lkopf and Alexander J. Smola [2002] Learning with Kernels, MIT Press.

    [4] Vladimir Cherkassky and Filip Mulier [1998] Learning from Data: Concepts, Theory, and Methods, John Wiley & Sons, Inc.

    [5] A. E. Hoerl, and R. W. Kennard [1970]  “Ridge Regression: Biased Estimation for Non-Orthogonal Problems,” Technometrics, Vol. 12, pp. 69-82.

    [6] B. Sch�lkopf, A. Smola, and K-R M�ller [1998] “Nonlinear Component Analysis as a Kernel Eigenvalue Problem,” Neural Computation, Vol. 10, 1299-1319, 1998.

    [7] Glenn Fung and Olvi L. Mangasarian, “Proximal Support Vector Machine Classifiers,” in Proceedings KDD 2001, San Francisco, CA.

    [8] Evgeniou, T., Pontil, and M. Poggio, T. [2000] “Statistical Learning Theory: A Primer,” International Journal of Computer Vision, Vol. 38(1), pp.  9-13.

    [9] Suykens, J. A. K. and Vandewalle, J. [1999] “Least-Squares Support Vector Machine Classifiers,” Neural Processing letters, Vol. 9(3), pp. 293-300, Vol. 14, pp. 71-84.

    [10] Suykens, J. A. K., van Gestel, T. de Brabanter, J. De Moor, M., and Vandewalle, J. [2003] Least Squares Support Vector Machines, World Scientific Pub Co, Singapore.

    [11] Ilse C. F. Ipsen, and Carl D. Meyer [1998] “The Idea behind Krylov Methods,” American Mathematical Monthly, Vol. 105, 889-899.

    [12] The Analyze/StripMiner code is available on request for academic use, or can be downloaded from www.drugmining.com.

    [13] M�ller, M. F., [1993] “A Scaled Conjugate Gradient Algorithm for Fast Supervised Learning,” Neural Networks, Vol. 6, pp.525-534.

    [14] A. N. Tikhonov and V. Y. Arsenin [1977] Solutions of ill-Posed Problems, W.H. Winston, Washinton D.C.

    [15] Bennett, K. P., and Embrechts, M. J. [2003] “An Optimization Perspective on Kernel Partial Least Squares Regression,” Chapter 11 in Advances in Learning Theory: Methods, Models and Applications, Suykens J.A.K. et al., Eds., NATO-ASI Series in Computer and System Sciences, IOS Press, Amsterdam, The Netherlands.

    [16] Keerthi, S. S., and Shevade S. K. [2003] “SMO Algorithm for Least Squares SVM Formulations,” Neural Computation, Vol. 15, pp. 487-507.

    [17] Robert H. Kewley, and Mark J. Embrechts [2000] “Data Strip Mining for the Virtual Design of Pharmaceuticals with Neural Networks,” IEEE Transactions on Neural Networks, Vol.11 (3), pp. 668-679.



Set Home | Add to Favorites

All Rights Reserved Powered by Free Document Search and Download

Copyright © 2011
This site does not host pdf,doc,ppt,xls,rtf,txt files all document are the property of their respective owners. complaint#downhi.com