Kernel Methods
Kernel methods enable application of classical linear data analysis techniques such as used in regression, classification, clustering, and principal component analysis, to non-linear problems. The novelty of the kernel method for non-linear analysis is that the theory underlying the algorithm to which the kernel is applied does not change and can still be described in terms of linear analysis. The kernel is typically applied to linear algorithms through a “kernel-trick” whereby dot-products in the linear feature space are replaced with kernel evaluations. The kernel typically represents a dot-product that is formed in a non-linearly mapped space of the inputs. The novelty of this approach is that no explicit expressions for the non-linear mapping are needed since only inner products from the feature space are employed in the mathematical formulations. Since no explicit expressions are formed, the mappings can be complex and high-dimensional, including the infinite dimensional case.
Keywords that describe the current research in the lab within the context of Kernel Methods are: Support Vector Machines, Active Set Methods, Revised Simplex Method, Quadratic Programming, Outlier Detection, Mahalanobis Distance, Dimesnionality Reduction.
Efficiently implemented active set methods have been successfully applied to Support Vector Machine (SVM) training. These active set methods offer higher precision and incremental training at the cost of additional memory requirements when compared to decomposition methods such as the Sequential Minimal Optimization (SMO). However, all existing active set methods must deal with singularities occurring within the inner problem solved at each iteration, a problem that leads to more complex implementation and potential inefficiencies. Here, we introduce a revised simplex method, originally introduced by Rusin, adapted for SVM training and show this is an active set method similar to most existing methods with the advantage of maintaining non-singularity of the inner problem. We compare performance to an existing active set method introduced by Scheinberg and demonstrate an improvement in training times, in some cases. We show our method maintains a slightly simpler implementation and offers advantages in terms of applying iterative methods to alleviate memory concerns. We also show performance of the active set methods when compared to state-of-the-art decomposition implementations such as SVMLight and SMO.
- Georgios Anagnostopoulos
- Christopher Sentelle
- Cong Li
Kernel-Machines.Org
This site contains publications related to kernel based methods submitted by a number of researchers in the field; it also contains calls for papers in conferences that solicit kernel based method papers. Relevant publications from the Journal of Machine Learning Research are also referenced here. The website is: http://kernel-machines.org
Kernel Methods for Pattern Analysis
This book, developed from lectures and tutorials, fulfills two major roles: firstly it provides practitioners with a large toolkit of algorithms, kernels and solutions ready to use for standard pattern discovery problems in fields such as bioinformatics, text analysis, and image analysis. Secondly, it provides an easy introduction for students and researchers to the growing field of kernel-based pattern analysis, demonstrating, with examples, how to handcraft an algorithm or a kernel for a new specific application, and covering all the necessary conceptual and mathematical tools to do so. The website for the book is: http://www.kernel-methods.net
Learning with Kernels: Support Vector Machines, Regularization, Optimization and Beyond
This text is written by Scholkopf and Smola, and, according to a review by Amir F. Atiya (appeared in the IEEE Transactions on Neural Networks), is a comprehensive treatment of the topics of support vector machines and kernel methods by two well known researchers in the field. The book is intended for the researcher who already has information about the subject and would like to gain depth into the topic. It is not recommended for the beginning researcher or student who wishes to learn SVMs fundamentals. A description of the text can be found at the following website: http://www.learning-with-kernels.org