MerQuR+Statistics Glossary

Statistics Glossary → Support Vector Machine (SVM)

Support Vector Machine (SVM)analysis

A classification and regression method that finds the boundary separating two classes with the widest possible gap (margin), and can build curved boundaries through kernel functions when needed.

Concept

Many boundaries could separate two classes; a support vector machine (SVM) chooses the one with the largest distance (margin) to the nearest observations of the two classes. A wide-margin boundary is more robust to small measurement fluctuations when classifying new observations. The boundary is determined only by the observations lying closest to it or on the wrong side; these are called support vectors, and the other observations do not affect where the boundary lies.

In real data the classes are rarely separated perfectly, so some observations are allowed to fall inside the margin or on the wrong side, and each violation is penalised through the parameter C. When a straight boundary is not enough, a kernel function is used: the data are implicitly mapped into a higher-dimensional space, a flat boundary is drawn there, and it corresponds to a curved boundary in the original variables. In the RBF kernel the similarity of two observations decreases with the distance between them: k(u, v) = exp(−γ‖u − v‖²). The regression form (SVR) ignores errors that stay within a band of width ε and penalises only observations falling outside it.

Because SVM relies on distances, it is sensitive to the scales of the variables; they should not be used without standardisation. The model does not produce probabilities directly; these are obtained afterwards by fitting a logistic curve to the decision values (Platt scaling). With non-linear kernels the coefficients cannot be interpreted, so the role of the variables is read from permutation importance and partial dependence. The result depends strongly on the choice of C and γ; these should be chosen by cross-validation and performance assessed on data the model has not seen.

When to use it

  • In classification problems with a moderate number of observations, a relatively large number of variables and a possibly curved boundary between the classes (text classification, image features, biological measurements).
  • If the effects of the variables must be interpreted as coefficients and odds ratios, logistic regression is preferred; if readable decision rules are needed, a decision tree.
  • With very large datasets, or many variables of mixed type (categorical and continuous), random forest or gradient boosting is usually more practical.

An everyday example

An apple cooperative wants to sort fruit into 'export' and 'domestic market' grades from measurements of diameter, colour intensity and sugar content. If the measurements are pictured as points on a chart, SVM draws the line that leaves the widest possible 'buffer strip' between the two groups; where that line lies is decided only by the hard-to-call apples nearest the boundary. If the groups cannot be separated by a straight line (for example, if both very small and very large apples go to the domestic market), a kernel function builds a curved boundary. The regression form of the same method could also predict an apple's shelf life in days.

Parameters in MerQur

Kernel (rbf / linear / poly / sigmoid)

Concept: The kernel is the function that measures how similar two observations are, and it sets the shape of the boundary: the linear kernel gives a straight boundary, the RBF kernel local, flexible curved boundaries, and the polynomial kernel curves that involve products of the variables.

Everyday example: If export apples are simply the 'large and red' ones, a straight line is enough. If both very small and very large apples go to the domestic market, the export apples form an 'island' in the middle; an RBF kernel can enclose this island, while a straight line cannot.

What it does: The shape of the decision boundary: rbf exp(-γ|u-v|²), linear u'v, poly (γu'v + coef0)^degree, sigmoid tanh(γu'v + coef0).

Effect on the result: The shape of the decision boundary, the number of support vectors and all test metrics change.

On the validation data: iris test accuracy rbf 0.9667 (47 support vectors), linear 1.0 (23), poly 0.9 (46), sigmoid 0.9 (45); Pima linear 0.65, rbf 0.60; Boston SVR linear R² 0.6064, rbf 0.8227.

→ SVM kernel function (rbf, linear, polynomial) and γ

C (cost)

Concept: C sets how heavily margin violations are penalised. A small C tolerates a few errors and builds a wide, calm boundary; a large C tries to fix every error in the training data and raises the risk of overfitting.

Everyday example: With a small C the machine accepts leaving a few borderline apples on the wrong side and keeps a wide buffer strip. With a large C it bends the boundary to put every apple on the right side; such a boundary may work worse on the next harvest.

What it does: Penalty for misclassification; a large C means a narrow margin.

Effect on the result: Small C gives a wide margin and many support vectors, large C a narrow margin and few support vectors; test performance changes.

On the validation data: iris C 0.1: 0.90 (95 support vectors); C 1: 0.9667 (47); C 100: 0.9667 (25). Boston C 10: R² 0.8278.

→ Cost parameter C and the ε-insensitive band

gamma

Concept: γ sets how far the influence of one observation reaches in the RBF (and polynomial, sigmoid) kernel. A large γ means each observation affects only its close neighbourhood, giving very wiggly boundaries; a small γ gives smoother, broader-scale boundaries.

Everyday example: With a large γ each apple 'influences' only the few apples very similar to it, and the boundary winds around individual apples. With a small γ similarity spreads over a wider area and the boundary is smoother.

What it does: Kernel width; 0 = "scale" = 1/(p·Var(Xs)); because X is standardized, 1/p.

Effect on the result: A larger γ gives a more local, more complex boundary (risk of overfitting); the number of support vectors and the performance change.

On the validation data: iris γ 0.05: 0.9333 (61 support vectors); 0.25 (scale): 0.9667; 1: 0.9333 (53). With the default "scale" the γ used is 0.25.

→ SVM kernel function (rbf, linear, polynomial) and γ

Degree (poly) and coef0

Concept: In the polynomial kernel the degree sets how curved the boundary may be (a parabola for degree two, more complex curves for degree three); coef0 is the constant term in the kernel and adjusts the weight of the lower-order terms.

Everyday example: A second-degree polynomial can build a boundary with a single bend, such as 'export grade once diameter and colour together exceed a certain level'. As the degree grows the boundary can bend more, but with few apples this flexibility can turn into memorising.

What it does: Polynomial degree (poly only); the constant term of poly and sigmoid.

Effect on the result: Changes the result only with the poly (degree) and poly/sigmoid (coef0) kernels; no effect with rbf and linear.

On the validation data: poly degree 2: 0.8333 (78 support vectors), 3: 0.9, 5: 0.8333; poly coef0 1: 0.9667 (24); sigmoid coef0 -1: 0.90. With the rbf kernel it changes no output (expected).

→ SVM kernel function (rbf, linear, polynomial) and γ

ε (epsilon)

Concept: In support vector regression, ε is the width of the band that decides which differences between prediction and actual value count as 'negligible'. Errors inside the band are not penalised; a wide band gives fewer support vectors and a coarser model.

Everyday example: When predicting shelf life, if an error of a few hours does not matter, the band is kept wide and the model adjusts only to apples it misses by more than half a day. If the band is widened too much, the model misses detail and the predictions become crude.

What it does: Width of the SVR epsilon-insensitive band (on the standardized target scale); not used in classification.

Effect on the result: As the band widens, fewer support vectors remain and the model becomes coarser; the regression R² changes.

On the validation data: Boston ε 0: R² 0.8232; 0.1: 0.8227; 0.5: 0.7779.

→ Cost parameter C and the ε-insensitive band

Test proportion, CV folds and repeats, Random seed

Concept: The data are split into a training set on which the model is built and a test set it never sees; cross-validation splits the training set into parts and uses each in turn for validation. The SVM solution is always the same for given data; what changes the results are the seed-dependent splits.

Everyday example: The cooperative trains the machine on 160 of 200 apples and uses the other 40 as an exam. Because the exam apples are chosen at random, another seed picks another 40 apples and the score shifts a little; the CV average smooths out this luck.

What it does: Stratified training/test split proportion; shuffled k-fold CV within the training set and its repeats; split and CV seed.

Effect on the result: The SVM solution is deterministic; only the split and the folds change, so the test and CV scores change.

On the validation data: iris test 0.1: 0.9333; 0.5: 0.96. CV with 2, 5, 10 folds: 0.96, 0.9667, 0.9666; 3x5 repeats 0.9622. Seed 1: the same accuracy 0.9667, a different split (49 support vectors). Boston SVR 5-fold CV R² 0.791 (test R² 0.823).

→ Train-test split (holdout)

Imbalanced classes: balanced weights

Concept: When one class is rare, the penalty for errors in that class is raised in inverse proportion to the class size. This stops the model from taking the easy route to high accuracy by ignoring the minority class.

Everyday example: If only 20% of the harvest is export grade, the machine can reach 80% accuracy by calling everything 'domestic'. With balanced weights, missing an export apple costs more and the machine is pushed to recognise these apples.

What it does: Class weight n/(K·n_k); libsvm multiplies C per class.

Effect on the result: The penalty of the minority class increases; sensitivity, F1 and the number of support vectors change. No effect with balanced classes.

On the validation data: Pima accuracy from 0.60 to 0.675, F1 from 0.3846 to 0.5806, support vectors from 98 to 109. It has no effect in iris because the training classes are balanced.

→ Class imbalance and balanced weights

Hyperparameter tuning (grid search + CV)

Concept: All combinations of pre-set values for C and γ are tried by cross-validation within the training set, and the combination with the best average performance is chosen. Because the test set plays no part in this choice, the final performance estimate stays honest.

Everyday example: The cooperative tests the 16 combinations of four C and four γ values on the training apples and picks the setting with the most consistent result; only then does it look at the exam apples.

What it does: A grid of C {0.1, 1, 10, 100} x γ {scale, 0.01, 0.1, 1} with shuffled k-fold CV on the training set.

Effect on the result: The model with the selected C and γ is reported; test performance changes with the selection.

On the validation data: iris best C 1, γ 0.1 (CV 0.975); Pima C 1, γ 0.1 (0.7937); Boston C 10, scale (R² 0.828). With the same folds the e1071 grid gave the same selection and score.

→ Hyperparameter tuning (grid search)

Probability threshold and Decision AUC

Concept: The raw output of an SVM is a decision value showing how far an observation lies from the boundary; Platt scaling turns it into a probability and the threshold is applied to that probability. The decision AUC measures how well the raw decision values rank the two classes, so it is not affected by the threshold.

Everyday example: How deep an apple sits on the export side of the boundary is its raw decision value; Platt turns this into a statement such as '70% probability of export grade'. If the cooperative wants to export only apples it is very sure of, it raises the threshold to 70%; the decision AUC shows how well the boundary ranks the apples, regardless of the threshold.

What it does: The threshold is applied to the Platt probability; the decision AUC is computed from the raw decision value without the Platt probability.

Effect on the result: The threshold changes the class predictions and accuracy; the decision AUC does not depend on the threshold or on Platt calibration.

On the validation data: Pima threshold 0.3: accuracy 0.675; 0.7: 0.575. The decision AUC in Pima is 0.7143. It has no effect for a multi-class response.

→ Classification threshold (cut-off)

Permutation importance, Save predictions, Additional metrics, Partial dependence, Chart

Concept: Permutation importance shows how much test performance drops when a variable is shuffled, and partial dependence how the prediction changes on average as a variable changes. These outputs are the main way to see what information a non-linear SVM relies on; saving predictions adds each observation's class and probability to the dataset.

Everyday example: If shuffling the sugar content of the exam apples among themselves clearly lowers the machine's score, sugar content matters for the decision. The partial dependence plot might also show that the probability of export grade first rises with diameter and then falls again for very large apples.

What it does: Permutation importance on the test set; writing prediction and probability columns to the dataset; additional metrics (Kappa, MCC, NIR, McNemar); partial dependence; plot.

Effect on the result: The model and test metrics do not change; additional tables and plots are produced.

On the validation data: Permutation importance is computed with repeated permutations; the additional metrics are reported; in classification the partial dependence uses the Platt probability.

→ Permutation importance

Example on real data

Data: Pima Indians (Pima.tr), R MASS package (n = 200)

Variables: target type (diabetes Yes/No), features number of pregnancies, glucose, blood pressure, skin thickness, BMI, pedigree function and age

  • Question: How accurately does a support vector machine with an RBF kernel built on seven clinical measurements classify diabetes (type)?
  • Model: RBF kernel, C = 1, gamma = 1/p (scale), X standardized on the training set; training n = 160, test n = 40 (stratified, seed 42).
  • Result: On the test set accuracy = .60, precision = .42, recall = .36, F1 = .38, ROC-AUC = .71; only 5 of 14 diabetic women are detected.
  • Generalization: 5-fold CV accuracy M = .78 (SD = .05); because the test set is small (n = 40), the test accuracy is clearly lower than the CV accuracy.
  • Limitation: Class imbalance (35% diabetic) lowers sensitivity; balanced class weights, threshold tuning or a grid search for C and gamma should be tried. The ROC-AUC is based on the Platt probability and is random because of the internal CV.
  • APA: An SVM (RBF, C = 1) on the test set (n = 40) gave accuracy = .60, F1 = .38, ROC-AUC = .71; 5-fold CV accuracy M = .78 (SD = .05).

Related concepts

SVM kernel function (rbf, linear, polynomial) and γCost parameter C and the ε-insensitive bandPermutation importanceTrain-test split (holdout)Hyperparameter tuning (grid search)Class imbalance and balanced weightsClassification threshold (cut-off)Variable importance (gain, cover, frequency)

The numbers in the examples are MerQur's output on independent, publicly available validation data; see the MerQur Technical Note for details.