Classificationintermediate

OvA vs OvO Multi-class Classification

“Extending binary classifiers to multi-class — tournament brackets for algorithms”

One-vs-All and One-vs-One strategies for extending binary classifiers to multi-class — decision boundaries, scalability, SVM applications, and when to use Softmax instead.

30 min

6 diagrams

6 Concepts Covered

Prerequisites

→SVM

→Logistic Regression

Concepts Covered

Multi-classDecision BoundariesOvAOvOSoftmaxClass Imbalance

Previous: Bagging, Boosting & Stacking Next: Hyperparameter Tuning

∑Key Formulas

OvA Classifiers

K binary classifiers, one per class vs. all others

OvO Classifiers

One binary classifier per pair of classes

Softmax

Normalizes K logits to a probability distribution

▶Interactive Simulation

Loading visualization…

⬡Model Architecture

Loading visualization…

🎯

The Multi-Class Problem

motivation

Many real problems have more than 2 classes: digit recognition (10 classes), species classification (100s), product categorization (1000s). Some algorithms (logistic regression, SVMs) are inherently binary. Two strategies extend them: OvA trains K classifiers, each separating class k from all others. OvO trains K(K-1)/2 classifiers for every pair. Neural networks with Softmax solve multi-class natively.

⚖️

OvA vs OvO vs Softmax

comparison

OvA: K classifiers, each uses all data. Fast training. Imbalanced (1 positive vs K-1 negatives). Good for large K.

OvO: K(K-1)/2 classifiers, each uses only 2 classes. Balanced but slow for large K (100 classes = 4950 classifiers).

Softmax (multinomial LR): single model, K outputs, trained with cross-entropy. Most efficient. Native to neural nets.

SVM convention: OvO is default in sklearn (historically performs slightly better). For neural nets, always Softmax.

</>

Softmax Multi-class Classification

code

python49 lines

import torch
import torch.nn as nn
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
from sklearn.multiclass import OneVsRestClassifier, OneVsOneClassifier
from sklearn.svm import SVC

class="tok-comment"># ── Sample data ────────────────────────────────────────────────────────
X_np, y_np = make_classification(n_samples=class="tok-num">300, n_features=class="tok-num">8,
                                  n_classes=class="tok-num">3, n_informative=class="tok-num">6, random_state=class="tok-num">42)
X_train_np, X_test_np, y_train_np, _ = train_test_split(
    X_np, y_np, test_size=class="tok-num">0.2, random_state=class="tok-num">42)

class="tok-comment"># ── PyTorch multiclass setup ───────────────────────────────────────────
K = class="tok-num">3                                       class="tok-comment"># number of classes
batch = class="tok-num">16

class="tok-comment"># Tiny class="tok-num">2-layer net for the demo
class SimpleNet(nn.Module):
    def __init__(self): super().__init__(); self.fc = nn.Linear(class="tok-num">8, K)
    def forward(self, x): return self.fc(x)

model = SimpleNet()
x = torch.randn(batch, class="tok-num">8)                   class="tok-comment"># one mini-batch
y = torch.randint(class="tok-num">0, K, (batch,))           class="tok-comment"># class indices

class="tok-comment"># Class weights (handle imbalance)
class_weights = torch.tensor([class="tok-num">1.0, class="tok-num">2.0, class="tok-num">1.5])   class="tok-comment"># weight rarer classes higher

class="tok-comment"># Softmax + Cross-Entropy (combined for numerical stability)
criterion = nn.CrossEntropyLoss(
    weight=class_weights,    class="tok-comment"># For imbalanced classes
    label_smoothing=class="tok-num">0.1      class="tok-comment"># Prevents overconfident predictions
)

class="tok-comment"># Model outputs raw logits (no softmax in forward pass)
logits = model(x)            class="tok-comment"># Shape: (batch, K)
loss = criterion(logits, y)  class="tok-comment"># y contains class indices
print(fclass="tok-str">"Multiclass CE loss: {loss.item():.4f}")

class="tok-comment"># Predictions
probs = torch.softmax(logits, dim=-class="tok-num">1)
preds = probs.argmax(dim=-class="tok-num">1)

class="tok-comment"># Sklearn: OvR (OvA) strategy
ovr = OneVsRestClassifier(SVC(kernel=class="tok-str">'rbf', probability=True))
ovo = OneVsOneClassifier(SVC(kernel=class="tok-str">'rbf'))
ovr.fit(X_train_np, y_train_np)
print(fclass="tok-str">"OvR accuracy: {ovr.score(X_test_np, _):.3f}")

?Knowledge Check

Progress is saved in your browser — no account needed.

Bagging, Boosting & Stacking

Hyperparameter Tuning

Need an AI engineer or data scientist?

I build custom ML models, AI agents, computer vision, and automation — from idea to production.

Get in touch View services