Fine-Tuning BERT for Production NLP: A Battle-Tested Guide

The Fine-Tuning Recipe

from transformers import BertForSequenceClassification, AdamW

model = BertForSequenceClassification.from_pretrained(
    'bert-base-multilingual-cased',  # for AR/FR/EN
    num_labels=3
)

# Discriminative learning rates
optimizer = AdamW([
    {'params': model.bert.embeddings.parameters(), 'lr': 1e-5},
    {'params': model.bert.encoder.layer[:6].parameters(), 'lr': 2e-5},
    {'params': model.bert.encoder.layer[6:].parameters(), 'lr': 3e-5},
    {'params': model.classifier.parameters(), 'lr': 5e-5},
])

Key Lessons

1. Start with LR=2e-5, batch=16, epochs=3-5
2. Warmup for 10% of steps prevents catastrophic forgetting
3. For Arabic: use CAMeL-BERT or AraBERT, not mBERT
4. ONNX export: 3x faster inference, no PyTorch needed