AI & Machine Learning Cheatsheet
Evaluation Metrics
Use this AI & Machine Learning reference while you build software engineering projects, review code for technical interview prep, or polish examples for a software engineer resume.
Why Evaluation Matters
The choice of metric determines what you optimize for. Accuracy looks fine on a 99%-negative dataset where the model predicts "negative" always. F1 rewards balance between precision and recall. AUC-ROC measures ranking ability regardless of threshold. Choose the metric that aligns with real-world cost of errors.
Classification Metrics
The Confusion Matrix
For binary classification, a confusion matrix has four cells:
Predicted Positive Predicted Negative Actual Positive TP FN Actual Negative FP TN
- True Positive (TP): correct positive prediction
- True Negative (TN): correct negative prediction
- False Positive (FP): negative labeled as positive (Type I error)
- False Negative (FN): positive labeled as negative (Type II error)
from sklearn.metrics import confusion_matrix, ConfusionMatrixDisplay import matplotlib.pyplot as plt cm = confusion_matrix(y_test, y_pred) disp = ConfusionMatrixDisplay(cm, display_labels=["Negative", "Positive"]) disp.plot(cmap="Blues") plt.show()
Derived Metrics
| Metric | Formula | Intuition |
|---|---|---|
| Accuracy | (TP+TN)/(P+N) | Overall correctness |
| Precision | TP/(TP+FP) | Of predicted positives, how many are real? |
| Recall (Sensitivity) | TP/(TP+FN) | Of actual positives, how many did we catch? |
| Specificity | TN/(TN+FP) | Of actual negatives, how many did we reject? |
| F1 Score | 2·Prec·Rec/(Prec+Rec) | Harmonic mean of precision and recall |
| Fβ Score | (1+β²)·P·R/(β²·P+R) | β>1 weights recall higher; β<1 weights precision |
| MCC | (TP·TN−FP·FN)/√(...) | Balanced, even with class imbalance; range [−1,1] |
| Balanced Accuracy | (Sensitivity+Specificity)/2 | Useful for imbalanced classes |
Precision vs. Recall tradeoff: - Spam filter: high precision (don't want good emails in spam) even at cost of recall - Cancer detection: high recall (don't miss cases) even at cost of precision - F1 balances both; F2 weights recall 2x
from sklearn.metrics import ( accuracy_score, precision_score, recall_score, f1_score, matthews_corrcoef, balanced_accuracy_score, classification_report ) print(classification_report(y_test, y_pred, target_names=class_names)) # Shows precision, recall, F1 per class + macro/weighted averages mcc = matthews_corrcoef(y_test, y_pred)
Multiclass Extensions
For K classes:
| Averaging | Method |
|---|---|
| Macro | Average metric per class equally (treats all classes equally regardless of size) |
| Weighted | Average weighted by class support (accounts for imbalance) |
| Micro | Aggregate TP, FP, FN across all classes, then compute |
| Per-class | Report separately for each class |
f1_macro = f1_score(y_test, y_pred, average="macro") f1_weighted = f1_score(y_test, y_pred, average="weighted")
ROC Curve and AUC
The ROC curve plots True Positive Rate (recall) vs. False Positive Rate (1 − specificity) at every decision threshold:
- At threshold 0 (predict all positive): TPR=1, FPR=1 → top-right corner
- At threshold 1 (predict all negative): TPR=0, FPR=0 → bottom-left corner
- Random classifier: diagonal line (AUC = 0.5)
- Perfect classifier: top-left corner (AUC = 1.0)
AUC-ROC (Area Under Curve): probability that the model ranks a random positive example higher than a random negative example.
from sklearn.metrics import roc_curve, roc_auc_score import matplotlib.pyplot as plt probs = model.predict_proba(X_test)[:, 1] fpr, tpr, thresholds = roc_curve(y_test, probs) auc = roc_auc_score(y_test, probs) plt.plot(fpr, tpr, label=f"AUC = {auc:.3f}") plt.plot([0,1],[0,1],"k--") plt.xlabel("FPR"); plt.ylabel("TPR") plt.title("ROC Curve"); plt.legend() plt.show()
Precision-Recall Curve (PR Curve)
For highly imbalanced datasets, ROC can be misleading (inflated by large TN pool). The PR curve is more informative: - Average Precision (AP) = area under PR curve - PR-AUC closer to 1 = better; random classifier's AP ≈ class prevalence
from sklearn.metrics import precision_recall_curve, average_precision_score precision, recall, thresh = precision_recall_curve(y_test, probs) ap = average_precision_score(y_test, probs) plt.plot(recall, precision, label=f"AP = {ap:.3f}") plt.xlabel("Recall"); plt.ylabel("Precision") plt.title("Precision-Recall Curve"); plt.legend() plt.show()
Log Loss (Cross-Entropy)
Measures probability calibration — penalizes confident wrong predictions heavily:
LogLoss = −(1/n) Σᵢ [yᵢ log p̂ᵢ + (1−yᵢ) log(1−p̂ᵢ)]
Lower = better (0 is perfect). Used as training loss for logistic regression and neural networks.
from sklearn.metrics import log_loss ll = log_loss(y_test, probs)
Regression Metrics
| Metric | Formula | Notes |
|---|---|---|
| MAE | (1/n) Σ|yᵢ−ŷᵢ| | Robust, same units as y |
| MSE | (1/n) Σ(yᵢ−ŷᵢ)² | Penalizes large errors; differentiable |
| RMSE | √MSE | Same units as y; most common |
| RMSLE | √(MSE of log(y+1)) | Good when target spans orders of magnitude |
| MAPE | (1/n) Σ|yᵢ−ŷᵢ|/yᵢ × 100% | Scale-free; undefined at y=0 |
| R² | 1 − SS_res/SS_tot | 1 = perfect; can be negative if model is worse than mean |
| Adjusted R² | R² penalized for p features | For comparing models with different # of features |
| Huber loss | MAE when |r|>δ, MSE otherwise | Robust + differentiable |
from sklearn.metrics import ( mean_absolute_error, mean_squared_error, mean_absolute_percentage_error, r2_score ) import numpy as np mae = mean_absolute_error(y_test, y_pred) rmse = np.sqrt(mean_squared_error(y_test, y_pred)) mape = mean_absolute_percentage_error(y_test, y_pred) * 100 r2 = r2_score(y_test, y_pred)
Ranking Metrics
Used for search, recommendation, and information retrieval tasks where the order of results matters.
| Metric | Description |
|---|---|
| Precision@k | Fraction of top-k retrieved items that are relevant |
| Recall@k | Fraction of all relevant items in top-k |
| NDCG@k | Normalized Discounted Cumulative Gain — rewards relevant items higher in ranking |
| MAP | Mean Average Precision across queries |
| MRR | Mean Reciprocal Rank of the first relevant item |
NDCG formula:
DCG@k = Σᵢ₌₁ᵏ rel_i / log₂(i+1)
NDCG@k = DCG@k / IDCG@k (IDCG = ideal DCG, all relevant items at top)
from sklearn.metrics import ndcg_score import numpy as np # y_true: relevance scores, y_score: predicted scores y_true = np.array([[3, 2, 3, 0, 1, 2]]) y_score = np.array([[3.7, 2.1, 2.8, 0.5, 1.2, 1.9]]) ndcg = ndcg_score(y_true, y_score, k=5)
Clustering Metrics
With Ground Truth
| Metric | Range | Meaning |
|---|---|---|
| Adjusted Rand Index (ARI) | [−1, 1] | Chance-corrected similarity to ground truth |
| Normalized Mutual Information | [0, 1] | Mutual info normalized by entropy |
| V-Measure | [0, 1] | Harmonic mean of homogeneity and completeness |
Without Ground Truth
| Metric | Range | Better |
|---|---|---|
| Silhouette | [−1, 1] | Higher |
| Davies-Bouldin | [0, ∞) | Lower |
| Calinski-Harabasz | [0, ∞) | Higher |
Cross-Validation
Never evaluate on training data. Use k-fold cross-validation for reliable estimates:
from sklearn.model_selection import ( cross_val_score, StratifiedKFold, KFold, cross_validate ) # Stratified k-fold (preserves class proportions) skf = StratifiedKFold(n_splits=5, shuffle=True, random_state=42) scores = cross_val_score(model, X, y, cv=skf, scoring="f1_weighted") print(f"F1: {scores.mean():.3f} ± {scores.std():.3f}") # Multiple metrics at once results = cross_validate( model, X, y, cv=5, scoring=["accuracy", "f1_macro", "roc_auc"], return_train_score=True )
Stratified k-fold is required for classification to ensure each fold has the same class distribution as the full dataset.
Statistical Significance of Metric Differences
When comparing two models on the same test set:
- McNemar test: for paired binary predictions (is model A's error pattern different from B's?)
- Bootstrap confidence intervals: resample predictions to estimate uncertainty in metric
- 5x2 CV paired t-test: run 5 iterations of 2-fold CV, use t-test on differences
from statsmodels.stats.contingency_tables import mcnemar # Contingency table: [both wrong, A right B wrong; A wrong B right; both right] result = mcnemar(contingency_table, exact=True) print(f"p-value: {result.pvalue:.4f}")
Choosing the Right Metric
| Problem | Class Balance | Metric |
|---|---|---|
| Binary classification | Balanced | Accuracy, F1, AUC-ROC |
| Binary classification | Imbalanced | F1, PR-AUC, MCC |
| Multiclass | Balanced | Macro F1, accuracy |
| Multiclass | Imbalanced | Weighted F1, per-class recall |
| Regression | No outliers | RMSE, R² |
| Regression | Outliers present | MAE, Huber |
| Regression | Target spans orders of magnitude | RMSLE, MAPE |
| Ranking / recommendation | — | NDCG@k, MAP, MRR |
| Probability calibration | — | Log loss, Brier score |