Data Preprocessing and Normalization

Why Preprocessing Matters

Raw data is almost never in the form models can learn from effectively. Missing values cause errors; unscaled features cause gradient instability; unremedied categorical columns need numerical encoding. Careful preprocessing often improves model performance more than switching algorithms.

Garbage in, garbage out — no model compensates for poorly prepared data.

Inspecting Raw Data

Before anything, understand what you have:

import pandas as pd
import numpy as np

df = pd.read_csv("data.csv")

print(df.shape)            # (n_rows, n_cols)
print(df.dtypes)           # column types
print(df.describe())       # stats for numeric columns
print(df.isnull().sum())   # missing values per column
print(df.duplicated().sum()) # duplicate rows

# Distribution of categorical columns
print(df["category"].value_counts())

Handling Missing Values

MAR = Missing At Random (missingness doesn't depend on the missing value itself) MNAR = Missing Not At Random (dangerous — the value being missing is informative)

from sklearn.impute import SimpleImputer, KNNImputer

# Numeric columns: fill with median
num_imputer = SimpleImputer(strategy="median")
X_num = num_imputer.fit_transform(df[num_cols])

# Categorical: fill with most frequent
cat_imputer = SimpleImputer(strategy="most_frequent")
X_cat = cat_imputer.fit_transform(df[cat_cols])

# Model-based: KNN imputation
knn_imp = KNNImputer(n_neighbors=5)
X_imputed = knn_imp.fit_transform(X)

# Add missingness indicator
df["age_missing"] = df["age"].isnull().astype(int)
df["age"] = df["age"].fillna(df["age"].median())

Feature Scaling / Normalization

Many algorithms are sensitive to feature magnitude: gradient descent converges faster, and distance-based algorithms (KNN, SVM, k-means) are biased by scale.

Algorithms that NEED scaling: logistic regression, linear SVM, KNN, k-means, PCA, neural networks, gradient descent in general.

Algorithms INSENSITIVE to scaling: decision trees, random forests, gradient boosting.

Standardization (Z-score Normalization)

z = (x − μ) / σ

Result has mean 0 and standard deviation 1. Most common choice. Works well when features are approximately Normal; robust to different scales.

Min-Max Normalization

x' = (x − xₘᵢₙ) / (xₘₐₓ − xₘᵢₙ) ∈ [0, 1]

Useful when you need bounded output (e.g., neural net input, pixel values). Sensitive to outliers — one extreme value compresses all others.

Robust Scaler

x' = (x − median) / IQR

Uses interquartile range instead of std — resistant to outliers. Good when data has extreme values.

Max Absolute Scaler

x' = x / |xₘₐₓ| ∈ [−1, 1]

Preserves sparsity (zeros stay zero). Useful for sparse feature matrices.

from sklearn.preprocessing import StandardScaler, MinMaxScaler, RobustScaler

scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)  # fit on train only!
X_test_scaled  = scaler.transform(X_test)        # apply same transform

Critical rule: always fit scalers on training data only. Fitting on the entire dataset leaks test-set statistics (data leakage).

Encoding Categorical Variables

Models require numeric inputs. Strategies depend on the cardinality and ordinality of the variable.

One-Hot Encoding (OHE)

Create a binary column per category. Use for nominal (unordered) categories with low cardinality (< ~20 unique values).

# Pandas
dummies = pd.get_dummies(df["color"], prefix="color", drop_first=True)

# Sklearn
from sklearn.preprocessing import OneHotEncoder
enc = OneHotEncoder(sparse_output=False, handle_unknown="ignore")
X_ohe = enc.fit_transform(df[["color"]])

Ordinal Encoding

Map categories to integers preserving order (e.g., small=0, medium=1, large=2). Use when the feature IS ordered.

Target Encoding (Mean Encoding)

Replace each category with the mean of the target within that category. High-cardinality categories. Must use out-of-fold (from k-fold) to prevent leakage.

Label Encoding

Map categories to arbitrary integers (0, 1, 2, …). Only use with tree-based models — linear models will interpret the integer ordering as meaningful.

Hashing Trick

Map categories to hash buckets. Handles unseen categories and high cardinality without storing a vocabulary.

Handling Outliers

Outliers can dominate loss functions, especially MSE.

Detection methods: - Z-score: |z| > 3 is often flagged - IQR rule: < Q1 − 1.5·IQR or > Q3 + 1.5·IQR - Isolation Forest, DBSCAN (algorithmic)

Treatment options: - Remove (if genuine data entry error) - Cap/clip (winsorize) to a percentile (e.g., clip to [1%, 99%]) - Log-transform to reduce skewness - Use robust loss functions (Huber, MAE instead of MSE)

# Winsorize
lower = df["price"].quantile(0.01)
upper = df["price"].quantile(0.99)
df["price"] = df["price"].clip(lower, upper)

# Log-transform (for positive, skewed features)
df["price_log"] = np.log1p(df["price"])   # log(1+x) safe for zeros

Feature Engineering

Transform existing features into more informative representations.

Polynomial Features

Add x², x³, x₁·x₂ etc. to model nonlinear relationships with a linear model.

from sklearn.preprocessing import PolynomialFeatures
poly = PolynomialFeatures(degree=2, include_bias=False)
X_poly = poly.fit_transform(X)  # adds x1², x2², x1*x2

Interaction Terms

Explicitly model how two features combine: income × age, price × discount.

Date/Time Features

Extract meaningful sub-features from timestamps: - Year, month, day, hour, weekday, quarter - Days since reference (ordinal time variable) - Is weekend, is holiday, is peak season - Cyclical encoding: sin(2π·month/12), cos(2π·month/12) — preserves circular structure

Text Features

• Bag of words / TF-IDF vectors
• Character count, word count, punctuation count
• Readability scores

Binning / Discretization

Convert continuous to categorical (e.g., age → age_group). Useful when the relationship is non-monotone in a linear model.

df["age_bin"] = pd.cut(df["age"], bins=[0,18,35,60,100],
                        labels=["youth","adult","middle","senior"])

Feature Selection

Remove irrelevant or redundant features to reduce overfitting, speed up training, and improve interpretability.

from sklearn.feature_selection import SelectKBest, mutual_info_classif, RFE
from sklearn.ensemble import RandomForestClassifier

# Filter: top-k features by mutual information
selector = SelectKBest(mutual_info_classif, k=10)
X_selected = selector.fit_transform(X_train, y_train)

# Embedded: feature importances from tree
rf = RandomForestClassifier().fit(X_train, y_train)
importances = rf.feature_importances_

The sklearn Pipeline

Combining preprocessing and model into one object prevents leakage and simplifies deployment:

from sklearn.pipeline import Pipeline
from sklearn.compose import ColumnTransformer
from sklearn.preprocessing import StandardScaler, OneHotEncoder
from sklearn.impute import SimpleImputer
from sklearn.ensemble import GradientBoostingClassifier

# Define transformations per column type
num_transformer = Pipeline([
    ("imputer", SimpleImputer(strategy="median")),
    ("scaler", StandardScaler()),
])
cat_transformer = Pipeline([
    ("imputer", SimpleImputer(strategy="most_frequent")),
    ("ohe", OneHotEncoder(handle_unknown="ignore")),
])

preprocessor = ColumnTransformer([
    ("num", num_transformer, num_features),
    ("cat", cat_transformer, cat_features),
])

# Full pipeline
pipeline = Pipeline([
    ("preprocessor", preprocessor),
    ("classifier", GradientBoostingClassifier()),
])

pipeline.fit(X_train, y_train)
y_pred = pipeline.predict(X_test)  # same transform applied automatically

Class Imbalance

When one class vastly outnumbers another (e.g., fraud: 0.1% positive rate), vanilla training learns to always predict majority class.

from sklearn.utils.class_weight import compute_class_weight

weights = compute_class_weight("balanced", classes=np.unique(y), y=y)
class_weight_dict = dict(zip(np.unique(y), weights))

# Pass to model
from sklearn.linear_model import LogisticRegression
model = LogisticRegression(class_weight=class_weight_dict)