Supervised, Unsupervised, Reinforcement

Three Paradigms of Machine Learning

The primary distinction in ML is how labels are provided during training. Each paradigm suits different problem structures and data availability.

Supervised Learning

In supervised learning you have a dataset D = {(x₁,y₁), …, (xₙ,yₙ)} where each input xᵢ has a known target yᵢ. The goal is to learn a function f: X → Y that generalizes to unseen inputs.

Regression vs. Classification

Regression: Y is continuous (house price, temperature, stock return) - Loss: mean squared error, mean absolute error, Huber loss - Output: single real number

Classification: Y is discrete (cat vs. dog, spam vs. not, digit 0–9) - Loss: cross-entropy (binary or categorical) - Output: class probabilities via softmax/sigmoid

Key Supervised Algorithms

The Supervised Learning Pipeline

from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)
model = RandomForestClassifier(n_estimators=100, random_state=42)
model.fit(X_train, y_train)
y_pred = model.predict(X_test)
print(f"Accuracy: {accuracy_score(y_test, y_pred):.3f}")

Semi-Supervised Learning

Uses a small labeled set plus a large unlabeled set. Common when labeling is expensive: - Self-training: train on labeled, predict unlabeled, add high-confidence predictions to training set, repeat - Label propagation: spread labels through a similarity graph - Foundation models like GPT are pre-trained unsupervised then fine-tuned with a small labeled set — this is the dominant semi-supervised paradigm today

Unsupervised Learning

No labels. The algorithm must discover structure, patterns, or compact representations in the data itself.

Clustering

Group similar examples together. The algorithm defines "similar" via a distance metric.

• k-means: partition into k clusters by minimizing within-cluster variance
• DBSCAN: density-based; finds arbitrarily-shaped clusters and noise
• Hierarchical: build a tree (dendrogram) of nested clusters

Dimensionality Reduction

Project high-dimensional data to lower dimensions while preserving structure: - PCA: maximize variance (linear) - t-SNE, UMAP: preserve local neighborhoods (nonlinear, for visualization) - Autoencoders: neural-network compression

Density Estimation

Model the underlying data distribution p(x): - Gaussian Mixture Models (GMM): weighted sum of Gaussians fit via EM - Kernel Density Estimation (KDE): non-parametric, smooth histogram - Normalizing flows, VAEs, diffusion models: deep generative density models

Anomaly Detection

Model what "normal" looks like; flag deviations. Useful when anomalies are too rare and diverse to label.

from sklearn.cluster import KMeans
import numpy as np

kmeans = KMeans(n_clusters=3, random_state=42)
kmeans.fit(X)
labels = kmeans.labels_          # cluster assignment per sample
centers = kmeans.cluster_centers_ # centroid per cluster

Self-Supervised Learning

A special case of unsupervised learning that creates proxy supervised tasks from unlabeled data by hiding part of the input and predicting it:

Self-supervised pre-training followed by supervised fine-tuning is the dominant paradigm for large models.

Reinforcement Learning

An agent interacts with an environment by taking actions and receiving rewards. It learns a policy (a mapping from states to actions) that maximizes cumulative discounted reward.

Core Components

The RL Objective

Maximize expected cumulative discounted reward:

G_t = r_t + γ r_{t+1} + γ² r_{t+2} + … = Σₖ γᵏ r_{t+k}

RL Algorithm Families

Bellman Equation

The key recursive equation in RL:

V(s) = Σₐ π(a|s) [r(s,a) + γ Σₛ' P(s'|s,a) V(s')]

Q-learning update (off-policy, model-free):

Q(s,a) ← Q(s,a) + α [r + γ max_a' Q(s',a') − Q(s,a)]

Exploration vs. Exploitation

The central RL dilemma: should the agent exploit the best known action, or explore to discover potentially better ones?

• ε-greedy: with probability ε pick random action, else greedy
• UCB (Upper Confidence Bound): optimism in face of uncertainty
• Thompson Sampling: sample from posterior over action values

RL from Human Feedback (RLHF)

Modern LLMs (ChatGPT, Claude) are fine-tuned with RL: 1. Supervised fine-tuning on demonstrations 2. Train a reward model from human preference comparisons 3. Use PPO to optimize the LLM policy against the reward model

# Simplified Q-learning loop
import numpy as np

Q = np.zeros((n_states, n_actions))
alpha, gamma, epsilon = 0.1, 0.99, 0.1

for episode in range(n_episodes):
    s = env.reset()
    done = False
    while not done:
        # Epsilon-greedy action selection
        if np.random.rand() < epsilon:
            a = env.action_space.sample()
        else:
            a = np.argmax(Q[s])
        s_next, r, done, _ = env.step(a)
        # Bellman update
        Q[s, a] += alpha * (r + gamma * np.max(Q[s_next]) - Q[s, a])
        s = s_next

Comparison of Learning Paradigms

Transfer Learning and Domain Adaptation

Transfer learning: pre-train on source domain (large data), fine-tune on target domain (small data).

Domain adaptation: source and target have same task but different input distributions.

Zero-shot / few-shot: model generalizes to new tasks without (or with very few) labeled examples. Large pre-trained models exhibit this behavior through in-context learning.

The modern workflow: 1. Pre-train large model on internet-scale unlabeled data (self-supervised) 2. Fine-tune (or RLHF) on task-specific labeled data 3. Prompt-engineer for zero/few-shot tasks at inference