Overview and Workflow

What is Machine Learning?

Machine learning is the study of algorithms that improve their performance on a task through experience (data), without being explicitly programmed for every case. The field sits at the intersection of statistics, linear algebra, and computer science.

If you are trying to learn Python for AI, use this page as a practical machine learning cheatsheet: most examples are Python because the data-science ecosystem is strongest there. JavaScript and TypeScript are useful for web demos, while Java and C++ show up when teams need production services or high-performance inference. For hands-on practice, keep the Python online compiler and Python cheatsheet nearby.

Three core ingredients: 1. Data — labeled or unlabeled examples of the problem 2. Model — a parameterized function that maps inputs to outputs 3. Learning algorithm — a procedure that adjusts model parameters to minimize error on training data

The Standard ML Workflow

Raw Data → Preprocessing → Feature Engineering → Model Selection
    → Training → Evaluation → Hyperparameter Tuning → Deployment → Monitoring

Each stage in detail:

1. Problem Definition

Define what you want to predict (target variable) and what type of problem it is:

2. Data Collection and Exploration

Exploratory Data Analysis (EDA) before any modeling: - Shape of data: df.shape, df.describe() - Missing values: df.isnull().sum() - Distribution of target variable (class imbalance?) - Correlations between features

import pandas as pd
import matplotlib.pyplot as plt

df = pd.read_csv("data.csv")
print(df.describe())
print(df["label"].value_counts())
df.hist(figsize=(12, 8))
plt.show()

3. Data Preprocessing

• Handle missing values (impute or drop)
• Encode categorical variables (one-hot, label encoding)
• Scale numerical features (standardization, min-max)
• Split into train / validation / test sets

4. Feature Engineering

Transform raw inputs into representations the model can use better: - Polynomial features (x² for nonlinear patterns) - Log-transform skewed distributions - Interaction terms (x₁ × x₂) - Domain-specific features (word count from text, day-of-week from timestamp)

5. Model Selection

Choose a model family based on data size, interpretability needs, and problem type. Start simple (linear model), then increase complexity only if needed.

6. Training

Training means finding parameters θ that minimize a loss function L(θ) over the training set:

θ* = argmin_θ (1/n) Σᵢ L(f(xᵢ; θ), yᵢ)

The optimization algorithm (gradient descent, Adam, etc.) iteratively updates θ based on gradients of L.

7. Evaluation

Evaluate on held-out data the model never saw during training. Use metrics appropriate for the problem: accuracy, F1, RMSE, AUC, etc.

Critical rule: never evaluate on training data — it gives optimistically biased estimates of real-world performance.

8. Hyperparameter Tuning

Model hyperparameters (learning rate, depth of tree, regularization strength) are NOT learned during training — they are set before training and tuned via: - Grid search - Random search - Bayesian optimization

Always tune on the validation set, not the test set.

9. Deployment and Monitoring

A model in production must be monitored for data drift (input distribution shifts) and concept drift (relationship between inputs and outputs changes). Retraining schedules depend on how quickly the world changes.

The Bias–Variance Tradeoff

Every model makes two kinds of errors:

Total expected error = Bias² + Variance + Irreducible Noise

The goal is to find the sweet spot of complexity that minimizes test error.

High Bias          Sweet Spot          High Variance
(underfitting)                          (overfitting)
    ↑                   ↑                    ↑
Linear model    Shallow tree (4 nodes)   Deep tree (1000 leaves)
on curved data   on this dataset         memorizes training set

Inductive Bias

Every learning algorithm embeds assumptions about what makes a good generalization. These are called inductive biases:

• Linear regression assumes the target is a linear function of inputs
• Nearest-neighbor assumes nearby points have similar labels
• Neural networks assume hierarchical feature composition is useful

There is no universally best algorithm (No Free Lunch theorem): for every algorithm that does well on some problems, there is an equal number of problems where it does poorly.

Data Splits

For small datasets, use k-fold cross-validation instead of a fixed split.

from sklearn.model_selection import train_test_split, cross_val_score
from sklearn.linear_model import LogisticRegression

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y
)

# Or with cross-validation
model = LogisticRegression()
scores = cross_val_score(model, X, y, cv=5, scoring="accuracy")
print(f"CV accuracy: {scores.mean():.3f} ± {scores.std():.3f}")

Key Python Libraries for AI and ML

Common Pitfalls

• Data leakage: test-set information bleeds into training (e.g., scaling using test-set statistics, target encoding without out-of-fold strategy)
• Ignoring class imbalance: 99% accuracy on a 1% positive-rate dataset means the model predicts "negative" always
• Premature optimization: jumping to complex models before baseline
• Not checking for duplicates: same row in train and test inflates scores
• Comparing models on different splits: always use the same held-out set for fair comparisons