Recommendation Systems

What are Recommendation Systems?

Recommendation systems predict user preferences for items they haven't seen yet, enabling platforms to surface relevant content, products, or media. Netflix, Spotify, Amazon, and YouTube all rely on sophisticated recommenders as core business infrastructure.

Two fundamental challenges: - Cold start: how to recommend for new users or new items with no history - Sparsity: users interact with a tiny fraction of all items (density <1% is common)

Problem Formulation

Given: - A set of users U (|U| = m) - A set of items I (|I| = n) - An interaction matrix R ∈ ℝ^{m×n} (ratings, clicks, purchases)

Goal: predict R̂ᵤᵢ for unobserved user-item pairs, or directly rank items per user.

Feedback types: - Explicit: star ratings, thumbs up/down - Implicit: clicks, views, time spent, purchases (more common and noisier)

Collaborative Filtering

Collaborative filtering assumes users who agreed in the past will agree in the future — no item content needed.

User-Based CF

Find users similar to the target user; aggregate their ratings:

ŷᵤᵢ = ȳᵤ + Σ_{v ∈ N(u)} sim(u,v) · (rᵥᵢ − ȳᵥ) / Σ sim(u,v)

Similarity measures: - Cosine: sim(u,v) = rᵤ · rᵥ / (‖rᵤ‖‖rᵥ‖) - Pearson: correlates mean-centered ratings - Jaccard: for binary interactions

Item-Based CF

Find items similar to what the user liked; recommend similar items:

ŷᵤᵢ = Σ_{j ∈ N(i)} sim(i,j) · rᵤⱼ / Σ sim(i,j)

Item-item CF is more stable than user-user CF (items change slower than user tastes) and scales better for large m.

from sklearn.metrics.pairwise import cosine_similarity
import numpy as np

# R: (n_users, n_items) matrix
R = np.array([[5,3,0,1], [4,0,0,1], [1,1,0,5], [0,0,3,4]])

# Item-item similarity
item_sim = cosine_similarity(R.T)   # (n_items, n_items)

# Predict user 0's rating for item 2
user_ratings = R[0, :]
rated_mask   = user_ratings > 0
pred = np.dot(item_sim[2, rated_mask], user_ratings[rated_mask]) \
       / np.sum(np.abs(item_sim[2, rated_mask]))

Limitations of memory-based CF: - O(m·n) memory; O(mk) inference - Sparse matrix — few overlapping ratings for similarity - Does not scale to 10M+ users/items

Matrix Factorization

Decompose R ≈ P · Qᵀ where: - P ∈ ℝ^{m×k}: user latent factor matrix - Q ∈ ℝ^{n×k}: item latent factor matrix - k ≪ min(m, n): latent dimension (e.g., 32–256)

Each user is represented by a vector pᵤ and each item by qᵢ; the predicted rating is:

r̂ᵤᵢ = pᵤᵀ qᵢ (+ user bias bᵤ + item bias bᵢ + global mean μ)

SVD++ includes implicit feedback (which items a user has interacted with, regardless of rating).

Training with SGD

Loss (only over observed ratings, + regularization):

L = Σ_{(u,i) observed} (rᵤᵢ − pᵤᵀqᵢ)² + λ(‖pᵤ‖² + ‖qᵢ‖²)

Gradient updates: epₛᵢ = rᵤᵢ − pᵤᵀqᵢ pᵤ ← pᵤ + α(eᵤᵢ · qᵢ − λ · pᵤ) qᵢ ← qᵢ + α(eᵤᵢ · pᵤ − λ · qᵢ)

# Using surprise library
from surprise import SVD, Dataset, Reader
from surprise.model_selection import cross_validate

reader = Reader(rating_scale=(1, 5))
data   = Dataset.load_from_df(df[["userId","movieId","rating"]], reader)

algo = SVD(n_factors=100, n_epochs=20, lr_all=0.005, reg_all=0.02)
cv_results = cross_validate(algo, data, measures=["RMSE","MAE"], cv=5)
print(f"RMSE: {cv_results['test_rmse'].mean():.4f}")

# Predict
algo.fit(data.build_full_trainset())
pred = algo.predict(uid="42", iid="101")  # user 42, movie 101
print(f"Estimated rating: {pred.est:.2f}")

Content-Based Filtering

Recommend items similar in content (features) to what the user liked.

1. Build an item feature matrix (TF-IDF of descriptions, genre flags, etc.)
2. Build a user profile as weighted average of item features (weighted by ratings)
3. Compute cosine similarity between user profile and all items

from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.metrics.pairwise import cosine_similarity
import numpy as np

# Item descriptions
descriptions = ["action thriller sci-fi", "romance comedy", "sci-fi adventure", ...]

tfidf   = TfidfVectorizer()
item_vecs = tfidf.fit_transform(descriptions)   # (n_items, n_words)

# User rated items: {item_id: rating}
user_ratings = {0: 5, 2: 4, 5: 3}

# User profile: weighted mean of rated item vectors
rated_vecs  = item_vecs[[i for i in user_ratings]]
weights     = np.array([r for r in user_ratings.values()]).reshape(-1, 1)
user_profile = (rated_vecs.multiply(weights)).sum(axis=0) / weights.sum()

# Score all items
scores = cosine_similarity(user_profile, item_vecs).flatten()
top_n  = np.argsort(scores)[::-1][:10]

Advantages: no cold start for items (only needs features), transparent recommendations, no privacy issues (no other users' data needed). Disadvantages: limited to known features; serendipity problem (only recommends similar items); user cold start still exists.

Hybrid Approaches

Combine collaborative and content-based:

LightFM supports hybrid factorization with user/item features:

from lightfm import LightFM
from lightfm.data import Dataset

dataset = Dataset()
dataset.fit(users, items, item_features=item_feature_list)

interactions, weights = dataset.build_interactions([(user, item, rating)])
item_features = dataset.build_item_features([(item, [feat1, feat2])])

model = LightFM(no_components=64, loss="warp",
                learning_rate=0.05, item_alpha=1e-6)
model.fit(interactions, item_features=item_features, epochs=30, num_threads=4)

Neural Collaborative Filtering (NCF)

Replace the dot product in MF with a neural network for more expressive user-item interactions:

class NCF(nn.Module):
    def __init__(self, n_users, n_items, embed_dim=64, layers=[128, 64, 32]):
        super().__init__()
        # GMF path
        self.user_emb_gmf = nn.Embedding(n_users, embed_dim)
        self.item_emb_gmf = nn.Embedding(n_items, embed_dim)
        # MLP path
        self.user_emb_mlp = nn.Embedding(n_users, embed_dim)
        self.item_emb_mlp = nn.Embedding(n_items, embed_dim)
        # MLP layers
        mlp_in = embed_dim * 2
        self.mlp = nn.Sequential(*[
            nn.Sequential(nn.Linear(in_d, out_d), nn.ReLU())
            for in_d, out_d in zip([mlp_in] + layers, layers)
        ])
        self.output = nn.Linear(embed_dim + layers[-1], 1)

    def forward(self, user, item):
        gmf = self.user_emb_gmf(user) * self.item_emb_gmf(item)
        mlp_input = torch.cat([self.user_emb_mlp(user),
                                self.item_emb_mlp(item)], dim=1)
        mlp_out = self.mlp(mlp_input)
        return torch.sigmoid(self.output(torch.cat([gmf, mlp_out], dim=1)))

Two-Tower Model

The dominant architecture for large-scale industrial recommenders (YouTube, Pinterest, TikTok):

• User tower: encode user features → user embedding
• Item tower: encode item features → item embedding
• Score: cosine similarity or dot product at inference

Training: negative sampling (random or hard negatives from in-batch), pairwise or BPR loss.

Inference: user tower runs online; item tower runs offline for all items. Use approximate nearest neighbor search (FAISS, ScaNN) to retrieve top-k items in milliseconds.

# Approximate Nearest Neighbor search with FAISS
import faiss
import numpy as np

d = 128   # embedding dimension
item_embeddings = np.random.randn(1_000_000, d).astype("float32")

index = faiss.IndexFlatIP(d)   # inner product (cosine if normalized)
index = faiss.IndexIVFFlat(faiss.IndexFlatIP(d), d, 1000)  # faster approximate
index.train(item_embeddings)
index.add(item_embeddings)

user_embed = np.random.randn(1, d).astype("float32")
distances, item_ids = index.search(user_embed, k=100)   # top-100 items

Implicit Feedback and BPR Loss

For implicit feedback (clicks, views), Bayesian Personalized Ranking (BPR) optimizes:

L = −Σ_{(u,i,j)} log σ(r̂ᵤᵢ − r̂ᵤⱼ)

where i is an observed item and j is an unobserved item (sampled). This teaches the model to rank observed items higher than unobserved ones.

Recommendation Evaluation Metrics

Always compute offline metrics against held-out interactions, NOT ratings when dealing with implicit feedback.

def ndcg_at_k(recommended, relevant, k=10):
    gains = [1 if item in relevant else 0 for item in recommended[:k]]
    dcg  = sum(g / np.log2(i + 2) for i, g in enumerate(gains))
    ideal = sum(1 / np.log2(i + 2) for i in range(min(len(relevant), k)))
    return dcg / ideal if ideal > 0 else 0.0

Cold Start Strategies

Contextual Recommendation

Incorporate context (time, location, device, weather) into predictions:

• Contextual pre-filtering: filter items by context, then recommend
• Contextual post-filtering: recommend, then filter/re-rank by context
• Contextual modeling: include context features in the model (CARS, FM/DeepFM)

Factorization Machines (FM): efficiently model all pairwise feature interactions:

ŷ = w₀ + Σᵢ wᵢxᵢ + Σᵢ<ⱼ (vᵢ·vⱼ)xᵢxⱼ

where vi are latent vectors. Handles sparse feature combinations (user × item × context).