Video and Multimodal Models

Video as a Modality

Video extends images with a temporal dimension: a video of T frames at resolution H×W×C is a tensor of shape (T, H, W, C). This introduces new challenges:

• Temporal modeling: action, motion, and causality span frames
• Massive data: a 1-minute 1080p video at 30fps = 1800 frames
• Temporal redundancy: adjacent frames are highly correlated
• Motion blur, occlusion, camera movement: unique visual phenomena

The core question: how to efficiently model BOTH spatial content and temporal dynamics?

Video Representations

Frame-Level (Image Backbone + Temporal Aggregation)

Extract CNN/ViT features per frame, then aggregate: - Mean/max pooling: fast but loses temporal order - LSTM on frame features: sequential, captures order - Temporal convolution over features: parallelizable

3D Convolutions

Extend 2D spatial convolutions to 3D (space-time):

Conv3D: filter size (t × h × w), processes spatial and temporal simultaneously.

C3D: first major 3D-CNN architecture for action recognition. I3D (Inflated 3D): initialize 3D conv from pre-trained 2D ImageNet weights by repeating kernels along time axis.

import torchvision.models.video as vm

# R3D-18: 3D ResNet
model = vm.r3d_18(weights="DEFAULT")
# Input: (batch, channels, time, height, width) = (B, 3, 16, 112, 112)
out = model(video_tensor)

# Slow-Fast networks
model = vm.slowfast_r50(weights="DEFAULT")

Factorized 3D convolutions (R(2+1)D): decompose 3D into 2D spatial + 1D temporal conv. Fewer parameters, more effective:

3D conv → Spatial 2D conv + Temporal 1D conv

Video Transformers

TimeSformer: factorized space-time attention. For each token (patch), apply: 1. Temporal attention: attend to same spatial position across T frames 2. Spatial attention: attend to all spatial positions within same frame

ViViT: 4 variants of space-time attention factorization.

VideoMAE: masked autoencoder for video — mask 90%+ of spatiotemporal patches and reconstruct. Extremely efficient pre-training.

from transformers import VideoMAEModel, VideoMAEImageProcessor

image_processor = VideoMAEImageProcessor.from_pretrained("MCG-NJU/videomae-base")
model = VideoMAEModel.from_pretrained("MCG-NJU/videomae-base")

# frames: list of PIL images
inputs = image_processor(list(frames), return_tensors="pt")
outputs = model(**inputs)
# outputs.last_hidden_state: (batch, n_patches*T, hidden_size)

Video Generation

Temporal Consistency Challenge

Generating video requires each frame to be coherent with neighbors — hard to ensure with per-frame generation.

Key insight: temporal coherence is analogous to long-range dependencies in text — attention mechanisms handle both.

Video Diffusion Models

Extend image diffusion to 3D data:

Make-A-Video, Imagen Video (2022): extend text-to-image diffusion by adding temporal attention layers and fine-tuning on video.

Sora (OpenAI, 2024): Diffusion Transformer (DiT) operating on video spacetime patches: - Videos treated as sequences of visual patches (analogous to token sequences in NLP) - Full spatial + temporal attention in a Transformer - Generates up to 1 minute of 1080p video

Veo 3 (Google, 2025) and Sora 2 (OpenAI, 2025) extend this recipe with synchronized audio generation and better physics/temporal consistency.

Latent Video Diffusion

Compress video to a 3D latent space with a video VAE, then run diffusion in latent space:

Video → Video Encoder → Latent (T/4, H/8, W/8, C) → Diffusion → Decode → Video

CogVideoX, HunyuanVideo, Wan 2.x, LTX-Video, Mochi: open-weight latent video diffusion (DiT backbones with full spatiotemporal attention).

from diffusers import CogVideoXPipeline
import torch

pipe = CogVideoXPipeline.from_pretrained(
    "THUDM/CogVideoX-5b",
    torch_dtype=torch.bfloat16
).to("cuda")

video = pipe(
    prompt="A panda eating bamboo in a forest",
    num_frames=49,
    num_inference_steps=50,
    guidance_scale=6.0
).frames[0]

Autoregressive Video Generation

Generate frame by frame conditioned on all previous frames (or their latent codes):

MAGVIT-v2: tokenize video frames with 3D VQVAE, then apply masked or autoregressive transformer. Strong video generation + prediction + editing.

Multimodal Models

A multimodal model processes and/or generates content across multiple modalities (text, image, audio, video) in a unified framework.

Why Multimodal?

Real-world intelligence requires connecting concepts across modalities: - "a red ball" — the text and the image are the same concept - A speech signal and its transcript are the same content - A video and its caption complement each other

CLIP (Contrastive Language-Image Pre-training)

Trains two encoders (image and text) to produce matching embeddings for paired data using contrastive loss:

L = −Σᵢ log(exp(sim(Iᵢ,Tᵢ)/τ) / Σⱼ exp(sim(Iᵢ,Tⱼ)/τ))

Each image embedding should be closer to its matching text than to any other text in the batch (and vice versa). Scales to 400M+ image-text pairs.

import torch
import clip

model, preprocess = clip.load("ViT-B/32", device="cuda")

image = preprocess(Image.open("dog.jpg")).unsqueeze(0).to("cuda")
text  = clip.tokenize(["a dog", "a cat", "a horse"]).to("cuda")

with torch.no_grad():
    image_features = model.encode_image(image)
    text_features  = model.encode_text(text)

    # Cosine similarity
    image_features /= image_features.norm(dim=-1, keepdim=True)
    text_features  /= text_features.norm(dim=-1, keepdim=True)
    similarity = (100.0 * image_features @ text_features.T).softmax(dim=-1)

print(f"Dog: {similarity[0,0]:.1%}, Cat: {similarity[0,1]:.1%}")

Applications of CLIP embeddings: zero-shot classification, image retrieval, guiding diffusion models, multimodal search.

Vision-Language Models (VLMs)

Combine a vision encoder with a language model decoder to enable visual question answering, image captioning, and visual reasoning.

Architecture pattern:

Image → ViT Encoder → Image Features → Projection Layer → LLM → Text output
Text prompt ─────────────────────────────────────────────────↗

LLaVA (Large Language and Vision Assistant): 1. CLIP ViT-L/14 image encoder 2. MLP projection to match LLM token dimension 3. Llama/Vicuna/Mistral language model

from transformers import LlavaNextProcessor, LlavaNextForConditionalGeneration
from PIL import Image

processor = LlavaNextProcessor.from_pretrained("llava-hf/llava-v1.6-mistral-7b-hf")
model     = LlavaNextForConditionalGeneration.from_pretrained(
    "llava-hf/llava-v1.6-mistral-7b-hf", torch_dtype=torch.float16
).to("cuda")

image = Image.open("chart.png")
conversation = [
    {"role": "user", "content": [
        {"type": "image"},
        {"type": "text", "text": "What does this chart show?"}
    ]}
]
prompt = processor.apply_chat_template(conversation, add_generation_prompt=True)
inputs = processor(prompt, image, return_tensors="pt").to("cuda")

output = model.generate(**inputs, max_new_tokens=200)
response = processor.decode(output[0], skip_special_tokens=True)

Frontier Multimodal Models

Audio-Language Models

Whisper (OpenAI): encoder-decoder Transformer trained on 680k hours of multilingual audio. Speech-to-text with near-human accuracy.

from transformers import pipeline

asr = pipeline("automatic-speech-recognition",
               model="openai/whisper-large-v3",
               device="cuda")
result = asr("audio.mp3", return_timestamps=True)
print(result["text"])

AudioCraft (Meta): AudioGen for text-to-audio, MusicGen for text-to-music.

Image-to-3D and Video-to-3D

NeRF (Neural Radiance Fields): fit a 5D function (x,y,z,θ,φ) → (color, density) to multi-view images. Enables novel view synthesis.

Gaussian Splatting (3DGS): represent scenes as collections of 3D Gaussians, optimized from photos. Real-time rendering at high quality.

Zero123, SyncDreamer: generate novel views from a single image using diffusion models.

Multimodal Embeddings and Alignment

ImageBind aligns 6 modalities via binding through images — since images are paired with text, audio, video, etc., the image modality acts as an anchor to align all others.

Video Understanding Tasks

SAM 2 (Segment Anything Model 2): extends SAM to video — prompt a point or box in one frame, track the segmentation mask through the video.