CNNs

The architecture behind nearly all computer vision breakthroughs. Understand the math of convolution, the design principles, and the landmark architectures.

Table of contents

1. Motivation: why convolutions for images
2. Discrete convolution (math)
3. Convolutional layer parameters and output size
4. Pooling layers
5. Receptive field
6. Backpropagation through convolution
7. CNN building blocks: BatchNorm, activation, skip connections
8. Landmark architectures
9. Transfer learning and fine-tuning
10. Object detection and segmentation (concepts)

1. Motivation: why convolutions for images

1.1 Problems with fully connected networks on images

A $224 \times 224 \times 3$ image has $\approx$ 150K pixels. A single FC layer mapping to 1000 hidden units has $150000 \times 1000 = 150M$ parameters — and that's one layer.

Worse: FC layers treat every pixel independently. They ignore the spatial structure of images: nearby pixels are correlated, patterns (edges, textures) appear in multiple locations.

1.2 Two key inductive biases

Translation equivariance: if a pattern (e.g., an edge) appears at position $(i,j)$ or $(i+5, j+3)$, the same detector should fire. Convolutions achieve this by sharing the same filter weights across all positions.

Locality: nearby pixels are more related than far-away pixels. Convolutions use small filters (e.g., $3\times3$) to capture local patterns.

These two biases hugely reduce parameters and encode useful image priors.

2. Discrete convolution (math)

2.1 1D convolution

For input $x \in \mathbb{R}^W$ and filter $k \in \mathbb{R}^F$:

In signal processing, true convolution flips the kernel: $(x \star k)[i] = \sum_f x[i-f]k[f]$. In deep learning the term "convolution" is used loosely — most frameworks implement cross-correlation (no flip). Since filters are learned, the flip doesn't matter.

2.2 2D convolution (single channel)

Input $\mathbf{X} \in \mathbb{R}^{H \times W}$, filter $\mathbf{K} \in \mathbb{R}^{k_H \times k_W}$:

Output size (no padding, stride 1): $(H - k_H + 1) \times (W - k_W + 1)$.

2.3 Multi-channel convolution

Input $\mathbf{X} \in \mathbb{R}^{C_\text{in} \times H \times W}$ ($C_\text{in}$ channels), filter $\mathbf{K} \in \mathbb{R}^{C_\text{out} \times C_\text{in} \times k_H \times k_W}$ ($C_\text{out}$ filters):

Output $\mathbf{Y} \in \mathbb{R}^{C_\text{out} \times H' \times W'}$.

Parameters per conv layer: $C_\text{out} \times C_\text{in} \times k_H \times k_W + C_\text{out}$ (bias).

3. Convolutional layer parameters and output size

3.1 Padding

Adding zeros around the input border.

• Valid (no padding): output shrinks by $k-1$ per side.
• Same padding: output size = input size. Padding $p = (k-1)/2$ for odd $k$.

3.2 Stride

Step size when sliding the filter.

Output spatial size (one dimension, e.g., height):

where $p$ = padding, $s$ = stride.

Example: $H=32$, $k=3$, $p=1$, $s=1$ → $H_\text{out} = \lfloor(32+2-3)/1\rfloor+1 = 32$ (same size). With $s=2$: $H_\text{out} = \lfloor(32+2-3)/2\rfloor+1 = 16$ (halved).

3.3 Dilation (atrous convolution)

Insert gaps between filter elements: with dilation $d$, filter $k$ covers a spatial range of $k + (k-1)(d-1) = d(k-1)+1$.

Output size: $H_\text{out} = \lfloor(H + 2p - d(k-1) - 1)/s\rfloor + 1$.

Effect: expands receptive field without increasing parameters or losing resolution. Used in semantic segmentation (DeepLab), WaveNet.

3.4 Depthwise separable convolution

Standard conv $C_\text{in} \times H \times W \to C_\text{out} \times H' \times W'$: $O(C_\text{in} \cdot C_\text{out} \cdot k^2 \cdot H' \cdot W')$ multiplications.

Depthwise separable (MobileNet idea):

1. Depthwise conv: apply one filter per input channel independently. Params: $C_\text{in} \times k^2$.
2. Pointwise conv: $1\times1$ convolution to mix channels. Params: $C_\text{in} \times C_\text{out}$.

Cost reduction factor: $\frac{1}{C_\text{out}} + \frac{1}{k^2}$. For $k=3, C_\text{out}=256$: ~$9\times$ fewer multiplications.

3.5 1×1 convolution

$k_H = k_W = 1$: acts as a pointwise linear combination of channels at each spatial location. Used to:

• Change number of channels (up- or down-project).
• Add nonlinearity without changing spatial size.
• Bottleneck in ResNets (reduce channels before expensive $3\times3$ conv).

4. Pooling layers

4.1 Max pooling

Takes max over a local region:

Backprop: gradient passed only to the location that achieved the maximum (others get 0). Uses a "switch" mask stored during forward pass.

4.2 Average pooling

Global average pooling (GAP): pool over entire spatial dimension → one value per channel. Replaces large FC layers at classifier head in modern architectures (ResNet, EfficientNet).

Why GAP beats FC: fewer parameters, translation invariant, acts as structural regularizer.

4.3 Strided convolution vs pooling

Modern architectures (ResNets, Transformers) increasingly use strided convolutions instead of pooling for spatial downsampling. Learned downsampling can preserve more information.

5. Receptive field

5.1 Definition

The receptive field of a neuron = the region of the input that influences its value.

For a stack of conv layers each with kernel size $k$ and stride 1:

• Layer 1: receptive field = $k \times k$.
• Layer 2: each neuron at layer 2 sees a $k \times k$ region of layer 1, each of which sees $k \times k$ of input → total $(2(k-1)+1)^2 = (2k-1)^2$ for stride 1.

Formula for $L$ layers of kernel $k$, stride 1:

With stride $s$ (effective): $\text{RF} = 1 + (k-1)\sum_{l=1}^{L}\prod_{l'=1}^{l-1} s_{l'}$.

5.2 Effective receptive field

In practice, not all pixels in the theoretical RF contribute equally. Neurons near the center contribute more. The effective RF is much smaller than the theoretical RF (approximately Gaussian-shaped). This motivates design choices like larger kernels, dilated convolutions, and pooling.

6. Backpropagation through convolution

6.1 Gradient w.r.t. input (needed for backprop through earlier layers)

This is a convolution of the output gradient with the flipped kernel — i.e., a transposed convolution (deconvolution).

6.2 Gradient w.r.t. filters

Again a cross-correlation between input and output gradients.

7. CNN building blocks: BatchNorm, activation, skip connections

7.1 Standard block pattern

Input → Conv(3×3) → BN → ReLU → Conv(3×3) → BN → (+skip) → ReLU → Output

7.2 ResNet residual block

where $F$ represents the two conv layers. If dimensions differ (stride 2), use a $1\times1$ conv projection on the skip:

Why it works:

• Gradient highway: $\partial\mathcal{L}/\partial\mathbf{x} = \partial\mathcal{L}/\partial\mathbf{y} \cdot (I + \partial F/\partial\mathbf{x})$. The identity term ensures gradient flows even if $\partial F/\partial\mathbf{x} \approx 0$.
• Easy to learn identity ($F \approx 0$) early in training.
• Enables training of very deep networks (50, 100, 1000+ layers).

7.3 Bottleneck block (ResNet-50+)

Reduce channels with $1\times1$, then $3\times3$, then expand with $1\times1$:

Input(256) → 1×1 Conv → 64ch → 3×3 Conv → 64ch → 1×1 Conv → 256ch → + skip

Reduces computation by ~4× vs non-bottleneck for same channel count.

8. Landmark architectures

8.1 LeNet-5 (LeCun, 1998)

First successful deep CNN for digit recognition. Pattern: Conv → Pool → Conv → Pool → FC → FC → Output. Established the basic design.

8.2 AlexNet (Krizhevsky et al., 2012)

Won ImageNet ILSVRC 2012 with top-5 error 15.3% (vs 26.2% previous year). Key innovations:

• ReLU activations (first large-scale use).
• Dropout in FC layers.
• Data augmentation (crops, flips).
• Multi-GPU training.
• Local Response Normalization (now obsolete, replaced by BN).

8.3 VGGNet (Simonyan & Zisserman, 2014)

Insight: replace large kernels with stacks of small $3\times3$ convs. Two $3\times3$ convs have the same receptive field as one $5\times5$ but fewer parameters ($2\times9C^2$ vs $25C^2$) and an extra nonlinearity. VGG-16/19 became standard feature extractors.

8.4 GoogLeNet / Inception (Szegedy et al., 2014)

Inception module: apply $1\times1$, $3\times3$, $5\times5$ convolutions and $3\times3$ max pooling in parallel; concatenate outputs along channel dimension. Captures features at multiple scales simultaneously.

$1\times1$ bottlenecks reduce channel count before expensive larger convolutions.

8.5 ResNet (He et al., 2015)

Residual connections (see §7.2). Enabled training of 152-layer networks. ResNet-50/101/152 are still widely used. Key insight: depth matters — deeper networks consistently better if we can train them.

8.6 DenseNet (Huang et al., 2017)

Each layer connected to all subsequent layers (not just the next):

where $[\cdot]$ denotes channel concatenation. Dense connectivity: maximum feature reuse, strong gradient flow, few parameters. Good when data is limited.

8.7 EfficientNet (Tan & Le, 2019)

Compound scaling: scale width ($C$), depth ($L$), and resolution ($H, W$) simultaneously via a scaling coefficient $\phi$:

Subject to $\alpha\beta^2\gamma^2 \approx 2$ (FLOP constraint). Achieved state-of-the-art with fewer parameters.

8.8 Architecture evolution summary

9. Transfer learning and fine-tuning

9.1 The idea

Train on large dataset (ImageNet: 1.2M images, 1000 classes), then transfer learned features to a new task.

Why it works: early layers learn general features (edges, textures, blobs) that are useful across tasks. Later layers become task-specific.

9.2 Strategies

9.3 Learning rate for fine-tuning

Use discriminative learning rates: lower LR for early (general) layers, higher for later (task-specific) layers:

9.4 Domain gap matters

Transferring from ImageNet to medical images: gap is large (natural RGB vs X-ray). May need more fine-tuning epochs, or pre-training on domain-specific data.

9.5 Code example

import torch
import torch.nn as nn
import torchvision.models as models

# Load pretrained ResNet-50
model = models.resnet50(weights='IMAGENET1K_V1')

# Option 1: Feature extraction — freeze all
for param in model.parameters():
    param.requires_grad = False

# Replace final FC layer for new task (e.g., 10 classes)
num_features = model.fc.in_features
model.fc = nn.Linear(num_features, 10)   # only this layer trains

# Option 2: Fine-tune all — use small LR
for param in model.parameters():
    param.requires_grad = True
# Use optimizer with LR ~1e-4 (much smaller than scratch ~1e-1)

optimizer = torch.optim.Adam([
    {'params': model.layer4.parameters(), 'lr': 1e-4},
    {'params': model.fc.parameters(),     'lr': 1e-3},
])

10. Object detection and segmentation (concepts)

10.1 Task definitions

10.2 Two-stage detectors (R-CNN family)

R-CNN: region proposals → CNN features per region → classifier. Fast R-CNN: CNN features for whole image → ROI pooling extracts region features. Faster R-CNN: replaces selective search with Region Proposal Network (RPN) — a small CNN that outputs objectness scores and box proposals from feature map.

RPN loss:

where $\hat{p}$ = objectness score, $\hat{t}$ = box regression offsets.

10.3 One-stage detectors

YOLO (You Only Look Once): divide image into $S\times S$ grid. Each cell predicts $B$ boxes and $C$ class probabilities in one pass.

SSD (Single Shot Detector): default boxes at multiple scales from different feature map levels.

One-stage vs two-stage:

• Two-stage: better accuracy, slower.
• One-stage: faster (real-time possible), slightly lower accuracy (modern ones close the gap).

10.4 Semantic segmentation

Fully Convolutional Network (FCN): replace FC layers with $1\times1$ convs → output spatial map. Upsampling via transposed convolution or bilinear interpolation.

U-Net: encoder-decoder with skip connections between encoder and decoder at same resolution. Enables precise localization. Dominant in medical imaging.

DeepLab: dilated convolutions for large receptive field without downsampling. Atrous Spatial Pyramid Pooling (ASPP): parallel dilated convs with different dilation rates → multi-scale features.

10.5 IoU (Intersection over Union)

Standard metric for boxes/masks:

mAP = mean AP over classes, where AP is area under precision-recall curve at IoU threshold (e.g., 0.5, or averaged over 0.5:0.95).

*File: notes/13_cnns.md — next: notes/14_rnns_and_sequences.md*