Neural Network Fundamentals

Build everything from first principles: from a single neuron up to a full deep network, forward pass, and the complete backpropagation derivation.

Table of contents

1. From linear models to neural networks
2. The perceptron and its limits
3. Multi-layer perceptron (MLP)
4. Activation functions
5. Forward pass (computation graph)
6. Loss functions for deep learning
7. Backpropagation — full derivation
8. Universal approximation theorem
9. Weight initialization
10. Vanishing and exploding gradients

1. From linear models to neural networks

1.1 Limitation of linear classifiers

A single linear model can only separate classes with a hyperplane. For the XOR problem:

No single line separates the 0s from the 1s. Minsky & Papert (1969) showed the perceptron cannot learn XOR — this was a key insight motivating multi-layer networks.

1.2 The key idea: learned feature transformations

Instead of hand-engineering features, learn a hierarchy of transformations:

Each layer transforms its input into a new representation. The final layer is (usually) a linear classifier on top of these learned features.

2. The perceptron and its limits

2.1 Original perceptron (Rosenblatt, 1958)

A single neuron:

Perceptron learning rule: for misclassified sample $(\mathbf{x}_i, y_i)$:

Perceptron convergence theorem: if data is linearly separable, the algorithm converges in a finite number of steps bounded by (\|\mathbf{x}\|_\max \cdot \|\mathbf{w}^\star\| / \text{margin})^2.

Limitation: non-differentiable step function → cannot use gradient descent. Solution: smooth differentiable activations.

3. Multi-layer perceptron (MLP)

3.1 Architecture

An MLP with $L$ layers. Denote:

• $\mathbf{a}^{(0)} = \mathbf{x}$ (input, dimension $n_0$)
• Layer $l$: weight matrix $\mathbf{W}^{(l)} \in \mathbb{R}^{n_l \times n_{l-1}}$, bias $\mathbf{b}^{(l)} \in \mathbb{R}^{n_l}$
• $n_l$ = number of neurons in layer $l$; $n_L$ = output size

Pre-activation (linear combination):

Post-activation (element-wise nonlinearity):

where $\sigma^{(l)}$ is an activation function.

Final output: $\hat{\mathbf{y}} = \mathbf{a}^{(L)}$.

3.2 Parameters

Total parameters:

4. Activation functions

4.1 Sigmoid

Derivative: $\sigma'(z) = \sigma(z)(1-\sigma(z))$. Max derivative = 0.25 (at $z=0$).

Problems:

• Saturates at large $|z|$: gradient → 0 (vanishing gradient).
• Not zero-centered: outputs always positive → all-positive or all-negative gradients for weights, causing zig-zag updates.
• Expensive ($\exp$).

Use: output layer for binary classification.

4.2 Tanh

Derivative: $\tanh'(z) = 1 - \tanh^2(z)$. Max derivative = 1 (at $z=0$).

Zero-centered (advantage over sigmoid), but still saturates at large $|z|$.

4.3 ReLU (Rectified Linear Unit)

Derivative: $\text{ReLU}'(z) = \mathbf{1}[z > 0]$ (subgradient: set to 0 at $z=0$).

Advantages:

• Does not saturate for $z > 0$.
• Sparse activation (exactly 0 for $z < 0$) — natural regularization.
• Fast to compute.
• Default choice for hidden layers in modern networks.

Problems:

• Dying ReLU: if $z < 0$ always (e.g., large negative bias, large learning rate update), gradient = 0 → neuron never updates.
• Not zero-centered.
• Unbounded (no saturation for large positive $z$).

4.4 Leaky ReLU and variants

Fixes dying ReLU: small but nonzero gradient for negative inputs.

PReLU (Parametric ReLU): $\alpha$ is learned per-neuron.

ELU (Exponential Linear Unit):

Smooth, zero-centered in expectation, theoretically motivated — but slower than ReLU.

4.5 GELU (Gaussian Error Linear Unit)

where $\Phi$ is the standard Gaussian CDF. Approximation: $\text{GELU}(z) \approx 0.5z(1 + \tanh[\sqrt{2/\pi}(z + 0.044715z^3)])$.

Stochastic interpretation: GELU = expected value of $z \cdot \mathbf{1}[z > \epsilon]$ where $\epsilon \sim \mathcal{N}(0,1)$. "Soft" stochastic gating.

Default activation in BERT, GPT, and most modern transformers.

4.6 Softmax (output layer, multi-class)

Properties: outputs sum to 1, all positive → valid probability distribution.

Numerical stability: subtract $\max_j z_j$ before exponentiating (doesn't change value):

Jacobian of softmax: $\partial \text{softmax}(\mathbf{z})_k / \partial z_j = p_k(\delta_{kj} - p_j)$, where $\delta_{kj}$ is Kronecker delta.

4.7 Comparison summary

5. Forward pass (computation graph)

5.1 Example: 2-layer network

Input $\mathbf{x} \in \mathbb{R}^{n_0}$, hidden layer $\mathbf{h} \in \mathbb{R}^{n_1}$, output scalar $\hat{y}$.

5.2 Vectorized over a mini-batch

For mini-batch $\mathbf{X} \in \mathbb{R}^{B \times n_0}$ (batch size $B$):

All operations are matrix multiplications → parallelizable on GPU.

6. Loss functions for deep learning

6.1 Binary cross-entropy

For binary labels $y_i \in \{0,1\}$, output $\hat{y}_i = \sigma(z_i) \in (0,1)$:

6.2 Categorical cross-entropy

For one-hot labels $\mathbf{y}_i \in \{0,1\}^K$, output $\hat{\mathbf{p}}_i = \text{softmax}(\mathbf{z}_i)$:

6.3 MSE (regression)

6.4 Huber loss (robust regression)

Quadratic near zero (smooth), linear for large errors (robust to outliers).

6.5 Numerical trick: log-sum-exp + softmax

In practice, compute cross-entropy directly from logits (numerically stable):

This is what torch.nn.CrossEntropyLoss and tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True) do internally.

7. Backpropagation — full derivation

7.1 The core idea

Backpropagation is reverse-mode automatic differentiation: efficiently compute $\partial \mathcal{L}/\partial \theta$ for all parameters $\theta$ by reusing computations via the chain rule.

Two passes:

1. Forward pass: compute and cache all intermediate values $\mathbf{z}^{(l)}, \mathbf{a}^{(l)}$.
2. Backward pass: propagate gradients from loss back to inputs using the chain rule.

7.2 Define the error signal

Define $\boldsymbol{\delta}^{(l)}$ (delta / error) as the gradient of the loss w.r.t. the pre-activation of layer $l$:

7.3 Output layer delta

For softmax + cross-entropy (very clean result):

Derivation: let $\mathcal{L} = -\sum_k y_k \log p_k$ where $p_k = \text{softmax}(z_k)$.

7.4 Backpropagating through a layer

By the chain rule, from layer $l+1$ to layer $l$:

where $\odot$ is element-wise multiplication and $\sigma'$ is the derivative of the activation.

7.5 Gradients w.r.t. parameters

For a mini-batch of size $B$:

where $\boldsymbol{\Delta}^{(l)} \in \mathbb{R}^{B \times n_l}$ contains $\boldsymbol{\delta}^{(l)}$ for each sample.

7.6 Complete backprop algorithm

Forward pass:
  for l = 1 to L:
    z[l] = W[l] @ a[l-1] + b[l]
    a[l] = activation(z[l])
  L = loss(a[L], y)

Backward pass:
  delta[L] = dL/dz[L]         # output layer delta
  for l = L-1 downto 1:
    dL/dW[l] = delta[l] @ a[l-1].T / B
    dL/db[l] = mean(delta[l], axis=0)
    delta[l] = (W[l+1].T @ delta[l+1]) * sigma'(z[l])

Parameter update (SGD):
  for l = 1 to L:
    W[l] -= lr * dL/dW[l]
    b[l] -= lr * dL/db[l]

7.7 Complexity

Forward pass: $O(\sum_l n_l \cdot n_{l-1})$ operations. Backward pass: same order — backprop has the same computational cost as 2 forward passes.

Memory: need to cache all activations $\mathbf{a}^{(l)}$ during forward pass for use in backward pass. Memory $\propto$ depth × batch size × layer size.

7.8 Automatic differentiation (autograd)

Modern frameworks (PyTorch, JAX, TensorFlow) implement dynamic or static computation graphs and handle backprop automatically via:

• Forward mode AD: accumulate derivatives forward. Efficient for $f: \mathbb{R} \to \mathbb{R}^n$.
• Reverse mode AD (backprop): accumulate derivatives backward. Efficient for $f: \mathbb{R}^n \to \mathbb{R}$ (i.e., scalar loss). This is what we always use in DL.

import torch
import torch.nn as nn

x = torch.tensor([1.0, 2.0], requires_grad=True)
W = torch.randn(3, 2, requires_grad=True)
loss = ((W @ x)**2).sum()
loss.backward()
print(W.grad)   # dL/dW computed automatically

8. Universal approximation theorem

8.1 Statement

Cybenko (1989) / Hornik (1991): An MLP with a single hidden layer of sufficiently many neurons and a non-polynomial activation function can approximate any continuous function on a compact subset of $\mathbb{R}^n$ to arbitrary precision.

Formally: for any $\varepsilon > 0$ and continuous $f: [0,1]^n \to \mathbb{R}$, there exists an MLP $g$ with one hidden layer such that $\sup_\mathbf{x} |f(\mathbf{x}) - g(\mathbf{x})| < \varepsilon$.

8.2 Implications and limits

• Existence does not imply learnability (may require exponentially many neurons, or gradient descent may not find the solution).
• Width vs depth tradeoffs: deep networks with $O(\text{poly})$ neurons can represent functions that require exponential neurons in a shallow network.
• The theorem motivates why neural nets can work, but not how to train them.

9. Weight initialization

9.1 Why initialization matters

• All-zero init: all neurons produce the same gradient (symmetry) → learn the same features. Symmetry breaking requires random init.
• Too-large init: activations saturate or explode.
• Too-small init: vanishing gradients.

9.2 Variance analysis

For layer $l$ with $n_l$ inputs, if weights $W_{ij} \sim \mathcal{N}(0, \sigma^2)$ and inputs have unit variance:

To keep variance stable across layers: $\sigma^2 = 1/n_{l-1}$.

9.3 Xavier / Glorot initialization

Derived for tanh/sigmoid activations. Balances forward and backward variance:

9.4 He / Kaiming initialization

Derived for ReLU activations. Since ReLU zeros out half the inputs on average:

Derivation: For ReLU, $\mathbb{E}[\text{ReLU}(z)^2] = \frac{1}{2}\mathbb{E}[z^2]$ (half the distribution contributes). To keep variance constant: $n_{l-1} \cdot \sigma^2 \cdot \frac{1}{2} = 1 \Rightarrow \sigma^2 = 2/n_{l-1}$.

# PyTorch uses Kaiming uniform by default for Conv and Linear
nn.init.kaiming_normal_(layer.weight, mode='fan_in', nonlinearity='relu')
nn.init.xavier_uniform_(layer.weight)

9.5 Orthogonal initialization

Initialize weight matrices as random orthogonal matrices. Preserves gradient norms exactly in linear networks; works well for RNNs.

10. Vanishing and exploding gradients

10.1 Formal analysis

In a deep network, the gradient of the loss w.r.t. early layer weights:

This is a product of $L-1$ matrices (each element also multiplied by a scalar derivative).

If the eigenvalues of $\mathbf{W}^{(l)\top} \cdot \text{diag}(\sigma')$ are $< 1$: gradients shrink exponentially → vanishing gradients. If eigenvalues are $> 1$: gradients grow exponentially → exploding gradients.

10.2 Sigmoid/tanh saturation

Sigmoid derivative: $\max = 0.25$ (at $z=0$). For all $|z| > 2$: derivative $< 0.1$. Product of 10 layers: $0.25^{10} \approx 10^{-6}$. Early layers receive negligible gradients.

10.3 Solutions

10.4 Skip connections (ResNets preview)

The $+\mathbf{a}^{(l)}$ term creates a direct path for gradients:

The identity $\mathbf{I}$ ensures gradients can flow even if $\partial F/\partial \mathbf{a}^{(l)} \approx 0$.

*File: notes/11_neural_network_fundamentals.md — next: notes/12_training_deep_networks.md*