Neural Networks & Deep Learning

Start here

A neural network is a stack of simple math steps, loosely inspired by brain neurons. Each unit (a neuron) does one thing: multiply inputs by weights, add a bias, pass the sum through a non-linear activation function. Stack thousands of neurons in layers, chain the layers together, and you get a network — "deep" just means many layers.

• Forward pass — run input through the layers to get a prediction.
• Loss — a number measuring how wrong that prediction was.
• Backpropagation — work backward from the loss, using the chain rule, to find how much each weight contributed to the error.
• Gradient descent — nudge every weight slightly in the direction that reduces the error.

This module is about that loop's mechanics — why non-linearity matters, which activation to use where, and what breaks (vanishing gradients, overfitting) when you stack many layers.

Core threads

• Forward pass: input → weighted sum + bias → activation → next layer. Backprop: chain rule applied layer by layer to compute ∂loss/∂weight, then gradient descent updates weights.
• Why non-linearity: stacking linear layers without activation collapses to one linear function — non-linear activations (ReLU, sigmoid, tanh) let the network approximate arbitrary functions (universal approximation).
• Activation functions: sigmoid saturates (vanishing gradient) — used for binary output layers; ReLU is cheap and avoids saturation for positive inputs but can "die"; softmax turns logits into a probability distribution for multi-class output.
• Vanishing/exploding gradients: deep networks multiply many small (or large) derivatives together → gradient shrinks (or blows up) across layers. Fixes: ReLU family, batch norm, residual connections, careful init (Xavier/He), gradient clipping.
• Batch normalization: normalizes layer inputs per mini-batch → stabilizes and speeds up training, acts as mild regularizer.
• Architectures at intuition level: CNNs exploit spatial locality via shared filters (good for images); RNN/LSTM/GRU process sequences with memory (good for time series/text, now mostly replaced by Transformers for NLP).

Viva angle

Be ready to derive the sigmoid derivative on paper (σ(1-σ)), explain why we don't just use a huge single linear layer, and reason about a given train/val loss curve (this doubles as a Module 7 scenario question).