Perceptrons & Neural Networks
One artificial neuron draws one straight line. Stack them, and they can draw almost anything.
From brain cells to a formula
A biological neuron receives signals through dendrites, sums them, and "fires" along its axon if the total is strong enough. McCulloch & Pitts (1943) turned this into a logic unit. An artificial neuron computes a weighted sum of its inputs plus a bias, then applies an activation function:
The weights say how important each input is; the bias shifts the firing threshold. Learning means finding good weights.
The perceptron learns a line
With two inputs, the perceptron outputs 1 on one side of the line w₁x₁ + w₂x₂ + b = 0 and 0 on the other side. It is a linear classifier. Rosenblatt's learning rule nudges the weights after every mistake:
Worked example: learning AND (η = 1, start w = (0, 0), b = 0)
| Epoch | Mistakes | Weights after epoch (w₁, w₂, b) |
|---|---|---|
| 1 | 1 (on input 1,1) | (1, 1, 1) |
| 2 | 3 | (2, 1, 0) |
| 3 | 3 | (2, 1, −1) |
| 4 | 2 | (2, 2, −1) |
| 5 | 1 | (2, 1, −2) |
| 6 | 0 ✓ | (2, 1, −2): 2x₁ + x₂ − 2 > 0 only for (1, 1) |
Epoch 1 in detail: inputs (0,0), (0,1), (1,0) give z = 0, so ŷ = 0, which is correct. Input (1,1) gives ŷ = 0 but y = 1, error +1, so w ← (0+1, 0+1) and b ← 0+1. The perceptron convergence theorem guarantees: if a separating line exists, the rule finds one in a finite number of steps. You can reproduce this exact table in the demo below.
Fix XOR with a hidden layer: train a small 2-3-1 sigmoid network with backpropagation.
The background colour will show the network's output: red ≈ 1, white ≈ 0.
The XOR problem, and the first neural winter
XOR outputs 1 when exactly one input is 1: (0,1) and (1,0) are positive, (0,0) and (1,1) are negative. No single straight line separates them, so a single perceptron can never learn XOR. Minsky & Papert proved such limits in Perceptrons (1969), and enthusiasm and funding for neural networks dropped for over a decade.
The fix: hidden layers
Combine simple lines: XOR = (x₁ OR x₂) AND NOT(x₁ AND x₂). Hand-set step-neuron weights:
| x₁, x₂ | h₁ (OR) | h₂ (NAND) | y = XOR |
|---|---|---|---|
| 0, 0 | 0 | 1 | 0 |
| 0, 1 | 1 | 1 | 1 |
| 1, 0 | 1 | 1 | 1 |
| 1, 1 | 1 | 0 | 0 |
A multilayer perceptron (MLP) stacks layers: input → one or more hidden layers → output. With nonlinear activations, a network with enough hidden units can approximate any continuous function to any accuracy (the universal approximation theorem). The hard part is learning the weights.
How networks learn: loss, gradients, backpropagation
- Forward pass: compute the prediction ŷ layer by layer.
- Loss: measure the error, e.g. MSE = (y − ŷ)² or cross-entropy for classification.
- Backward pass (backpropagation): use the chain rule to compute ∂Loss/∂w for every weight, from the output back to the input.
- Gradient descent: w ← w − η · ∂Loss/∂w, a small step downhill (the same idea as Chapter 3's continuous local search).
- Repeat over many epochs (passes over the data), usually on mini-batches.
Backpropagation was popularised by Rumelhart, Hinton & Williams (1986). The step function has zero gradient almost everywhere, which is why trained networks use activations with useful derivatives, like sigmoid or ReLU (ReLU is differentiable everywhere except at 0, where a convention is used).
The network guesses, measures how wrong it was, figures out which weights were most to blame, and nudges each one a little in the direction that reduces the error. It does this millions of times.
Deep learning: many layers, learned features
Deep learning uses networks with many hidden layers. Early layers learn simple features (edges), later ones learn combinations (eyes, wheels, words). Nobody hand-designs the features. The big 2012 breakthrough (AlexNet) came from three things together: large datasets, GPUs and better training tricks.
| Architecture | Key idea | Typical use |
|---|---|---|
| MLP (fully connected) | Every neuron connects to every neuron in the next layer | Tabular data, small problems |
| CNN (convolutional) | Small filters slide across an image and share weights | Images, defect detection, medical scans |
| RNN / LSTM | A hidden state carries memory along a sequence | Older speech and text models, time series |
| Transformer | Attention: each token weighs the other tokens (in LLMs, only the earlier ones) | LLMs, translation, also vision (Chapter 10) |
Neural networks also power deep reinforcement learning (DQN from Chapter 5): the network replaces the giant Q-table.
Training pitfalls
- Overfitting: millions of weights can memorise the training set. Remedies: more data, validation-based early stopping, dropout, weight decay (regularisation).
- Learning rate: too high and the loss explodes; too low and training takes forever.
- Local minima and plateaus: like hill climbing, gradient descent can stall. Random initialisation and momentum-based optimisers (e.g. Adam) help.
- Black box: it's hard to explain why a deep net decided something. That matters for high-risk uses (Chapter 10).
Face unlock on phones (CNNs), voice assistants (speech networks), machine translation and chatbots (Transformers), quality inspection on factory lines, crop monitoring from satellite images, and AlphaFold's protein-structure prediction (2020) are all deep neural networks.
Rosenblatt's Mark I Perceptron (1958) was a physical machine: a 20×20 grid of light sensors wired to motor-driven potentiometers that physically turned to change the weights. The press predicted it would soon "walk, talk, see, write". It took about 60 more years and a few billion times more computing power.
Myth: "Artificial neural networks work just like the brain." Fact: they are loosely inspired by neurons, but real neurons use spikes, chemistry and feedback in ways ANNs don't model. Backpropagation is not how the brain is known to learn.
1. Why can't a single perceptron learn XOR?
2. With w = (2, 1), b = −2 and a step activation, what is the output for input (1, 0)?
3. What does backpropagation compute?
