Neural Networks
From a single perceptron to deep networks — weights, bias and activation, gradient descent, backpropagation, multi-layer perceptrons and recurrent networks.
On this page
- The Artificial Neuron
- The Single-Cell Perceptron
- The Single-Layer Perceptron & its limits
- How a perceptron learns — Gradient Descent
- Backpropagation
- Multi-Layer Perceptron (MLP)
- Activation Functions
- Loss Functions & Optimizers
- Recurrent Neural Networks (RNN)
- Beyond MLPs — CNNs & Transformers
- Practice Questions
The Artificial Neuron
Neural networks are the engine of Deep Learning — ML with networks of many layers. They are loosely inspired by the brain.
| Biological neuron | What it does | Artificial equivalent |
|---|---|---|
| Dendrites | Receive signals from other neurons | Inputs (x) |
| Synapses | Control how strong each signal is | Weights (w) |
| Cell body / Soma | Adds up all the signals | Summation (z) |
| Axon | Sends out the final decision | Output (y) |
The Single-Cell Perceptron
The perceptron (Rosenblatt, 1958) was the first algorithmic model of a neuron. It is a tiny decision-maker that does just three steps.
Step C: if z ≥ 0 → output 1 else → output 0 Multiply inputs by weights, add the bias, then apply an activation to decide.
| Component | Role |
|---|---|
| Inputs (x) | The data features |
| Weights (w) | How important each input is — learned during training |
| Bias (b) | A threshold/offset; lets the activation shift so it need not pass through the origin |
| Weighted sum (z) | z = Σ(w·x) + b — the linear part |
| Activation function | Decides if the neuron "fires"; adds non-linearity |
z = (2 × 8) + (1 × 20) + (−15) = 16 + 20 − 15 = 21. Since z ≥ 0 → output 1 → order pizza!
From hard threshold to probability — the sigmoid activation
A plain step ("if z ≥ 0 → 1") is harsh. Often we want a probability, so we pass z through the sigmoid (introduced earlier):
This "dimmer switch" squashes any z into (0, 1): big positive z → ≈1, big negative z → ≈0, z = 0 → 0.5.
import numpy as np
def perceptron(inputs, weights, bias):
z = np.dot(inputs, weights) + bias # weighted sum
return 1 if z >= 0 else 0 # step activation
x = [8, 20] # hunger, money
w = [2, 1] # weights
b = -15 # bias
print("Decision:", perceptron(x, w, b)) # 1 = order pizza
The Single-Layer Perceptron & its limits
A single-layer perceptron is one layer of neurons mapping inputs directly to outputs. It is fundamentally a linear classifier — it draws a single straight line (or hyperplane) to separate two classes.
The fix? Stack neurons into hidden layers — a Multi-Layer Perceptron — which can learn non-linear boundaries and solve XOR.
How a perceptron learns — Gradient Descent
The update rule
Forward pass: prediction = 0.1·44.5 + 0.2·39.3 + 0 = 4.45 + 7.86 = 12.31.
Error: 10.4 − 12.31 = −1.91 (predicted too high).
Update (learning rate 0.0001): gradient term −2(y−ŷ) = 3.82. For w₁: slope = 3.82×44.5 = 170.1, so w₁_new = 0.1 − 0.0001×170.1 = 0.083.
Re-check: new prediction ≈ 10.96 — the error shrank from −1.91 to −0.56. The network is now strictly better.
Variants of Gradient Descent
- Batch GD — uses the whole dataset for each update (stable but slow).
- Stochastic GD (SGD) — updates after each single sample (fast, noisy).
- Mini-batch GD — updates after a small batch (the practical default).
Backpropagation
The two passes of training
- Forward pass — data flows input → output; the network makes a prediction and the loss is computed.
- Backward pass (backpropagation) — the error is propagated backward; the chain rule computes the gradient ∂Loss/∂w for every weight. Gradient descent then updates the weights.
Training terminology
| Term | Meaning |
|---|---|
| Epoch | One complete pass through the entire training dataset |
| Batch size | Number of samples processed before the weights update once |
| Iteration | One weight update = Total samples ÷ Batch size (per epoch) |
from keras import models, layers
from keras.optimizers import SGD
# A simple single-layer network
model = models.Sequential()
model.add(layers.Dense(1, input_dim=2, activation='sigmoid'))
# loss measures error; SGD does gradient descent + backprop
model.compile(optimizer=SGD(learning_rate=0.01),
loss='mse', metrics=['mae'])
# epochs = full passes over the data
model.fit(X_train, y_train, epochs=10, batch_size=32)
Multi-Layer Perceptron (MLP)
Earlier we saw that a single perceptron is a linear classifier — it cannot solve XOR. The fix is to stack neurons into layers.
The three kinds of layers
- Input layer — receives the raw features X.
- Hidden layer(s) — perform feature extraction; "hidden" because you don't see their internal work.
- Output layer — produces the final prediction.
Forward propagation
Data flows input → output. At each neuron: (1) compute the weighted sum Z = WX + B, then (2) apply a non-linear activation A = f(Z).
Activation Functions
| Function | Range | Use / Notes |
|---|---|---|
| Sigmoid σ(z)=1/(1+e⁻ᶻ) | (0, 1) | Binary classification output (probability). Con: vanishing gradient. |
| Tanh hyperbolic tangent | (−1, 1) | Zero-centred (better than sigmoid). Con: still suffers vanishing gradients. |
| ReLU max(0, z) | [0, ∞) | Industry standard for hidden layers. Fast, fixes vanishing gradient for positive z. Con: "Dead ReLU" (neurons stuck at 0). |
| Softmax | (0, 1), sums to 1 | Output layer for multi-class classification — turns raw scores (logits) into probabilities that sum to 1. |
[2.0, 1.0, 0.1] → Softmax → [0.7, 0.2, 0.1]. The outputs are now probabilities that sum to 1.0, so the network can say "70% class A, 20% class B, 10% class C".
Loss Functions & Optimizers
Loss functions — the scorecard
- MSE (Mean Squared Error) — for regression (predicting numbers).
- Cross-Entropy / Categorical Cross-Entropy — for classification; heavily penalises confident wrong predictions.
Optimizers — how the weights are updated
- SGD — basic stochastic gradient descent; can be slow and stick in local minima.
- Momentum — accumulates velocity in a consistent direction, dampening oscillations.
- Adam — combines momentum with adaptive learning rates; the default choice for most problems.
from keras import models, layers
from keras.layers import Input
# MLP for classifying 28x28 images (flattened to 784) into 10 classes
model = models.Sequential([
Input(shape=(784,)),
layers.Dense(64, activation='relu'), # hidden layer 1
layers.Dense(32, activation='relu'), # hidden layer 2
layers.Dense(10, activation='softmax') # output: 10 classes
])
model.compile(optimizer='adam',
loss='categorical_crossentropy',
metrics=['accuracy'])
model.fit(x_train, y_train, epochs=10,
validation_data=(x_test, y_test))
Recurrent Neural Networks (RNN)
How an RNN remembers
At each time step t, the RNN combines the current input with the previous hidden state:
This loop means the hidden state at the end is a (nested) function of all earlier inputs — so the network "remembers" context when predicting the next word.
What RNNs are used for
- Next-word prediction & text generation
- Sentiment classification of sentences
- Machine translation
- Time-series forecasting (stock prices, weather)
from keras.models import Sequential
from keras.layers import Embedding, SimpleRNN, Dense
model = Sequential()
model.add(Embedding(input_dim=100, output_dim=8)) # word -> vector
model.add(SimpleRNN(16)) # the sequence reader
model.add(Dense(1, activation='sigmoid')) # 0 = negative, 1 = positive
model.compile(loss='binary_crossentropy',
optimizer='adam', metrics=['accuracy'])
model.fit(padded_data, labels, epochs=50)
preds = model.predict(test_pad) # e.g. "movie was good"
print("Prediction:", preds)
Beyond MLPs — CNNs & Transformers
Standard dense (MLP) layers ignore structure. Specialised architectures handle structured data better:
| Architecture | Designed for | Key idea |
|---|---|---|
| CNN (Convolutional NN) | Images / spatial data | Filters that capture spatial patterns (edges, shapes) |
| RNN / LSTM | Sequences / time-series | Hidden state carries memory across time steps |
| Transformer | Text / sequences (modern) | Self-attention — processes the whole sequence at once (see Transformers & LLMs) |
Perceptron arithmetic and the learning process are common exam material.
In a perceptron, what do the weights represent?
Each weight scales how strongly its input influences the decision — exactly like synapse strength in a biological neuron. Weights are what the network learns.
A single-layer perceptron cannot solve the XOR problem because:
A single-layer perceptron is a linear classifier (one straight line). XOR's classes cannot be separated by any single line, so hidden layers are required.
Why is a bias term necessary in a neuron?
Without a bias the decision boundary is locked through the origin. The bias is an offset that shifts the activation left/right, giving the model the flexibility to fit real data.
What happens during training if the learning rate is set to 0?
New weight = old weight − (learning rate × gradient). If the learning rate is 0, the step size is 0, so weights never change and no learning occurs.
Backpropagation uses which mathematical rule to compute gradients?
Backpropagation applies the chain rule layer by layer, from the output backward to the input, to find how each weight affects the final loss.
One complete pass through the entire training dataset is called:
An epoch = one full pass over all training data. A batch is a subset processed before one update; an iteration is one update.
A learning rate that is far too large will most likely cause the loss to:
Too-large steps overshoot the valley of the loss landscape, so the loss bounces around or even grows. Too-small steps make training painfully slow.
A perceptron has inputs x₁=3, x₂=5; weights w₁=0.4, w₂=0.2; bias b=−2. Compute z and the step output (1 if z ≥ 0).
z = (0.4×3) + (0.2×5) + (−2) = 1.2 + 1.0 − 2.0 = 0.2. Since 0.2 ≥ 0, the step activation outputs 1.
A dataset has 5000 samples and the batch size is 100. How many iterations make up one epoch?
Iterations per epoch = total samples ÷ batch size = 5000 ÷ 100 = 50. The weights are updated 50 times in one epoch.
Write a Python function that implements a perceptron with a step activation, given a list of inputs, weights, and a bias.
def perceptron(inputs, weights, bias):
# Step A & B: weighted sum
z = bias
for x, w in zip(inputs, weights):
z += x * w
# Step C: step activation
return 1 if z >= 0 else 0
print(perceptron([3, 5], [0.4, 0.2], -2)) # z = 0.2 -> 1
print(perceptron([1, 1], [0.4, 0.2], -2)) # z = -1.4 -> 0
Modify a neuron to output a probability using the sigmoid activation instead of a hard step. Test with inputs [2, 3], weights [0.5, 0.5], bias 0.
import numpy as np
def neuron(inputs, weights, bias):
z = np.dot(inputs, weights) + bias
return 1 / (1 + np.exp(-z)) # sigmoid activation
prob = neuron([2, 3], [0.5, 0.5], 0)
print("Probability:", round(prob, 4))
print("Class:", 1 if prob >= 0.5 else 0)
z = 0.5·2 + 0.5·3 + 0 = 2.5; σ(2.5) ≈ 0.924, so the predicted class is 1.
In one or two sentences, explain the difference between the forward pass and backpropagation.
The forward pass sends data from inputs through the network to produce a prediction and compute the loss. Backpropagation then goes backward, using the chain rule to calculate how much each weight contributed to that loss (its gradient), so gradient descent can update the weights to reduce error.
What allows a Multi-Layer Perceptron to learn non-linear patterns that a single perceptron cannot?
Hidden layers plus non-linear activations let the network bend decision boundaries. Without non-linear activations, stacked layers collapse into one linear function.
Which activation function is the industry standard for hidden layers?
ReLU = max(0, z) is fast and avoids the vanishing-gradient problem for positive inputs, making it the default for hidden layers.
For the output layer of a 10-class classification problem, you should use:
Softmax converts raw scores into probabilities that sum to 1 across all classes — ideal for multi-class output. Sigmoid is for binary output.
If a deep network used no activation functions (purely linear), what would happen?
A composition of linear functions is still linear. Non-linear activations are what let multiple layers represent complex, curved patterns.
What makes an RNN suitable for sequential data like text?
The hidden state acts as memory — it passes context from earlier steps forward, so word order and history influence the prediction.
Which optimizer is the common default choice, combining momentum with adaptive learning rates?
Adam (Adaptive Moment Estimation) blends momentum with per-parameter adaptive learning rates and is the practical default. ReLU/Softmax are activations, not optimizers.
The vanishing gradient problem in deep networks using sigmoid is best mitigated by:
ReLU keeps a gradient of 1 for positive inputs, so gradients do not shrink toward zero through many layers — unlike sigmoid/tanh which squash values.
Apply the ReLU activation to each value: −3, 0, 2.5, −0.1, 7.
ReLU(z) = max(0, z). Negative values and 0 become 0; positive values pass through unchanged: −3→0, 0→0, 2.5→2.5, −0.1→0, 7→7.
Using Keras, build an MLP for binary classification with input dimension 20, two hidden layers (16 and 8 neurons, ReLU), and a suitable output layer. Compile it.
from keras import models, layers
from keras.layers import Input
model = models.Sequential([
Input(shape=(20,)),
layers.Dense(16, activation='relu'), # hidden 1
layers.Dense(8, activation='relu'), # hidden 2
layers.Dense(1, activation='sigmoid') # binary output
])
model.compile(optimizer='adam',
loss='binary_crossentropy',
metrics=['accuracy'])
model.summary()
Binary output → 1 neuron + sigmoid + binary_crossentropy loss.
Build a Keras model with an Embedding layer, a SimpleRNN layer, and a sigmoid output, for binary sentiment classification.
from keras.models import Sequential
from keras.layers import Embedding, SimpleRNN, Dense
model = Sequential()
model.add(Embedding(input_dim=5000, output_dim=16)) # words -> vectors
model.add(SimpleRNN(32)) # reads the sequence
model.add(Dense(1, activation='sigmoid')) # 0/1 sentiment
model.compile(loss='binary_crossentropy',
optimizer='adam', metrics=['accuracy'])
Embedding turns word indices into dense vectors; SimpleRNN processes them in order; the sigmoid output gives a sentiment probability.
Why is the choice of output-layer activation different for regression, binary classification and multi-class classification?
The output activation must match the kind of answer needed. Regression needs any real number, so the output is linear (no activation). Binary classification needs a single probability in (0,1), so sigmoid. Multi-class needs probabilities across many classes that sum to 1, so softmax.
Why might a basic RNN struggle to remember the start of a very long sentence, and what architectures fix this?
Over long sequences, gradients are multiplied many times and shrink toward zero (vanishing gradient), so the RNN effectively "forgets" early information. LSTM and GRU add gating mechanisms to retain long-range context, and Transformers use self-attention to access any position directly.