Installing PyTorch or TensorFlow is the easiest and ideal way to start with deep learning, but it comes with a cost. These libraries wrap the underlying math in a black box. If you only ever code with them, you aren't really learning how to build a neural network, you're just learning how to call an API to train a model.
Here, we will build a neural network from scratch and train a model with it. A neural network is eventually is just one small loop, repeated.
NOTE
1. FORWARD - run the input through the network to get a prediction
2. LOSS - measure how wrong the prediction is (one number)
3. BACKWARD - figure out how each weight contributed to that wrongness (gradients)
4. UPDATE - nudge every weight a little in the direction that reduces the loss
5. ZERO - reset the gradients, and repeat
STEP 01: A neuron is embarrassingly simple
A single neuron takes some inputs, multiplies each by a weight, adds them up with a bias, and passes the result through a squashing function:
output = activation(w1*x1 + w2*x2 + ... + wn*xn + b)The weights basically ask one single question, "how much do I care about each input." That's it. The one part worth pausing on is the activation function here. 1 Some common activation functions are tanh relu sigmoid Why is it there?
Because without it, stacking layers of neurons is just pointless. A linear function of a linear function is still just a linear function. Mathematically, two stacked layers with no activation function collapse into one: W2(W1·x) = (W2·W1)·x. You could stack a hundred layers and only ever draw straight lines. The activation adds a bend. That bend is what lets a network learn curved, complicated patterns of real life data. Linear layers stretch and rotate the input; the activation folds. You need folds to make interesting shapes.
A Single Neuron In 1943, neurophysiologist Warren McCulloch and mathematician Walter Pitts wrote a paper on how artificial neurons might work
STEP 02: Autograd is the part you must build yourself
I will be honest, backpropagation was the hardest for me to wrap my head around. But it is not a special algorithm bolted onto neural networks. It's just the chain rule from calculus, applied automatically. And you can build the whole machine in about 60 lines.
The trick is to stop thinking of a number as just a number. Instead, every number remembers how it was computed, what operations and what inputs produced it. If every number carries that history, then once we know how wrong the final answer was, we can walk backwards through the history and hand each number its share of the blame. That "share of the blame" is the gradient.
So for this, I needed a class Value that wraps a number and remembers its parentage, where it came from:
class Value: def __init__(self, data, _children=(), _op=""): self.data = data # the actual number self.grad = 0.0 # its gradient (blame), filled in later self._prev = set(_children) # the Values that produced this one self._op = _op # which operation made it (for debugging) self._backward = lambda: None # HOW to pass blame to the parentsThe magic of this piece of code is that every operation not only computes the result, it also stores a little function (_backward) that knows how to distribute this result's gradient to its inputs. Look at multiplication:
def __mul__(self, other): other = other if isinstance(other, Value) else Value(other) out = Value(self.data * other.data, (self, other), "*") def _backward(): self.grad += other.data * out.grad # d(a*b)/da = b other.grad += self.data * out.grad # d(a*b)/db = a out._backward = _backward return outRead that _backward closure slowly, because it is backprop. If out = a * b, then a small change in a changes out in proportion to b (that's just calculus: d(a·b)/da = b). So a's share of the blame is b * out.grad, its local derivative times the blame that flowed into out from above. That last part is the chain rule: local derivative × downstream gradient. Every operation follows the identical pattern - compute the value, define how to pass the gradient back:
def __add__(self, other): other = other if isinstance(other, Value) else Value(other) out = Value(self.data + other.data, (self, other), "+") def _backward(): self.grad += out.grad # addition just passes the gradient straight through other.grad += out.grad out._backward = _backward return out def tanh(self): t = math.tanh(self.data) out = Value(t, (self,), "tanh") def _backward(): self.grad += (1 - t * t) * out.grad # derivative of tanh is (1 - tanh^2) out._backward = _backward return outNow the finale - backward(), the thing that used to be a black box:
def backward(self): # 1. Sort all Values so every node comes AFTER the ones it depends on. topo, visited = [], set() def build(v): if v not in visited: visited.add(v) for child in v._prev: build(child) topo.append(v) build(self) # 2. The final output's gradient with respect to itself is 1. self.grad = 1.0 # 3. Walk BACKWARDS, letting each node push blame to its parents. for node in reversed(topo): node._backward()The first time I ran this and checked the gradient against a hand-computed derivative and they matched, backprop stopped being magic. It's just the chain rule, run in reverse over a graph.
STEP 03: A network is just neurons stacked
A network with 4 inputs, 2 hidden layers and 4 outputs
Once you have a neuron, a layer is a list of neurons, and a network (MLP = multi-layer perceptron) is a list of layers. Notice these are plain, explicit loops. no clever one-liners, because I wanted to see every step:
class Neuron: def __init__(self, n_inputs, nonlin=True): self.w = [] for _ in range(n_inputs): # small random init, scaled by 1/sqrt(n) so tanh doesn't saturate self.w.append(Value(random.uniform(-1, 1) * n_inputs ** -0.5)) self.b = Value(0.0) self.nonlin = nonlin def __call__(self, x): act = self.b for wi, xi in zip(self.w, x): act = act + wi * xi # weighted sum return act.tanh() if self.nonlin else act # last layer stays linear (raw logits) def parameters(self): return self.w + [self.b]class Layer: def __init__(self, n_inputs, n_neurons, nonlin=True): self.neurons = [] for _ in range(n_neurons): self.neurons.append(Neuron(n_inputs, nonlin)) def __call__(self, x): out = [] for neuron in self.neurons: out.append(neuron(x)) return out def parameters(self): params = [] for neuron in self.neurons: params += neuron.parameters() return paramsclass MLP: def __init__(self, n_inputs, layer_sizes): sizes = [n_inputs] + layer_sizes self.layers = [] for i in range(len(layer_sizes)): is_last = (i == len(layer_sizes) - 1) self.layers.append(Layer(sizes[i], sizes[i + 1], nonlin=not is_last)) def __call__(self, x): for layer in self.layers: x = layer(x) # output of one layer feeds the next return x def parameters(self): params = [] for layer in self.layers: params += layer.parameters() return paramsSTEP 04: The loss - how wrong are we?
For classification, the right loss is cross-entropy. Intuitively it measures surprise:
the model outputs a probability for each class, and the loss is how shocked it is by the correct answer. If it gave the true class 99% probability, surprise is near zero; if it gave it 1%, surprise is huge. It does not care what the model's top guess was, only how much probability it put on the truth.
def cross_entropy_loss(model, Xb, yb): total = Value(0.0) for x, target in zip(Xb, yb): inputs = [Value(i) for i in x] logits = model(inputs) # 10 raw scores # stable softmax: subtract the max logit first so exp() doesn't overflow m = logits[0].data for z in logits: if z.data > m: m = z.data exps = [] for z in logits: exps.append((z - m).exp()) denom = Value(0.0) for e in exps: denom = denom + e p_true = exps[target] / denom # probability assigned to the correct class total = total + (-(p_true.log())) # surprise = -log(p_true) return total * (1.0 / len(Xb))The whole thing boils down to that last line: -log(probability of the correct answer), averaged over the batch. (The subtract the max trick is just numerical hygiene so exp()doesn't blow up to infinity, a small thing that will absolutely ruin your day if you skip it.)
Claude Shannon, whom Claude is named after, introduced the mathematical foundation of cross-entropy in 1948 in his paper on information theory
STEP 05: The optimizer - nudge the weights downhill
Backprop tells us the direction each weight should move to reduce the loss. The optimizer decides how far to move. The simplest is gradient descent; here's Adam, the one everybody actually uses, which adapts the step size per-weight using running averages of the gradient:
class Adam: def __init__(self, params, lr=0.01, beta1=0.9, beta2=0.999, eps=1e-8): self.params = params self.lr, self.beta1, self.beta2, self.eps = lr, beta1, beta2, eps self.t = 0 self.m = [0.0 for _ in params] # running avg of the gradient self.v = [0.0 for _ in params] # running avg of the gradient squared def zero_grad(self): for p in self.params: p.grad = 0.0 def step(self): self.t += 1 for i in range(len(self.params)): p = self.params[i] g = p.grad self.m[i] = self.beta1 * self.m[i] + (1 - self.beta1) * g self.v[i] = self.beta2 * self.v[i] + (1 - self.beta2) * (g * g) m_hat = self.m[i] / (1 - self.beta1 ** self.t) # bias correction v_hat = self.v[i] / (1 - self.beta2 ** self.t) p.data -= self.lr * m_hat / (v_hat ** 0.5 + self.eps)Don't sweat every symbol. The gist: keep a smoothed memory of recent gradients (m), scale the step so consistently-large-gradient directions don't overshoot and tiny-gradient directions don't stall (v), and step. Note zero_grad() , that's step 5 of our loop, resetting the accumulated gradients so the next iteration starts clean. (Forget it, and gradients from every past step pile up and your training detonates. Ask me how I know.)
Putting it together: the training loop
Now the five steps we have seen so far, in code, training on 8×8 images of handwritten digits:
model = MLP(64, [16, 10]) # 64 pixels -> 16 hidden -> 10 digit classesopt = Adam(model.parameters(), lr=0.01)batch_size, n_epochs = 16, 12for epoch in range(n_epochs): order = list(range(len(Xtr))) random.shuffle(order) # shuffle each epoch for start in range(0, len(Xtr), batch_size): batch_idx = order[start:start + batch_size] Xb = np.array(Xtr)[batch_idx] yb = np.array(Ytr)[batch_idx] opt.zero_grad() # 5. clear old gradients loss = cross_entropy_loss(model, Xb, yb) # 1+2. forward + loss loss.backward() # 3. backprop opt.step() # 4. update weightsFour lines in the inner loop. zero_grad, forward+loss, backward, step. That's a neural network training. Run it, and this "thing", built entirely from a Value class I wrote myself climbs to ~93% accuracy on held-out digits. No framework. No magic. Just the chain rule and a loop.
The moment that high accuracy value came up, the wall I'd been stuck at for years was just... gone.