How Machines Learn by Bending Space

Chapter 1 ended at a wall. One neuron draws one straight line, and plenty of questions aren’t line-shaped. We got past it by cheating: I placed two neurons by hand, one for each edge of a band.

This chapter answers the two questions that leaves. What is a network actually doing when it solves a problem no line can solve? And how does it work that out on its own?

The short version of the first answer: it bends the space the data lives in, until a straight line is enough. Eight figures below, all of them interactive. Drag things.

Sixty animals, two numbers each

Cats and dogs. For every animal you get two measurements, how big it is and how loud it is, both scored 0 to 10. You want a rule that reads those two numbers and says cat or dog.

The pattern underneath is simple. Cats are all fairly middling. Dogs are the extremes: a chihuahua and a mastiff are nothing alike, and dogs run from silent to relentless. So the cats sit in a clump and the dogs surround them.

I made this data up and shaped it deliberately. It is about the smallest problem there is that a straight line genuinely cannot do.

That first switch is the important one. The table and the plot hold exactly the same information. One is unreadable and the other gives it away instantly.

Two numbers per animal means two axes, so each animal is a point, and sixty rows become a cloud with a shape. That habit is the most useful one in the subject. Data is points in a space, and everything from here is geometry. It scales, too. An animal with thirty measurements is a single point in thirty dimensions. You can’t picture that, but the reasoning still works.

The line has nowhere to go

Press Show me the best line and you get 68.3%. That isn’t an estimate. I checked every angle and every offset, and the best a line manages is 41 animals out of 60. Guessing “dog” every single time already gets 53.3%, so the line has bought us almost nothing.

The reason is easy to see once you look. A line splits the plane in two, and both halves are infinite. The cats are a small island. Any infinite half big enough to hold the island also swallows a large arc of the ring around it.

You could reach for a circle instead. But then you supplied the answer: you looked, saw a ring, and picked the matching formula. The next dataset is two crescents, and the one after is a spiral. We want a method that works out its own shape.

Don’t bend the line. Bend the paper.

Imagine the sixty animals printed on a sheet of rubber. You get one tool, a straight cut right across the sheet. Flat on the table, you’re stuck.

But nobody said you have to cut it flat. Stretch the sheet first. Squash it. Fold it over on itself. Get the cats bunched into one corner, and then make your single straight cut. Now let the rubber spring back. The cut comes back with it, as a closed curve wrapped around the cats.

You never drew a curve. You drew a straight line in a bent space.

That is meant literally. In a classifier the final layer is one neuron doing exactly what Chapter 1 described: weights, a bias, one straight boundary. Everything in front of it exists to deliver the data somewhere that boundary will work.

So: what bending can a layer do, and how does it choose?

What one layer does to a space

Two neurons side by side, both reading the same two inputs, is all a layer is. Write them one under the other and the notation folds into a matrix:

$$ \begin{aligned} z_1 &= w_{11}x_1 + w_{12}x_2 + b_1\\ z_2 &= w_{21}x_1 + w_{22}x_2 + b_2 \end{aligned} \;\;\Longleftrightarrow\;\; \mathbf{z} = W\mathbf{x} + \mathbf{b} $$

Nothing new happened. Row 1 of \(W\) is the first neuron’s weights, row 2 is the second’s. The matrix is bookkeeping.

What is new is the question we ask. Chapter 1 asked what a neuron says about an animal. Now ask where the animal ends up. The layer takes a point at \((x_1, x_2)\) and hands back two numbers, which is another position in another space. A layer is a map from space to space, and it moves every point at once, including the empty parts where there is no data.

That is why the figure draws a grid. The dots show what happens to your animals. The grid shows what happens to the space between them.

The four numbers in \(W\) decide where the two directions go. Big numbers stretch, a negative one flips, two similar rows squash everything onto a line. The biases only slide the result sideways.

On No bend, that is the complete list of what a layer can do: rotate, scale, shear, flip, slide. A lot of freedom, and nowhere near enough.

Two layers with no bend are one layer

The obvious next move is to stack. If one layer reshapes the space, two should reshape it twice as much. They don’t.

$$ W_2\big(W_1\mathbf{x} + \mathbf{b}_1\big) + \mathbf{b}_2 \;=\; \underbrace{\big(W_2W_1\big)}_{\text{one matrix}}\mathbf{x} \;+\; \underbrace{\big(W_2\mathbf{b}_1 + \mathbf{b}_2\big)}_{\text{one bias}} $$

Multiply out the brackets and the two layers have quietly become one. A single matrix and a single bias do the same job. You paid for a second layer and got nothing.

Put plainly: a rotate-and-stretch followed by another rotate-and-stretch is still a rotate-and-stretch. Repeat it a hundred times and the rubber sheet never creases.

Easier to watch than to argue about. Press the button and follow the same sixty animals all the way down through both layers.

Chapter 1 said in one line that layers collapse without a bend. Now you have watched it happen. The activation function is not a finishing touch on a neuron’s output. It is the only reason depth exists.

What a bend buys you: a fold

So a bend is essential. But what does it actually do? “It makes things non-linear” sounds like an answer and isn’t one.

To see it properly, drop to one dimension, where nothing can hide. Same shape as before: one measurement per animal, cats in the middle, dogs at both ends.

On a line, a straight cut is a single threshold. Pick a number, call everything above it a dog. It fails for the same reason as before, because no threshold separates a middle from two ends. The best you can do is 72.7%.

Now give the problem two ReLU neurons. ReLU turns negatives into zero and lets positives through unchanged.

• Neuron A computes \(h_A = \mathrm{ReLU}(\text{size} - a)\): how far above \(a\) you are, and zero if you are below it.
• Neuron B computes \(h_B = \mathrm{ReLU}(b - \text{size})\): how far below \(b\) you are, and zero if you are above it.

Each animal arrives with one number and leaves with two, so it lands somewhere on a plane. Watch where.

That picture is the whole chapter. The number line has been folded twice, once at each edge. The middle section, every cat, is crushed onto a single point at the corner, and the two outer sections are lifted away along different axes.

The crushing is the important half. Below \(a\), neuron A outputs zero for everything, so it has thrown away the difference between one cat and another. For this question that difference never mattered.

And the two groups of dogs started at opposite ends of the line, about as far apart as they could be. After the fold they are both far from the corner but in different directions, so one cut catches both. Their positions relative to each other changed completely, which no amount of rotating or stretching can do.

Many folds, then folds of folds

Back to two dimensions. A ReLU neuron still folds, just along a line instead of at a point. On one side of that line the space passes through untouched, and on the other it is flattened onto zero. One neuron, one crease.

Now count creases against what you need. Two creases can carve out a strip between two lines, which is exactly the band Chapter 1 solved by hand. But a strip runs off to infinity at both ends, and to trap the cats you need an enclosure. Two lines can never enclose anything. Three can.

That is why the last figure in this chapter fails with two neurons and works with three, and why the boundary comes out as a polygon rather than a smooth circle. It can only change direction where it crosses a crease.

More neurons in a layer means more creases. More layers means folding a space that has already been folded, so one crease in layer 2 can show up as several separate pieces back in the original space. That compounding is why deep networks do more than wide shallow ones with the same number of weights.

This is also what people mean by “learning features”. Each layer hands the next one a new space, and the stack keeps moving the data until one straight line can finish the job. In an image network the early layers end up organised by edges, the middle ones by textures and parts, the last by whole objects. Nobody designed that. It falls out of folding in whatever direction lowers the score.

Which brings us to the second half of the chapter. So far, I have set every number by hand.

So what is “learning”?

Be concrete about what there is to learn. The network we will build for the ring has two inputs, four hidden neurons and one output. Count its knobs:

• each hidden neuron has 2 weights and 1 bias, so \(4 \times 3 = 12\)
• the output neuron has 4 weights and 1 bias, so \(5\)

Seventeen numbers, and that is the entire network. It has no memory, no rules, and no idea what an animal is. Learning means finding seventeen numbers that make the answer usually right, and nothing else. The famous models are that same sentence with a bigger number.

Turning seventeen knobs at random is hopeless, so we need three things:

1. a score for how badly we are doing, as one number;
2. a way to know, for each knob, which way to turn it;
3. a rule for how far to turn before checking again.

Those are loss, backpropagation and gradient descent. Everything called “training” is those three in a loop.

Loss: one number for how wrong you are

The obvious score is accuracy, and it is the wrong one, for the same reason the step function was the wrong activation in Chapter 1. Nudge a weight and accuracy usually does not move at all. It sits still, sits still, then jumps when some animal crosses the boundary. A score that mostly ignores you cannot tell you which way to go.

So we score the network’s confidence instead, which does move whenever the weights move. The output neuron ends in a sigmoid, so it gives a number between 0 and 1: its probability that this animal is a cat. If the truth is cat, the penalty is

$$ \text{penalty} = -\log(\hat{y}) $$

Read it as a table, not a formula. Say 0.99 and you pay 0.01, so there is nothing to fix. Say 0.5 and you pay 0.69, because shrugging costs something. Say 0.01 and you pay 4.6. As the answer heads for zero the penalty runs off to infinity.

The truth is either 0 or 1, and rather than write an if-statement we fold both cases into one line by multiplying each by something that is zero when it does not apply:

$$ L \;=\; -\Big[\, \underbrace{t\log\hat{y}}_{\text{alive only when } t=1} \;+\; \underbrace{(1-t)\log(1-\hat{y})}_{\text{alive only when } t=0} \Big] $$

With \(t = 1\) the second term vanishes, and with \(t = 0\) the first does. It is an if-statement in arithmetic clothing, and it is called cross-entropy. Average it over all sixty animals and you have one number for the health of the whole network. That number is what we push downhill.

Gradient descent: always step downhill

Picture the loss as a landscape. Every possible setting of the knobs is a position, and the height there is the loss. Training is walking downhill.

Seventeen knobs makes a seventeen-dimensional landscape, which nobody can draw. So here is the smallest honest version: one measurement (an animal’s weight in kilos), one weight, one bias. Two knobs is exactly as many as a picture will hold.

The orange arrow is the only thing the algorithm knows. It is built from two numbers, the slope of the loss in each direction, written \(\partial L/\partial w\) and \(\partial L/\partial b\). Ignore the notation and read it as a question:

If I nudged \(w\) up by a hair, would the loss go up or down, and how sharply?

That is all a derivative is here. A positive slope means increasing \(w\) makes things worse, so go the other way. Which is the whole update rule:

$$ w \;\leftarrow\; w \;-\; \eta\,\frac{\partial L}{\partial w} $$

The minus sign is “downhill”. The \(\eta\) is the learning rate, meaning how big a stride to take, and the figure shows why it matters. Too small and you will be there all week. Too big and you bound over the valley and land higher than you started.

Two caveats. Real landscapes are bumpy and have millions of dimensions, so there is no guarantee of finding the lowest point, and in practice a good-enough valley is fine. And real training measures the slope on a small random batch rather than every example, which is far cheaper and gives a noisy version of the same arrow. That is stochastic gradient descent.

Backpropagation: who is to blame?

We have a score, and we know a slope is what we need. One problem is left, and it held the field up for years.

The loss is computed from \(\hat{y}\), the final answer, so the slope for the output neuron’s weights is easy enough. But what about \(w_{11}\), buried in the first layer? It never touches the loss. It affects a hidden neuron, which affects the output, which affects the answer, which affects the loss. How can it know which way to move?

First, in words

Think of a kitchen. A dish goes out and comes back too salty. The head chef knows exactly how wrong it was, which is the loss. She did not make it herself, though. She combined two sauces from two cooks, so she splits the blame by how much of each sauce went in. The sauce that made up most of the dish takes most of the blame; the one that contributed a spoonful takes almost none.

Each cook now has a number of their own: how wrong their sauce was, and in which direction. And each of them passes blame back to their own ingredients the same way.

That is backpropagation, and the analogy is unusually tight. “Proportion of the dish” is literally the weight on the wire. Blame flows backwards along the same wires the data came forwards along, splitting at every junction by those same weights. A neuron deep in a network never needs to know what the loss is, or what the network is for. It only needs the number handed back to it.

Then, in symbols

The chain rule makes that exact, and its intuition is arithmetic you already trust. If \(a\) moves \(b\) twice as fast, and \(b\) moves \(c\) three times as fast, then \(a\) moves \(c\) six times as fast. Sensitivities multiply along a path:

$$ \begin{aligned} \frac{\partial L}{\partial w_{11}} \;&=\; \frac{\partial L}{\partial \hat{y}} \cdot \frac{\partial \hat{y}}{\partial o} \cdot \frac{\partial o}{\partial h_1} \cdot \frac{\partial h_1}{\partial z_1} \cdot \frac{\partial z_1}{\partial w_{11}} \\[6pt] &=\; \underbrace{(\hat{y} - t)}_{\text{how wrong}} \;\cdot\; \underbrace{v_1}_{\text{A's say}} \;\cdot\; \underbrace{f'(z_1)}_{\text{A's alertness}} \;\cdot\; \underbrace{x_1}_{\text{A's input}} \end{aligned} $$

Four factors, and each one is a number you already have:

• The first two collapse to just \(\hat{y} - t\). Sigmoid and cross-entropy were built for each other and the messy parts cancel. Predicted 0.14 when the answer was 1? Then it is \(-0.86\), and that one number carries both how wrong you were and which way.
• \(v_1\) is the weight from neuron A to the output. That is the “how much of the dish was your sauce” term.
• \(f'(z_1)\) is the slope of the activation, \(1 - h_1^2\) for tanh and simply 1 or 0 for ReLU. A neuron sitting in its flat region gets a zero here and no blame at all, which is right. It was not listening.
• \(x_1\) is what flowed down that wire. A weight can only be blamed for what passed through it.

So the whole thing is (ŷ − t) × v₁ × f′(z₁) × x₁, four numbers that were already lying around from the forward pass. That is why we keep them.

Two things follow from that.

The first is why backpropagation is fast. The obvious alternative is to wiggle each weight, re-run the network and see what happened, at one full forward pass per weight. A million weights would mean a million passes per step and nobody would have trained anything. Backprop works out the blame for each neuron once and shares it among all the weights feeding it, so the whole backward sweep costs about one forward pass however many weights there are. That is why deep learning is possible at all.

The second is what happens when the chains get long. Every layer multiplies in another \(f'(z)\). A sigmoid’s slope is at most 0.25, so ten sigmoid layers shrink the blame by 0.25 ten times over, which is about one part in a million. The early layers get a signal so faint they stop learning. That is the famous vanishing gradient, and it is the practical reason ReLU took over: on its positive side the slope is exactly 1, so the chain passes through undamped however deep it goes. Chapter 1 said ReLU “let networks get deep”. This is the mechanism.

Watch it bend the space by itself

That is every part. Loss says how bad things are, backprop turns that into a slope for all seventeen knobs, gradient descent nudges them downhill, and the loop runs a few hundred times.

Nobody places a neuron. Nobody mentions rings, circles or folds. Press the button.

Look at what those four dashed lines are. Each one is a single neuron doing the only thing a neuron can do, which is Chapter 1’s straight line. On its own, none of them is a rule about cats. But the four creases together fence off a region, and the output neuron, also just a straight line, adds them up into the closed curve you can see.

And that curve is not really curved. In the space the last neuron sees, the four-dimensional space of what neurons A, B, C and D say, the boundary is perfectly flat. What you are looking at is that flat cut, seen through four folds, springing back into the original space.

The whole thing, in about thirty lines

All of it, with no framework. This is the network from the last figure:

import numpy as np

def relu(z):    return np.maximum(0, z)
def sigmoid(z): return 1 / (1 + np.exp(-z))

X = (animals - 5) / 2.5      # (60, 2) size and loudness, centred on zero
t = labels                   # (60,)   1 = cat, 0 = dog

rng = np.random.default_rng(0)
W = rng.normal(0, 1, (4, 2))   # 4 hidden neurons, 2 inputs each
b = np.zeros(4)
V = rng.normal(0, 1, 4)        # the output neuron's 4 weights
c = 0.0
lr = 0.6

for step in range(600):
    # ---- forward: fold the space, then cut it -------------------
    Z = X @ W.T + b                    # (60, 4) each neuron's total
    H = relu(Z)                        # the bend. this is the fold
    o = H @ V + c                      # (60,)   the straight cut
    y = sigmoid(o)                     # (60,)   P(cat)
    loss = -np.mean(t * np.log(y + 1e-9) + (1 - t) * np.log(1 - y + 1e-9))

    # ---- backward: hand the blame back down the wires -----------
    do = (y - t) / len(X)              # dL/do, the two derivatives that cancelled
    gV = H.T @ do                      # how much each hidden neuron mattered
    gc = do.sum()
    dZ = np.outer(do, V) * (Z > 0)     # share it out, then through the bend
    gW = dZ.T @ X                      # and finally onto each weight
    gb = dZ.sum(axis=0)

    # ---- step downhill ------------------------------------------
    W -= lr * gW;  b -= lr * gb;  V -= lr * gV;  c -= lr * gc

print(loss, np.mean((y > 0.5) == t))    # 0.02  1.0  -> all sixty correct

Two lines are worth pointing at. H = relu(Z) is the fold. Delete it and the whole stack collapses into a single line, exactly as the algebra said, and the accuracy drops to 53%. And dZ = np.outer(do, V) * (Z > 0) is backpropagation in one expression: V shares the blame out by influence, and (Z > 0) is ReLU’s derivative, zeroing the blame for any neuron that was not listening.

About a hundred thousand arithmetic operations, and it finds a fence in the plane without being told that fences exist. PyTorch does the same thing, with the backward pass written for you.

What you now know

• Data is points in a space, and the shape of the cloud is the problem.
• A layer is a map from space to space. It moves every point at once, not just the ones you have data for.
• With no activation function a layer can only rotate, stretch, shear and slide, and any stack of those is a single layer. Depth would be pointless.
• The bend lets a layer fold, crushing regions together and tearing neighbours apart. Enough folds turn an impossible question into a trivial one, and the last neuron finishes it with one straight cut.
• Loss turns “how are we doing” into one number that moves whenever the weights move, which accuracy does not.
• Backpropagation hands blame backwards along the wires, splitting it by weight, for about the cost of one forward pass. Gradient descent then steps every knob against its own slope.

Chapter 3 takes this somewhere it cannot go yet. Sixty animals and two measurements is a toy. A photograph is fifty thousand numbers, and the same approach falls apart in a specific and interesting way. Fixing that is how we get to networks that actually see.