What Is a Neuron?

Every neural network you’ve heard of is built out of one small part, copied over and over. The same part sits inside the model that finishes your sentences and the one that spots a tumour in a scan. It’s simpler than you’d expect. It multiplies a few numbers, adds them up, and checks whether the total clears a bar. That’s the whole thing.

This chapter builds that part from scratch. You don’t need any machine learning, and you don’t need calculus. You do need to play with it. There are four figures below and all of them move, so drag the sliders and see what happens. You’ll pick it up much faster that way than by reading about it.

One small, concrete job

It’s much easier to follow this with a real example, so we’ll use the same one the whole way through. You have data on a group of students. For each student you know two numbers: how many hours they studied, and how many hours they slept the night before the exam. You also know whether they passed.

You want a rule. Give it those two numbers for a new student, and it should answer pass or fail.

That’s it. Two numbers in, one yes or no out. Swap the two numbers for the pixels of a photo and the answer for “is this a cat”, and it’s the same job. The maths doesn’t care what the numbers mean.

Two obvious ideas that don’t work

Before the real answer, it helps to watch two simple rules fail. Each failure tells us something the neuron needs.

Attempt 1: just add the two numbers

Try studied + slept > 12 and it breaks straight away. It treats an hour of studying and an hour of sleeping as worth the same. They aren’t. Studying probably counts for more. Sleep counts, but less. And if you added a third input like hours spent on your phone, more of it should make passing less likely, not more.

So plain adding is too blunt. We need a way to say how much each input counts, including counting against.

Attempt 2: weights on their own

Closer, but there’s still a hole. The rule now says pass when w₁·x₁ + w₂·x₂ > 0. Think about a student who studied 0 hours and slept 0 hours. The total is 0 whatever the weights are. That student sits exactly on the fence forever, and no amount of tuning will move them.

The bigger issue is that we can’t make the rule strict or lenient. We can change how much each input counts, but not where the bar sits. Passing ought to need a decent total, not just anything above zero.

Turn the knobs yourself

Put both fixes together and you have an artificial neuron. Here it is, wired up and running. The two circles on the left are the inputs. Each wire’s thickness is its weight, so a thick wire matters a lot and a thin one barely registers. When a wire turns orange the weight has gone negative, and that input now counts against passing.

Two things to notice.

First, there are two kinds of number here, and mixing them up is the most common early confusion. The inputs \(x_1, x_2\) are the data. They change with every student and you don’t choose them. The weights \(w_1, w_2\) and the bias \(b\) are the neuron’s settings. They stay the same for every student, and they’re the only things you can adjust. Learning means adjusting those three numbers, nothing else.

Second, the work happens in two steps: add everything up, then squash the total into an answer. Written out, that’s the full definition:

$$ z \;=\; \underbrace{w_1 x_1 + w_2 x_2 + \dots + w_n x_n}_{\text{each input, scaled by how much it counts}} \;+\; \underbrace{b}_{\text{the bar}} $$

$$ y \;=\; f(z) $$

or in the form you’ll see in textbooks:

$$ y = f\!\left(\sum_{i=1}^{n} w_i x_i + b\right) $$

The \(\sum\) just means “add them all up”, the same as the figure. \(f\) is the squashing step, which gets its own section shortly. If you can read that formula and picture the sliders you were dragging, you’ve got the neuron. Everything after this follows from it.

Those three numbers draw a line

This is the part that made it click for me.

The neuron says pass when \(w_1 x_1 + w_2 x_2 + b > 0\), and fail when it’s negative. Right on the fence, where it can’t decide, \(w_1 x_1 + w_2 x_2 + b = 0\). If you ever met \(y = mx + c\) at school, that’s the same thing written differently. It’s the equation of a straight line.

So let’s plot the students. Each mark is one of 46 students, with hours studied along the bottom and hours slept up the side. Green circles passed, orange triangles failed. The black line is the neuron’s decision boundary, and the shading shows which answer it gives where.

That button is worth a closer look, because it’s all that machine learning really is.

The loss is one number that says how badly the neuron is doing right now. Low is good, and zero is perfect. Training keeps asking the same narrow question: if I made \(w_1\) a bit bigger, would the loss go up or down? Then it steps the other way. Same for \(w_2\), same for \(b\). Do that a few hundred times and the line finds its own place.

Nobody told it where the line should go. All it ever knew was whether it was getting warmer or colder. We’ll pull that apart properly in a later chapter. For now the point is that learning is just small repeated adjustments to the weights and the bias.

Why the activation function matters

We skipped over \(f\), the second step. It’s time to look at it, because the choice matters more than it seems.

After the adding up, the neuron is holding a raw number \(z\). It could be 0.3, or −47. That’s awkward, since we wanted a yes or a no, and “negative forty-seven” is neither. So we push \(z\) through a function that turns it into something we can use.

The obvious choice is a step: positive becomes 1, negative becomes 0. That’s exactly the fence we described, and it’s what the first artificial neuron used back in 1943. It also turns out to be a bad idea, and you can see why from its shape.

The problem with the step is short. If the output doesn’t move when you nudge a weight, training has nothing to steer by. Training only ever asks “warmer or colder?”. With a step function the answer is “exactly the same”, right up until it flips. There’s no signal left.

The sigmoid fixes that by rounding the corner off. A small nudge to a weight now makes a small change to the output, so there’s always a direction to move in. Its output also sits between 0 and 1, so you can read it as confidence. 0.92 means fairly sure they passed. 0.51 means no idea. That’s what the How sure toggle showed on the previous figure: the same line, with the certainty fading out as you get close to it.

Inside big networks today the usual choice is ReLU, which is about as simple as it gets. Negatives become zero, and positives pass through unchanged. It’s one comparison, and it never flattens out on the positive side, which is what let networks get deep without the training signal dying out. Sigmoid still turns up at the end of a network, where you want a probability.

Where one neuron gives up

One neuron, one straight line. That’s a real limit, and it’s easier to feel than to argue about.

Same students, same two measurements, but the truth underneath is different this time. In this group the ones who passed studied a sensible amount. Barely studying fails, which you’d expect. Studying obsessively also fails, because of burnout, no sleep, and panic in the exam hall. Passing sits in a band in the middle.

Your job is to separate green from orange with one line. Have a real go before you read on.

Look at what solved it. Neuron A learned a partial opinion that’s useless on its own: “they studied at least a bit.” Neuron B learned another one: “they didn’t overdo it.” Neither is a rule for passing. But something that fires only when both of them fire is exactly the band we wanted.

That’s the whole trick of deep learning, and you’ve just watched it at the smallest possible scale. You can’t make one neuron smarter than a straight line. What you can do is stack neurons in layers, let each one form a rough partial opinion, and let the next layer combine those opinions into something no single line could say.

Ten lines, no libraries

None of this needs a framework. Here’s the neuron from the first figure, written out plainly:

import numpy as np

def neuron(x, w, b):
    z = np.dot(w, x) + b            # step 1: add it all up
    return 1 / (1 + np.exp(-z))     # step 2: squash it to 0-1 (sigmoid)

x = np.array([6.0, 7.0])            # studied 6 h, slept 7 h   <- the data
w = np.array([1.0, 0.6])            # studying counts more     <- the settings
b = -9.0                            # the bar to clear         <- the settings

print(neuron(x, w, b))              # 0.5498...  -> just about passes

And the two-neuron network that cracked the band. A layer is just the weight vectors stacked into a matrix, so a single @ runs every neuron at once:

W = np.array([[ 1.0, 0.0],          # neuron A: "studied enough?"
              [-1.0, 0.0]])         # neuron B: "not too much?"
b = np.array([-2.6, 7.4])

def layer(x):
    return (W @ x + b > 0).astype(float)   # both neurons at once

def network(x):
    h = layer(x)                    # -> [A fires?, B fires?]
    return float(h[0] and h[1])     # output fires only if both do

print(network(np.array([5.0, 6.2])))   # 1.0  -> pass
print(network(np.array([9.0, 6.2])))   # 0.0  -> studied too much

That W @ x + b is the line every deep learning library is running underneath, a few hundred times over, on much bigger matrices. There’s no extra ingredient hidden inside PyTorch.

What you now know

• A neuron multiplies each input by a weight, adds a bias, and squashes the total. Two steps, and five numbers in the first figure.
• Weights say how much each input counts, and negative weights count against. The bias sets how hard the neuron is to convince.
• On a graph, one neuron is a straight line. Weights tilt it, the bias slides it.
• Learning is just nudging those weights and that bias over and over, in whichever direction makes the loss smaller.
• The activation function has to bend. Without a bend there’s nothing to learn from, and layers collapse into a single line.
• One neuron can never be more than a line, so we use several and combine them. That’s a neural network.

In Chapter 2 we stop placing those two neurons by hand and let a layer find its own partial opinions. That’s where backpropagation comes in.