A neuron that learns

We've now got tokens, embeddings, and a way to measure similarity. We have nothing that learns. The bigram model in chapter 1 was trained by counting; that's not learning, that's bookkeeping. To go further we need a function whose parameters get adjusted in response to its mistakes.

A neuron is the smallest possible such function. It takes a vector input, weighs it, sums, adds a bias, squashes through a non-linearity:

That single line is the whole story of feed-forward networks. Stack a bunch in parallel and you get a layer; stack layers and you get a deep network. But the neuron itself is what learns. Let's build one, watch it work, then save the update rule locally.

1. The forward pass

Write the function that takes inputs x = [x_0, x_1], weights w = [w_0, w_1], bias b, and returns the neuron's output. The script defines sigmoid first, then uses it inside the forward pass.

The bars show the neuron's output for several test inputs. With w = [1, 0.5] and b = 0, points where w · x + b is positive should produce outputs > 0.5 (warm), and negative should be < 0.5 (cool). The neuron is just a weighted sum followed by a squash; everything else in this chapter is mechanics for adjusting w and b automatically.

2. The loss function

Given the neuron's output p and the true label y (0 or 1), we need a number that says "how bad was that prediction?". The standard choice is binary cross-entropy:

It's near zero when p matches y, and grows large in the other direction. The asymmetry — log(p) for the y=1 case, log(1-p) for y=0 — makes the gradient clean later.

Add a small epsilon (~1e-12) before taking logs to avoid log(0) when the model is overconfident.

The bars show the loss across six (prediction, true label) combinations. When prediction matches label, loss is small (green). When they disagree — especially when the model is confident in the wrong answer — loss is large (red).

3. The gradient

Now the magic. The composition of sigmoid + binary cross-entropy has a beautiful property: the gradient of the loss with respect to (w · x + b) is just p − y. That is, the error itself is the gradient. From there:

Write the gradient function. The chapter compares your output to a reference computation on a known test case.

If the comparison shows ✓, your gradient agrees with the closed form. This is the lemma that makes logistic regression and neural networks practical: there's no need to differentiate the sigmoid by hand — the algebraic shortcut takes care of it.

4. Train the neuron

Now we put it together. Use the dataset of two clusters (~60 points in 2D), iterate gradient descent, and watch the decision boundary fit.

The scatter on the left shows the data; the orange line is the decision boundary w · x + b = 0, where the neuron's output crosses 0.5. After training, the line should slice between the two clusters — a piece of geometry the neuron figured out by repeatedly adjusting w and b to lower the loss.

The line on the right is the loss curve over training iterations. It should slope down monotonically, leveling off as the neuron approaches the best linear fit it can produce.

5. Add the first learning step locally

Create llm/nn.py with the simplest trainable unit:

"""Small neural-network pieces before the PyTorch rewrite."""
from __future__ import annotations
 
import math
 
 
# [1]
def sigmoid(x: float) -> float:
    return 1.0 / (1.0 + math.exp(-x))
 
 
# [2]
def neuron(x: list[float], w: list[float], b: float) -> float:
    return sigmoid(sum(xi * wi for xi, wi in zip(x, w)) + b)
 
 
# [3]
def bce(pred: float, target: float) -> float:
    eps = 1e-12
    pred = min(max(pred, eps), 1.0 - eps)
    return -(target * math.log(pred) + (1.0 - target) * math.log(1.0 - pred))
 
 
def neuron_step(
    x: list[float],
    target: float,
    w: list[float],
    b: float,
    lr: float,
) -> tuple[list[float], float, float]:
    pred = neuron(x, w, b)
    # [4]
    error = pred - target
    # [5]
    next_w = [wi - lr * error * xi for wi, xi in zip(w, x)]
    next_b = b - lr * error
    return next_w, next_b, bce(pred, target)

Read the step as a tiny training loop:

• [1] sigmoid squashes any score into the 0-1 range.
• [2] neuron computes weighted sum + bias, then turns it into a probability-like output.
• [3] bce is the loss: confident wrong answers are punished hard.
• [4] error = pred - target is the key simplification. If the target is 1 and the prediction is too low, error is negative. If the target is 0 and the prediction is too high, error is positive.
• [5] updates weights and bias against that error. Each weight update is scaled by xi, so active input dimensions get credit or blame.

This is the first file in your project that updates parameters because of a mistake. That idea scales all the way to chapter 13.

What the neuron can't do

Run the same training procedure on data that isn't linearly separable — say, points arranged in a circle with the inner ring being class 0 and the outer ring class 1 — and the neuron will fail. No line through the input space can carve out a circular region. The neuron is fundamentally a linear classifier with a sigmoid on the output.

That's the limitation we attack in chapter 6: one neuron makes one boundary line; a layer of neurons makes many boundary lines and combines them into something flexible enough to handle XOR, spirals, and other non-linear datasets. Same training algorithm; the model gets bigger.

Going further

• Appendix · Backprop by hand — derives the gradients for this same kind of network step by step, then verifies the result against PyTorch's loss.backward(). Optional but recommended if "where do the gradients come from?" is still fuzzy.
• Andrej Karpathy's "Building micrograd" — derives the same gradient by hand and builds a tiny autograd engine that handles the chain rule for arbitrary expressions. We'll use one of those in chapter 6.
• 3Blue1Brown's neural networks series — visual derivation of gradient descent over a network.
• The full reference implementations live in lib/ml/nn/{linear,activations,loss}.ts.

Next up: stacking layers — one neuron is one line. Multiple neurons in a layer are multiple lines. Combine them and you can carve out anything.