Rakibul HaqueOnto the next

A GAN from scratch, small enough to watch

Most GAN tutorials hand you a face generator. You run it, images appear, and you learn almost nothing about the mechanism, because everything interesting is buried under a convolutional stack and 40 minutes of GPU time.

So I built the smallest GAN I could still call a GAN, in plain JavaScript with no libraries, and put it in the browser where you can watch it train. Two multilayer perceptrons, hand-written backprop, Adam written out by hand. The whole thing fits in a screen of code and converges in seconds.

The target distribution is a ring: sample an angle uniformly, set the radius to 1.8 plus Gaussian noise with standard deviation 0.1. That sounds trivially easy, and fitting it is easy. The point is that a ring has structure a single blob cannot fake. If the generator quietly gives up and covers only one arc, you see it immediately. That failure has a name, mode collapse, and in 2-D it is impossible to hide.

The two networks

G:  z (2-D noise) -> 16 tanh -> 16 tanh -> 2 linear     (a point on the plane)
D:  x (2-D point) -> 16 lrelu -> 16 lrelu -> 1 sigmoid  (probability x is real)

The generator never sees real data. It only ever receives gradients that arrive through the discriminator, which is the part people find strange at first: G improves purely by exploiting what D has learned about the real distribution.

The discriminator minimizes the usual binary cross entropy on a batch of real and fake points:

L_D = -log D(x_real) - log(1 - D(G(z)))

The generator does not minimize the negation of that. It maximizes the probability that D calls its samples real:

L_G = -log D(G(z))

The detail that makes it train at all

That last choice is the non-saturating loss from the original GAN paper, and it is the difference between training and staring at a dead animation.

The textbook minimax objective asks G to minimize log(1 - D(G(z))). Early in training, D easily spots the fakes, so D(G(z)) sits near zero. Look at the gradient of log(1 - D) there: it is nearly flat. The generator is at its worst and receives almost no signal, which is exactly backwards. Flipping to -log D(G(z)) puts the steep part of the curve where the generator is weakest. Same fixed point, vastly better gradients.

Two smaller choices matter more than they look:

Leaky ReLU in the discriminator, with slope 0.2 on the negative side. A dead ReLU in D is worse than a dead unit in an ordinary classifier, because D is the only path through which G receives information. If a unit outputs zero and has zero gradient, it stops relaying anything, and G goes partially blind.

Adam with beta1 = 0.5 instead of the usual 0.9. The DCGAN convention, and the reason is that the objective is non-stationary: D’s loss surface is being reshaped by G at every step, and vice versa. Heavy momentum means charging downhill on a hill that has already moved. Halving beta1 shortens the memory.

What convergence looks like, and why it is confusing

Here is the part that trips people up. In normal supervised learning, loss going down is good and loss near zero is great. In a GAN, a discriminator loss near zero means D has won, G is producing garbage that is trivially detectable, and the gradients feeding G have collapsed. Training is over in the worst way.

What you actually want is both losses hovering near log 2, about 0.693. That is the value you get when D outputs 0.5 everywhere, meaning it genuinely cannot tell real from fake and is reduced to guessing. Equilibrium, not victory.

I ran the same code offline for 4000 steps to check it behaves. Measuring the generated samples against the target radius of 1.8:

step   500    radius 1.75 ± 0.42    coverage 12/12
step  1500    radius 1.84 ± 0.31    coverage 12/12
step  2500    radius 1.82 ± 0.19    coverage 12/12
step  4000    radius 1.93 ± 0.18    coverage 12/12

Both losses sat near 0.69 the whole way, exactly where they should. The mean radius locks on early, and the interesting quantity is the spread: ± 0.42 shrinking to ± 0.18 is the cloud tightening from a fuzzy annulus onto the actual ring.

Coverage is my cheap mode-collapse detector: split the circle into 12 angular bins and count how many contain a reasonable share of samples. 12/12 means the generator is using the whole ring. If it had collapsed onto one arc, that number would drop and stay dropped, no matter how good the loss looked.

Things worth trying in the demo

The live version has a learning rate slider and a toggle that paints the discriminator’s decision surface behind the points, so you can see what D currently believes.

  • Turn the field on and watch it early. D carves a clean boundary around the ring almost immediately. As G catches up, that structure dissolves into noise, because D can no longer separate the two distributions. The field going flat is the visual signature of the losses reaching 0.69.
  • Push the learning rate up. The equilibrium stops being a resting point and becomes a cycle. The cloud orbits the ring, overshoots, comes back. Adversarial training is not descending into a minimum, it is two players chasing each other, and it only looks like convergence when neither can gain.
  • Reset a few times. Same architecture, same hyperparameters, different random seed, visibly different paths to the same place. Worth internalizing before you trust a single GAN training curve.

The thing I did not expect from building this: how much of GAN practice is not about the networks at all. The architecture here is three lines. Almost every decision that determines whether it works, the non-saturating loss, the leaky activations, the lowered momentum, exists to keep a gradient pathway alive between two models fighting each other. That is the actual engineering problem, and a 2-D toy shows it more clearly than a face generator ever did.

← All writing