Neural Networks Explained

One neuron

A neural network is built from a very small idea repeated at scale. The unit, loosely inspired by biological neurons, does one thing: it takes several numbers in, combines them, and passes one number out.

The combination is a weighted sum. Each input gets multiplied by a weight, the results are added up, and a bias value is added on top:

```

z = w1x1 + w2x2 + ... + wn*xn + b

```

Then an activation function transforms that sum into the neuron's output. Without this step, the whole network would just be one big linear formula no matter how many neurons you stacked, and linear formulas can only draw straight dividing lines through data. The activation function bends the math, which is what lets networks learn curves, boundaries, and genuinely complicated patterns.

A neuron in code is barely anything:

```python

import numpy as np

def neuron(x, w, b):

z = np.dot(w, x) + b

return max(0, z) # ReLU activation: negative -> 0, positive -> itself

x = np.array([0.5, -0.2, 0.8])

w = np.array([0.4, 0.9, -0.3])

print(neuron(x, w, 0.1)) # one number out

```

The weights and bias are the knobs. Learning means adjusting them until the neuron's output is useful.

Layers

One neuron can only do so much. Networks arrange neurons in layers. The input layer takes the raw data: pixel brightness values for an image, word vectors for text. Each hidden layer transforms what the previous layer produced. The output layer produces the final answer: a category, a number, a probability.

Information flows in one direction during a prediction. Each layer computes its weighted sums, applies its activation function, and hands the results to the next layer. This is called the forward pass. A network with two hidden layers of a hundred neurons each is doing tens of thousands of these tiny computations per prediction, all of them just multiply, add, and clip.

Depth is where the interesting behavior comes from. Early layers tend to pick up simple features. In image networks, the first layer responds to edges and color gradients. The next layer combines those into textures and simple shapes. Deeper layers assemble parts: an eye, a wheel, a window. Nobody programs this hierarchy. It falls out of training because each layer builds on the features the layer below found useful.

Activation functions

The choice of activation function shapes how a network trains. A few have survived decades of use.

The sigmoid squashes any input into the range 0 to 1. It was the default in early networks and still shows up at the output when you want a probability for a yes-or-no question. Its weakness is that for large positive or negative inputs, its slope flattens to nearly zero, and tiny slopes mean tiny learning signals.

Tanh is similar but outputs between -1 and 1, centered on zero, which trains a little better in practice.

ReLU, the rectified linear unit, outputs the input directly when it is positive and zero otherwise. It is almost embarrassingly simple, and it became the standard for hidden layers because it is cheap to compute and does not flatten out for large positive values. Its flaw is the "dead neuron" problem: once a neuron's weights push it to always output zero, no signal flows through it and it stops learning.

Softmax is different. It takes a whole layer's outputs and turns them into numbers that sum to one, like probabilities. It sits at the end of classifiers that pick among several categories.

How they learn

A network starts with random weights and makes random guesses. Learning is the process of turning those guesses into good ones, and it runs in a loop with three steps.

First, the forward pass runs training examples through the network and produces predictions. Second, a loss function measures how far the predictions were from the right answers, as a single number. Third, backpropagation figures out how much each weight contributed to the error and nudges every weight a little in the direction that would have reduced it.

Backpropagation is just the chain rule from calculus applied systematically. If the loss depends on a layer's output, and that output depends on the layer's weights, then the effect of a weight on the loss is the product of those dependencies, chained layer by layer from the output back to the input. Doing this naively would be impossibly expensive. The insight is that each layer's computation can reuse the results from the layer after it, so the whole backward sweep costs about as much as the forward pass.

This algorithm was popularized in 1986 by Rumelhart, Hinton, and Williams, and it is what made training multi-layer networks practical. The update applied to each weight after backpropagation is gradient descent: step against the gradient, scaled by a learning rate.

A short history

The story starts in 1943 with the McCulloch-Pitts neuron, a mathematical model of a brain cell built from logic gates. In 1958, Frank Rosenblatt built the perceptron, the first trainable version: a single layer of adjustable weights that learned to classify inputs from examples.

Then came the setback. In 1969, Minsky and Papert showed that a single-layer perceptron could not learn simple patterns like XOR, where the answer depends on a combination of inputs rather than any one of them. Funding dried up and neural network research went quiet for years.

The 1986 backpropagation paper reopened the door by showing how to train networks with hidden layers, which have no trouble with XOR. Progress was slow through the 1990s and 2000s, held back by small datasets and weak hardware. The dam broke in 2012 when a deep network called AlexNet won an image recognition contest by a wide margin, and the modern era began.

One mathematical result is worth knowing. The universal approximation theorem, proved around 1989, says that a network with a single hidden layer can approximate any continuous function if the layer is wide enough. This guarantees that neural networks are expressive enough for the job. It says nothing about whether training will actually find the right weights, which is the hard part in practice.

What they are good at, and what they are not

Neural networks excel wherever there is a mountain of examples and the pattern is too tangled to write down: recognizing speech, translating languages, detecting fraud, predicting protein shapes. They are poor at tasks with little data, at reasoning that needs explicit steps, and at telling you why they decided what they decided. A network's knowledge lives in millions of numbers with no labels, which makes it powerful and opaque at the same time.