The Artificial Neuron

Formally, an artificial neuron is a function $f: \mathbb{R}^n \to \mathbb{R}$ parameterized by a weight vector $\mathbf{w} = (w_1, \dots, w_n) \in \mathbb{R}^n$ and a scalar bias $b \in \mathbb{R}$. Given an input vector $\mathbf{x} = (x_1, \dots, x_n)$, the neuron first computes a pre-activation (also called the logit or net input):

This is an affine transformation of the input: a linear combination of the $x_i$, shifted by a constant. The bias $b$ can be folded into the weighted sum by appending a constant input $x_0 = 1$ with weight $w_0 = b$, giving the compact form $z = \sum_{i=0}^n w_i x_i$. This trick is used constantly in implementations because it lets you treat the bias as "just another weight." The neuron then applies an activation function $\phi: \mathbb{R} \to \mathbb{R}$ to the pre-activation to produce its output, or activation:

The activation function is what makes the neuron nonlinear — without it, $a$ would just be an affine function of $\mathbf{x}$, and composing many such neurons in layers would still only ever produce an affine function overall (an affine function of an affine function is affine). Common choices — the step function, sigmoid, tanh, ReLU — are the subject of Activation Functions; for now, treat $\phi$ as an arbitrary fixed nonlinear function chosen by the network designer, not learned. The weights $w_i$ encode how much each input feature matters and in which direction (positive weight → excitatory-like influence, negative weight → inhibitory-like influence), loosely mirroring the biological synapse strengths discussed in the biological-inspiration background. The bias $b$ shifts the threshold at which the neuron "activates," letting the neuron fire even when all inputs are zero, or preventing it from firing until the weighted evidence crosses some non-zero threshold. Crucially, $\mathbf{w}$ and $b$ are the neuron's only learnable parameters — everything else ($n$, the choice of $\phi$) is fixed architecture.

Computing a single neuron's output, given its parameters and an input, is a direct, deterministic procedure — there is no search or iteration involved for a single forward pass. Training the weights (adjusting $\mathbf{w}, b$ so the neuron's outputs match desired targets) is a separate concern belonging to The Perceptron and later gradient-based learning articles; here we focus purely on the forward computation and its structure.

Step-by-step mechanism: 1. Receive an input vector $\mathbf{x} \in \mathbb{R}^n$. 2. Multiply each input $x_i$ by its corresponding weight $w_i$. 3. Sum all the weighted inputs. 4. Add the bias $b$ to get the pre-activation $z$. 5. Apply the activation function $\phi$ to $z$ to get the output $a$. 6. Return $a$ (a single scalar) — this becomes an input to the next neuron(s), or the network's final prediction. ```text FUNCTION NEURON_FORWARD(x[1..n], w[1..n], b, phi):

# x: input vector, w: weight vector, b: bias scalar, phi: activation function

# Precondition: len(x) == len(w) == n, n >= 1 z <- b # start accumulator with bias FOR i FROM 1 TO n: z <- z + w[i] * x[i] # accumulate weighted sum a <- phi(z) # apply nonlinearity RETURN a # scalar output FUNCTION LAYER_FORWARD(x[1..n], W[1..m][1..n], b[1..m], phi):

# A layer of m neurons sharing the same input x, each with its own (w_j, b_j) outputs[1..m] FOR j FROM 1 TO m: outputs[j] <- NEURON_FORWARD(x, W[j], b[j], phi) RETURN outputs ``` Two implementation details matter in practice. First, `NEURON_FORWARD` is exactly a dot product plus a scalar bias, so real implementations replace the explicit loop with a vectorized dot product (`z = dot(w, x) + b`), which is both faster and numerically cleaner. Second, `LAYER_FORWARD` shows that a layer is just many neurons evaluated on the same input in parallel, each with independent parameters — stacking this operation, with the output of one layer feeding the input of the next, is precisely how a multi-layer network is built; a layer's computation is a matrix-vector product $\mathbf{z} = W\mathbf{x} + \mathbf{b}$ followed by an elementwise $\phi$.

Time complexity. Computing $z = \mathbf{w}^\top \mathbf{x} + b$ requires $n$ multiplications and $n$ additions (or $n{-}1$ additions plus one for the bias), so the weighted sum is $\Theta(n)$ time. Applying $\phi$ to the scalar $z$ costs $O(1)$ (constant time, assuming $\phi$ is a fixed elementary function like sigmoid or ReLU, each evaluated in $O(1)$ arithmetic/transcendental operations). Total: $\Theta(n)$ time for one neuron's forward pass, which is optimal — you provably cannot do better than $\Omega(n)$, because every one of the $n$ inputs must be read and incorporated at least once for the output to depend on it correctly. For a full layer of $m$ neurons sharing an $n$-dimensional input, the cost is $\Theta(mn)$ — a matrix-vector multiply $W\mathbf{x}$ where $W \in \mathbb{R}^{m\times n}$ — which is also optimal for the general dense case, since the output layer has $mn$ independent weight entries, each of which must be read.

Space complexity. A single neuron needs $\Theta(n)$ space to store its weight vector plus $O(1)$ for the bias and the scalar intermediate $z$, so $\Theta(n)$ total. A layer of $m$ such neurons needs $\Theta(mn)$ space for the weight matrix $W$ plus $\Theta(m)$ for the bias vector, dominated by $\Theta(mn)$.

Correctness argument. "Correctness" for a single neuron's forward pass is not about an emergent property like convergence — it is about verifying the computation matches its mathematical specification exactly. This follows directly from the algorithm's structure:

• Invariant of the accumulation loop: after processing the $i$-th input in `NEURON_FORWARD`, the accumulator `z` holds exactly $b + \sum_{k=1}^{i} w_k x_k$. This is true trivially before the loop starts ($z = b$, i.e., the sum over zero terms), and each iteration preserves it by adding exactly one more term $w_i x_i$. After the loop terminates (at $i = n$), the invariant gives $z = b + \sum_{k=1}^n w_k x_k$, which is precisely the definition of the pre-activation.
• Final step: applying $\phi$ to this value produces $a = \phi(z)$, matching the definition of the neuron's output exactly. Hence the algorithm computes the specified function, not an approximation of it.

The deeper correctness question — why nonlinearity is necessary — is a separate and important claim. Suppose you compose two neurons without any nonlinearity ($\phi(z) = z$, the identity). Layer 1 computes $\mathbf{h} = W_1\mathbf{x} + \mathbf{b}_1$; layer 2 computes $\mathbf{y} = W_2\mathbf{h} + \mathbf{b}_2$. Substituting:

which has the same form as a single affine layer with weight matrix $W_2 W_1$ and bias $W_2\mathbf{b}_1 + \mathbf{b}_2$. By induction, any number of stacked linear neurons collapses algebraically to one linear layer — depth buys nothing in expressive power without nonlinear activations. This is precisely why $\phi$ must be nonlinear for multi-layer networks to represent functions beyond affine maps, and it is the mathematical justification for the entire idea of "deep" learning building on this atomic unit.

Tradeoffs, honestly. The neuron model is a drastic simplification: it ignores timing (spikes reduced to one static number), ignores the geometry and history-dependence of biological dendrites, and treats all inputs as instantaneously and independently combinable. This is a feature for tractability (differentiable, parallelizable, easy to reason about) but a limitation if you want to model temporal dynamics — which is why recurrent and spiking-neuron variants exist as separate, more specialized models built later on top of this foundation.

Consider a neuron with $n = 3$ inputs, meant to loosely score "is this a risky loan applicant?" using three standardized features: $x_1$ = debt-to-income ratio, $x_2$ = late payments in the last year, $x_3$ = years of credit history. Suppose the (already learned, or hand-set for this example) parameters are:

Take a specific applicant with $\mathbf{x} = (0.6,\; 3,\; 2)$ (a moderate debt ratio, 3 late payments, 2 years of history).

Tracing the pseudocode directly: `NEURON_FORWARD` initializes `z ← b = -1.0`. After `i=1`: `z ← -1.0 + 2.0·0.6 = 0.20`. After `i=2`: `z ← 0.20 + 1.5·3 = 4.70`. After `i=3`: `z ← 4.70 + (-0.8)·2 = 3.10`. This matches step 5 above exactly (order of accumulation doesn't matter for a finite sum of real numbers, confirming the loop invariant). Then `a ← sigmoid(3.10) ≈ 0.9569`. The output $a \approx 0.957$, close to 1, signals "high risk" under this (illustrative, not real-world-calibrated) neuron. Notice the mechanics that make weights meaningful: $w_3 = -0.8$ is negative, so more years of credit history pushes z down (toward "low risk"), while $w_2 = 1.5 > 0$ means late payments push strongly toward "high risk" — exactly the signed, weighted-influence behavior the model is designed to express. Changing a single input, say $x_2 = 0$ late payments instead of 3, changes $z$ to $3.10 - 4.50 = -1.40$, giving $a = \text{sigmoid}(-1.40) \approx 0.1978$ — a large qualitative swing, illustrating how sensitive the sigmoid output is near the middle of its range and how flat it becomes at the extremes (a fact that matters greatly once you study Activation Functions and vanishing gradients).

The artificial neuron traces to the 1943 collaboration between neurophysiologist Warren McCulloch and logician Walter Pitts, who proposed the first mathematical model of a neuron as a binary threshold unit, showing that networks of such units could compute logical functions. Their model lacked learning — the weights were fixed by hand — but it established that a neuron could be treated as a formal computational device rather than a biological one, a conceptual leap that made everything in this article possible.

Every layer of a modern deep neural network — the fully connected layers in a multilayer perceptron, the filters inside a convolutional layer, the query/key/value projections inside a transformer's attention block — is fundamentally a bank of artificial neurons computing weighted sums plus bias, followed by an elementwise nonlinearity. Frameworks like PyTorch and TensorFlow implement layers such as `Linear`/`Dense` as exactly the matrix form $W\mathbf{x}+\mathbf{b}$ described here, and GPUs are architected with matrix-multiply units precisely because this operation — repeated at massive scale — dominates the computational cost of training and inference in systems from image classifiers to large language models. Understanding the single neuron's forward computation and why nonlinearity is essential is the prerequisite for understanding backpropagation, universal approximation theorems, and the design tradeoffs behind every activation function you will encounter later, including in The Perceptron and Activation Functions.

Ask the tutor: How does the choice of activation function $\phi$ change what a single neuron — or a layer of them — can represent?