Logistic Regression and the Sigmoid

Logistic regression is a discriminative probabilistic model for binary classification: given input $x \in \mathbb{R}^d$ and label $y \in \{0, 1\}$, it directly models the conditional distribution $P(y \mid x)$ rather than the joint distribution of $x$ and $y$. The model computes a linear combination of features, called the score or logit,

and then applies the sigmoid (logistic) function

to obtain a probability

The sigmoid is the right choice of "squashing function" for several converging reasons. First, it is a smooth, strictly increasing bijection from $\mathbb{R}$ to $(0,1)$, so any real-valued score becomes a valid probability without clipping or truncation. Second, it is the inverse of the logit (log-odds) function,

which means logistic regression is exactly the statement that log-odds are linear in the features:

This is the defining structural assumption of the model — not that the probability itself is linear (it cannot be, since probabilities are bounded), but that the log-odds are. Third, the sigmoid arises naturally as the posterior class probability whenever the class-conditional feature distributions belong to the exponential family with equal dispersion (for example, two Gaussians with a shared covariance matrix), which is exactly the generative assumption behind Linear Discriminant Analysis; logistic regression can be seen as directly modeling that same functional form without committing to the generative assumptions. A key algebraic fact used throughout the derivation is the sigmoid's self-referential derivative:

This single identity is what makes the gradient of the training objective come out so cleanly.

Setting up the likelihood. Given a training set $\{(x_i, y_i)\}_{i=1}^n$ with $y_i \in \{0,1\}$, and writing $p_i = \sigma(w^\top x_i + b)$, each label is modeled as a Bernoulli draw:

Assuming the examples are independent given their inputs, the likelihood of the whole dataset is

Maximizing $L$ is equivalent to maximizing its logarithm, and by convention we minimize the negative log-likelihood, which is exactly the binary cross-entropy loss:

This is the same objective foreshadowed in Maximum Likelihood and Cross-Entropy Loss: fitting logistic regression is maximum-likelihood estimation of a Bernoulli model with a linear-in-logit mean function.

Computing the gradient. Using the chain rule and $\sigma'(z) = \sigma(z)(1-\sigma(z))$, the gradient of $J$ with respect to $w$ collapses to a strikingly simple form:

The error term $(p_i - y_i)$ — predicted probability minus true label — is the "residual" of logistic regression, playing the same role that $(\hat y_i - y_i)$ plays in ordinary least squares. This is no coincidence: both are instances of generalized linear models, where the gradient of the negative log-likelihood always takes the form (residual) × (feature vector).

Fitting by gradient descent. Because there is no closed-form solution (unlike ordinary least squares), parameters are fit iteratively, exactly as in Gradient Descent for Parameter Estimation: ``` Input: data {(x_i, y_i)}_{i=1..n}, learning rate eta, iterations T, L2 penalty lambda >= 0 Initialize w <- 0 (vector of length d), b <- 0 for t = 1 .. T: grad_w <- 0 (vector), grad_b <- 0 for i = 1 .. n: z_i <- dot(w, x_i) + b p_i <- 1 / (1 + exp(-z_i)) # sigmoid error_i <- p_i - y_i grad_w <- grad_w + error_i x_i grad_b <- grad_b + error_i grad_w <- grad_w + lambda w # optional L2 regularization w <- w - eta grad_w / n b <- b - eta grad_b / n if ||grad_w|| < tolerance and |grad_b| < tolerance: break return w, b ``` Two practical notes are folded into the pseudocode. First, `1 / (1 + exp(-z_i))` should be computed with a numerically stable formulation (e.g. branching on the sign of `z_i`) to avoid overflow for large negative `z_i`. Second, an L2 penalty `lambda * ||w||^2` is commonly added to the loss to prevent the weights from diverging to infinity on linearly separable data — a subtlety explained under correctness below. At inference time, given a new $x$, compute $p = \sigma(w^\top x + b)$ and classify $\hat y = 1$ if $p \ge 0.5$ (equivalently, if $w^\top x + b \ge 0$), else $\hat y = 0$. The threshold 0.5 is a default; it can be moved to trade off precision and recall.

Time complexity. Each gradient-descent iteration touches every training example once to compute a dot product of length $d$ and accumulate a $d$-dimensional gradient update, giving $O(nd)$ work per iteration. Over $T$ iterations, full-batch gradient descent costs $O(Tnd)$. Stochastic or mini-batch gradient descent instead updates on one example (or a small batch $b$) at a time, costing $O(d)$ (or $O(bd)$) per step, trading a noisier gradient for far cheaper individual updates — the same tradeoff described generally in Gradient Descent for Parameter Estimation. Prediction on a new point costs $O(d)$: one dot product and one sigmoid evaluation.

Space complexity. Storing the parameters costs $O(d)$; storing the dataset costs $O(nd)$, which dominates in most regimes. Gradient descent needs no extra memory beyond one gradient vector of size $O(d)$, unlike methods that build an $n \times n$ Gram matrix.

Why the optimization is easy: convexity. The cross-entropy loss $J(w, b)$ is a convex function of $(w, b)$. This follows from writing

using $\log(1+e^{z}) - yz$ as the per-example loss. The term $-y_i(w^\top x_i + b)$ is linear in $(w,b)$, hence convex (and concave). The term $\log(1+e^z)$ is the softplus function of $z = w^\top x_i + b$; its second derivative with respect to $z$ is $\sigma(z)(1-\sigma(z)) > 0$, so softplus is strictly convex in $z$, and composing a convex function with an affine map of $(w,b)$ preserves convexity. A sum of convex functions is convex, so $J$ is convex. Consequently:

• Any local minimum of $J$ is a global minimum.
• Gradient descent with a sufficiently small, fixed or decaying learning rate is guaranteed to converge toward a global minimum (or, in degenerate cases discussed below, toward the boundary of the feasible region).
• Second-order methods (Newton–Raphson, i.e. Iteratively Reweighted Least Squares) converge locally at a quadratic rate because the Hessian

is positive semi-definite everywhere, another consequence of $\sigma(z)(1-\sigma(z)) \ge 0$.

The linear-separability caveat. If the training data are perfectly linearly separable, the maximum-likelihood solution does not exist as a finite point: pushing $\|w\| \to \infty$ in the separating direction drives every $p_i$ toward the correct label (0 or 1), decreasing the loss without bound but never reaching a minimizer. This is not a bug in the algorithm — the loss genuinely has no finite minimizer — and is why L2 (or L1) regularization is standard practice: adding $\lambda \|w\|^2$ makes the objective strictly convex and coercive, guaranteeing a unique finite minimizer for any $\lambda > 0$.

Honest tradeoffs. Logistic regression assumes a linear decision boundary in the original (or transformed) feature space; if the true boundary is curved, the model underfits unless you engineer nonlinear features or use basis expansions/kernels. It also implicitly assumes the log-odds are linear, an assumption that can fail even when a linear boundary would suffice, requiring diagnostic checks (e.g., partial residual plots) in careful applied work. Compared to Linear Discriminant Analysis, logistic regression makes fewer distributional assumptions (it is discriminative, not generative) and tends to be more robust when class-conditional distributions are non-Gaussian, at the cost of being less efficient (statistically) when the generative assumptions genuinely hold.

Consider a tiny 1-D dataset: hours studied $x$ versus pass/fail label $y$.

Suppose after training (via the gradient descent loop above, several hundred iterations, $\eta = 0.5$) we converge approximately to $w \approx 2.0$, $b \approx -4.5$ (illustrative rounded values; exact numbers depend on the optimizer path, but this is close to what gradient descent finds on this data). Let's trace predictions and the per-example loss contribution at this fitted point.

Total loss $\approx 0.618$, average $\approx 0.103$ nats per example — small, consistent with a well-separated dataset. Note the model does not report certainty ($p=0$ or $p=1$) even though the classes are perfectly separated by $x = 2.25$; it reports high but finite confidence, exactly because of the regularization (or early stopping) that prevents $\|w\|$ from blowing up as discussed above.

Gradient check at this point, to sanity-check convergence: take $x = 1.5, y = 0$. Then $p = 0.182$, error $= p - y = 0.182$. This example's contribution to $\partial J/\partial w$ is $0.182 \times 1.5 = 0.273$, a small positive number nudging $w$ slightly downward — but symmetric contributions from the $y=1$ points on the other side push $w$ upward by comparable amounts, and at convergence these very nearly cancel across the whole dataset, which is exactly the stationarity condition $\nabla_w J \approx 0$ that gradient descent is driving toward.

Decision boundary. Setting $z = 0$: $2.0x - 4.5 = 0 \Rightarrow x = 2.25$ hours. Any student studying more than 2.25 hours is predicted to pass; this threshold is the model's single learned "cutoff," and the smooth sigmoid around it quantifies how confident that cutoff is at intermediate study times.

The logistic function itself predates machine learning by over a century, originating in 19th-century population-growth models, but its use for statistical classification and its marriage with maximum-likelihood estimation is what created modern logistic regression.

Logistic regression is the workhorse baseline of applied predictive modeling: credit scoring and loan-default models at banks, click-through-rate and conversion-rate prediction in online advertising, spam and fraud detection, and clinical risk scores (like predicting readmission or disease risk) all commonly start from — and often ship — a regularized logistic regression, prized for being fast, interpretable (coefficients have an odds-ratio interpretation), and well-calibrated. In modern deep learning, the sigmoid and its multi-class generalization, the softmax, are the near-universal final layer for classification networks, converting the last linear layer's raw scores into probabilities, and the cross-entropy loss derived here is the loss function trained by backpropagation in essentially every classifier network. The same mathematics also underlies logistic components of generalized linear models used throughout applied statistics, and it is the natural jumping-off point for comparing decision-boundary philosophies against Linear Discriminant Analysis (generative, Gaussian-based) and Support Vector Machines and Maximum Margin (margin-based, non-probabilistic by default).

Ask the tutor: How would the training dynamics and final decision boundary change if I used the hinge loss instead of cross-entropy on this same linear score?