Logistic Regression

Logistic regression works in three steps: compute a weighted sum of the input features, pass that sum through the sigmoid function to get a probability, then use that probability to make a class prediction.

Step 1 — The Linear Combination

Explanation: Each input feature x is multiplied by a learned weight w, all terms are summed, and a bias b is added — this is identical to linear regression up to this point.

• $x_1, x_2, \dots, x_n$ — the input features (e.g., age, income, word counts)
• $w_1, w_2, \dots, w_n$ — learned weights, one per feature
• $b$ — the bias term (shifts the decision boundary)
• $z$ — the raw score, which can be any real number

Step 2 — The Sigmoid Function

Explanation: The sigmoid squashes any real number z into the range (0, 1), making it interpretable as a probability.

• $e$ — Euler's number (~2.718)
• $\sigma(z)$ — the predicted probability that the input belongs to class 1
• When $z$ is very large, $\sigma(z) \approx 1$; when very negative, $\sigma(z) \approx 0$; when $z = 0$, $\sigma(z) = 0.5$

Step 3 — The Decision

Explanation: The predicted class y-hat is determined by whether the probability crosses the chosen threshold (0.5 by default).

Training — Log Loss The model is trained by minimising binary cross-entropy loss (also called log loss):

Explanation: Log loss penalises the model heavily when it is confident but wrong, and lightly when it is uncertain — pushing the model to make well-calibrated probability predictions.

• $m$ — number of training examples
• $y^{(i)}$ — the true label (0 or 1) for example $i$
• $\hat{p}^{(i)}$ — the predicted probability for example $i$

Python Implementation ```python from sklearn.linear_model import LogisticRegression from sklearn.datasets import load_breast_cancer from sklearn.model_selection import train_test_split from sklearn.preprocessing import StandardScaler from sklearn.metrics import accuracy_score, classification_report

# Load a real dataset data = load_breast_cancer() X, y = data.data, data.target

# Split into train/test sets X_train, X_test, y_train, y_test = train_test_split( X, y, test_size=0.2, random_state=42 )

# Feature scaling is important for logistic regression! scaler = StandardScaler() X_train = scaler.fit_transform(X_train) X_test = scaler.transform(X_test)

# Train the model model = LogisticRegression(max_iter=1000) model.fit(X_train, y_train)

# Predict classes and probabilities y_pred = model.predict(X_test) y_prob = model.predict_proba(X_test)[:, 1] # probability of class 1 print(f"Accuracy: {accuracy_score(y_test, y_pred):.3f}") print(classification_report(y_test, y_pred)) ``` ```python

# Understanding the threshold — changing it matters! import numpy as np

# Lower threshold: catch more positives (higher recall) custom_threshold = 0.3 y_pred_custom = (y_prob >= custom_threshold).astype(int) print(f"At threshold 0.3, accuracy: {accuracy_score(y_test, y_pred_custom):.3f}")

# Inspect learned weights — tells you which features matter most feature_names = data.feature_names weights = model.coef_[0] top_features = sorted(zip(feature_names, weights), key=lambda x: abs(x[1]), reverse=True)[:5] print("\nTop 5 most influential features:") for name, weight in top_features: direction = "↑ positive" if weight > 0 else "↓ negative" print(f" {name:35s} weight={weight:+.3f} ({direction})") ```

Linear Regression · Decision Trees · Naive Bayes · Cross-Entropy Loss · Gradient Descent