Linear Regression

Linear regression fits a line (or plane) to your data by finding the set of weights that minimize prediction errors. Here is the machinery from top to bottom.

The Prediction Formula

Explanation: The predicted value ŷ is a weighted sum of all input features plus a bias term, where the model learns the weights w during training.

• $\hat{y}$: the model's prediction (e.g., predicted house price)
• $w_0$: the bias (or intercept) — the baseline prediction when all features are zero
• $w_1 \ldots w_n$: learned weights (coefficients) for each feature
• $x_1 \ldots x_n$: the input feature values for one example

The Loss Function: Mean Squared Error

Explanation: The model's error is measured as the average of the squared differences between predictions and true values, penalizing large mistakes more heavily.

• $m$: the number of training examples
• $\hat{y}_i$: the predicted value for example $i$
• $y_i$: the true (actual) value for example $i$
• Squaring: ensures errors are always positive and punishes big misses harder than small ones

The model's goal is to find the weights $w_0, w_1, \ldots, w_n$ that make this number as small as possible.

The Optimal Weights (Normal Equation)

Explanation: For small datasets, you can compute the perfect weights in one shot using this closed-form formula — no iterative training required.

• $\mathbf{X}$: the feature matrix (each row is one training example)
• $\mathbf{X}^T$: the transpose of the feature matrix
• $\mathbf{y}$: the vector of true output values
• $(\cdot)^{-1}$: matrix inverse — only works if the matrix is invertible (features are not perfectly correlated)

In practice, for larger datasets we use Gradient Descent instead of inverting the matrix, since matrix inversion is computationally expensive at scale. --- Here is a minimal working example in Python using scikit-learn: ```python import numpy as np from sklearn.linear_model import LinearRegression import matplotlib.pyplot as plt

# --- 1. Create some toy data ---

# Imagine: house size (sq ft) → price ($1000s) np.random.seed(42) X = np.random.uniform(500, 3500, size=(50, 1)) # feature: size y = 0.15 * X.ravel() + np.random.normal(0, 30, 50) # true price + noise

# --- 2. Fit the model --- model = LinearRegression() model.fit(X, y)

# --- 3. Inspect what was learned --- print(f"Intercept (w0): {model.intercept_:.2f}") print(f"Coefficient (w1): {model.coef_[0]:.4f}")

# Intercept (w0): 12.43

# Coefficient (w1): 0.1489 ← close to true 0.15 ✓

# --- 4. Make a prediction --- size = np.array(2000) predicted_price = model.predict(size) print(f"Predicted price for 2000 sq ft: ${predicted_price[0]:.1f}k")

# --- 5. Evaluate --- from sklearn.metrics import mean_squared_error, r2_score y_pred = model.predict(X) print(f"MSE: {mean_squared_error(y, y_pred):.2f}") print(f"R²: {r2_score(y, y_pred):.4f}")

# R² close to 1.0 = great fit; 0.0 = no better than predicting the mean ```

What R² tells you: R² (R-squared) measures the proportion of variance in `y` explained by your model. An R² of 0.85 means your model explains 85% of the variation in the target — the remaining 15% is noise or missing features. --- Here is a slightly more advanced example with multiple features and feature engineering: ```python from sklearn.preprocessing import PolynomialFeatures from sklearn.pipeline import Pipeline

# Add polynomial features (x, x², x³) — still linear regression! poly_model = Pipeline([ ("poly_features", PolynomialFeatures(degree=2, include_bias=False)), ("linear_reg", LinearRegression()) ]) poly_model.fit(X, y) print(f"Poly R²: {r2_score(y, poly_model.predict(X)):.4f}")

# Still using LinearRegression — just with engineered features ```

Logistic Regression · Ridge Regression · LASSO Regression · Support Vector Machines · Decision Trees