Supervised Learning — Linear Regression

The first and most fundamental supervised model. Learn how a straight line can predict numbers, how it learns the "best" line, the assumptions it relies on, and where it breaks.

3.1 What Linear Regression is

It is a regression task — the output is a continuous number (house price, salary, temperature, sales). It is usually the first supervised model taught because it is:

• Simple — conceptually easy to grasp.
• Interpretable — the slope directly tells you the effect of each feature.
• Fast — computationally efficient, even on large datasets.
• Effective — performs well when the relationship really is roughly linear.

Two types

• Simple Linear Regression — one input variable. Revenue = f(loaves)
• Multiple Linear Regression — many inputs. Revenue = f(loaves, advertising, weekday, discount)

The core idea is identical: fit a line/plane that best predicts the output.

3.2 Model Formulation

The simple linear regression equation:

You may also see it written y = mx + c where m is the slope and c the intercept — identical idea.

3.3 How the model learns — Ordinary Least Squares

Many straight lines could pass through a cloud of points. Which one does the model pick? The one that makes the smallest overall mistakes.

Residuals (prediction errors)

Why square the errors?

If you simply add raw errors, positive and negative residuals cancel out — a bad line could look "good" by accident. So Linear Regression squares every error (removing the sign), adds them up, and chooses the line with the smallest total. This is Ordinary Least Squares (OLS).

The closed-form OLS formulas

Doing it in code — by hand then with scikit-learn

import pandas as pd

df = pd.DataFrame({"Size": [800,1000,1200,1500,1800],
                   "Price": [40,60,65,75,110]})
x, y = df["Size"], df["Price"]
x_mean, y_mean = x.mean(), y.mean()

# slope and intercept from the OLS formulas
m = ((x - x_mean) * (y - y_mean)).sum() / ((x - x_mean) ** 2).sum()
c = y_mean - m * x_mean
print(f"Price = {m:.4f} x Size + {c:.2f}")
print("Predict 1400 sq ft:", m * 1400 + c)

from sklearn.linear_model import LinearRegression

X = df[["Size"]]       # 2D for sklearn
y = df["Price"]

model = LinearRegression()
model.fit(X, y)

print("Slope (coef_):", model.coef_[0])
print("Intercept:", model.intercept_)
print("Predict 1400:", model.predict([[1400]]))

Diagnostic plots

• Residuals vs Fitted plot — checks linearity & constant variance. Points should scatter randomly around 0.
• Q-Q plot — checks if residuals are normally distributed. Points should lie along the diagonal.

3.4 The Seven Assumptions

Linear Regression only works well if these assumptions are approximately true. Examiners frequently ask you to name or recognise them.

3.5 Limitations of Linear Regression

• Only models linear relationships — it fails badly on curved or complex non-linear patterns.
• Extremely sensitive to outliers — a single extreme point can drastically tilt the fitted line (because OLS squares errors, big errors dominate).
• Struggles with correlated predictors — multicollinearity makes coefficient estimates unstable and hard to interpret.
• Omitted variable bias — leaving out an important feature distorts the coefficients of the remaining ones.

Choose, check, and read every explanation — these mirror the exam style closely.

from sklearn.linear_model import LinearRegression

X = [[1], [2], [3], [4], [5]]
y = [3, 5, 7, 9, 11]              # the true rule is y = 2x + 1

model = LinearRegression()
model.fit(X, y)

print("Slope:", model.coef_[0])
print("Intercept:", model.intercept_)
print("Predict x=8:", model.predict([[8]]))

from sklearn.model_selection import train_test_split
from sklearn.linear_model import LinearRegression

X = df[['Hours', 'PrevScore']]   # multiple features
y = df['Result']

X_train, X_test, y_train, y_test = train_test_split(
    X, y, train_size=0.75, random_state=42)

model = LinearRegression()
model.fit(X_train, y_train)

print("Coefficients:", model.coef_)     # one weight per feature
print("Intercept:", model.intercept_)