Data Preprocessing

Models are only as good as the data fed to them. This lecture covers cleaning that data — handling missing values, removing outliers, scaling features and encoding text into numbers.

2.1 Why preprocessing matters

Think of an ML model as a Ferrari engine and data as the fuel. Pour in contaminated fuel and even a Ferrari breaks down. The preprocessing pipeline has four stages we care about: load → inspect → clean (missing values + outliers) → transform (scale + encode).

Inspecting data first

Before cleaning anything, you must understand the data. Key pandas tools:

2.2 Handling Missing Values

Why is data missing? (three mechanisms)

Strategy 1 — Deletion

• Row deletion — drop rows with missing values. Use only when missing data is small (< 5%) and random.
• Column deletion — drop a whole column when it has > 60% missing data.

df_dropped = df.dropna()        # drop every row containing any NaN

Strategy 2 — Imputation (Mean / Median / Mode)

Imputation means filling the gaps instead of deleting them.

import pandas as pd
df = pd.read_csv('loan_data.csv')

# Mean imputation — risky if outliers exist
df['Age'] = df['Age'].fillna(df['Age'].mean())

# Median imputation — safe choice for skewed columns like Income
df['Income'] = df['Income'].fillna(df['Income'].median())

# Mode imputation — for categorical columns
df['City'] = df['City'].fillna(df['City'].mode()[0])

Strategy 3 — Time-series filling

For time-series data (e.g. stock prices) values depend on neighbouring days:

• Forward Fill (ffill) — propagate the last valid observation forward.
• Backward Fill (bfill) — use the next valid observation to fill the gap.

2.3 Handling Outliers

Outliers pull the mean toward them and distort analysis. Algorithms like Linear Regression are highly sensitive to them. The boxplot is the standard visual tool — its box holds the middle 50% of data, whiskers extend 1.5×IQR, and points beyond are flagged as outliers.

Method 1 — Z-Score

The Z-score tells you how many standard deviations a point is from the mean.

Method 2 — IQR (Interquartile Range)

More robust than Z-score because it uses quartiles, not the mean/std (which are themselves affected by outliers). Invented by John Tukey.

# 1. Calculate Q1 and Q3
Q1 = df['Income'].quantile(0.25)
Q3 = df['Income'].quantile(0.75)

# 2. Compute the IQR and the fences
IQR = Q3 - Q1
lower = Q1 - 1.5 * IQR
upper = Q3 + 1.5 * IQR

# 3. Keep only rows inside the fences
df_clean = df[(df['Income'] >= lower) & (df['Income'] <= upper)]
print("Rows removed:", len(df) - len(df_clean))

Handling strategies

• Trimming — completely remove the outlier rows. Pro: simple. Con: data loss.
• Capping (Winsorizing) — replace outliers with the upper/lower limit. Pro: preserves data size. Con: modifies the distribution.

2.4 Feature Scaling

Min-Max Scaling (Normalization)

Squashes every value into the range [0, 1].

Standard Scaling (Z-Score Standardization)

Centres data around mean 0 with standard deviation 1. Handles outliers slightly better than Min-Max.

from sklearn.preprocessing import MinMaxScaler, StandardScaler

# Min-Max: scales every value into [0, 1]
mm = MinMaxScaler()
df['Income'] = mm.fit_transform(df[['Income']])   # note: double brackets = 2D

# Standard: mean 0, std 1
ss = StandardScaler()
df['Salary'] = ss.fit_transform(df[['Salary']])

2.5 Data Encoding Methods

Two kinds of categorical data

• Ordinal data — categories with a natural order: Low < Medium < High; S < M < L < XL; B.Tech < M.Tech < PhD.
• Nominal data — categories with no order: Red, Green, Blue; New York, Paris, Tokyo; Dog, Cat, Bird.

Method 1 — Label Encoding (for Ordinal data)

Assigns each category a unique integer (S=0, M=1, L=2).

from sklearn.preprocessing import LabelEncoder

le = LabelEncoder()
df['Education'] = le.fit_transform(df['Education'])
# B.Tech -> 0,  M.Tech -> 1,  PhD -> 2  (order is preserved)

Method 2 — One-Hot Encoding (for Nominal data)

Creates a separate binary column for each category — 1 if present, 0 if not. No fake ordering is introduced.

import pandas as pd

# 'Color' has values Red / Green / Blue
df = pd.get_dummies(df, columns=['Color'], drop_first=True)
# Creates columns: Color_Green, Color_Blue
# (Red is the dropped baseline)

drop_first=True drops one column to avoid the Dummy Variable Trap (multicollinearity — one column being perfectly predictable from the others).

Choose your answer, then check it. Coding questions reveal a full solution with output.

import pandas as pd

df = pd.read_csv('data.csv')

# Count missing values in every column
print(df.isnull().sum())

# Fill missing Age with the median (robust to outliers)
df['Age'] = df['Age'].fillna(df['Age'].median())

print("Missing in Age after fill:", df['Age'].isnull().sum())

Q1 = df['Salary'].quantile(0.25)
Q3 = df['Salary'].quantile(0.75)
IQR = Q3 - Q1

lower = Q1 - 1.5 * IQR
upper = Q3 + 1.5 * IQR

df_clean = df[(df['Salary'] >= lower) & (df['Salary'] <= upper)]
print("Before:", len(df), " After:", len(df_clean))