Jaber Notes · 6 of 16

Feature Engineering

Scaling, encoding, missing data, outliers, selection, TF-IDF, leakage.

The unglamorous work that decides most projects: scaling and encoding choices, MCAR/MAR/MNAR missing-data theory, outlier handling, feature selection families, and the pipeline discipline that prevents leakage.

"Feature engineering is the art of turning raw data into a representation that makes learning easy." Most ML competitions are won here, not by model choice.

Why feature engineering matters
Numerical features
Categorical features
Missing data
Outlier handling
Feature creation and transformation
Feature selection
Curse of dimensionality
Text features (basic)
Data leakage (critical)
Sklearn pipeline

1. Why feature engineering matters

A model can only use the information in its input features. If the right information is not present (or is obscured), no algorithm can learn it. Feature engineering:

Encodes domain knowledge into the model.
Converts raw formats (text, dates, categories) into numbers.
Scales features so that gradient-based methods work well.
Reduces noise and irrelevant signal.

Principle: apply transformations that make the relationship between features and target more linear and the features more Gaussian-like (many algorithms assume these implicitly).

2. Numerical features

2.1 Scaling

Most algorithms assume features are on a similar scale. Without scaling, features with larger ranges dominate distances and gradients.

StandardScaler (Z-score normalization):

x' = \frac{x - \mu}{\sigma}.

Output: mean 0, std 1. Assumes roughly Gaussian distribution. Common default for linear models, neural nets, SVMs, PCA.

MinMaxScaler:

x' = \frac{x - x_\min}{x_\max - x_\min} \in [0,1].

Preserves relative distances. Sensitive to outliers (one extreme value compresses everything else).

RobustScaler:

x' = \frac{x - \text{median}}{\text{IQR}}.

Uses median and interquartile range → robust to outliers. Good for skewed data with outliers.

MaxAbsScaler: divides by max absolute value → output in $[-1,1]$ . Useful for sparse data (does not shift center).

When scaling is NOT needed: tree-based models (decision trees, random forests, gradient boosting). They split on thresholds and are invariant to monotone transforms.

2.2 Log transform

For right-skewed features (e.g., income, population, prices):

x' = \log(x + c), \quad c > 0 \text{ to handle zeros.}

Compresses the right tail, making distribution more symmetric. Use when values span multiple orders of magnitude.

Box-Cox transform (general power transform):

x^{(\lambda)} = \begin{cases} (x^\lambda - 1)/\lambda & \lambda \neq 0 \\ \log x & \lambda = 0. \end{cases}

sklearn.preprocessing.PowerTransformer estimates optimal $\lambda$ via MLE.

2.3 Binning (discretization)

Convert continuous feature to categorical bin: e.g., age → {child, teen, adult, senior}.

Makes model robust to slight outliers.
Loses precision within bins (information loss).
Useful when relationship is non-monotone and you don't want to over-engineer nonlinear transforms.

2.4 Polynomial features

Add $x_1^2, x_2^2, x_1 x_2, \ldots$ to allow linear model to fit nonlinear boundaries. Degree- $d$ polynomial features of $p$ features: $\binom{p+d}{d}$ total features — grows fast.

from sklearn.preprocessing import PolynomialFeatures
poly = PolynomialFeatures(degree=2, include_bias=False)
X_poly = poly.fit_transform(X)

3. Categorical features

3.1 Ordinal encoding

Assign integers: {small, medium, large} → {0, 1, 2}. Only valid when there is a meaningful order. sklearn.preprocessing.OrdinalEncoder.

3.2 One-hot encoding (OHE)

Create one binary column per category. "cat" → [1,0,0], "dog" → [0,1,0], "fish" → [0,0,1].

No artificial ordering implied.
Creates high-dimensional sparse features for high-cardinality categories.
Dummy variable trap: for linear models, drop one category per feature (multicollinearity if intercept included). sklearn handles this with drop='first' or drop='if_binary'.

from sklearn.preprocessing import OneHotEncoder
enc = OneHotEncoder(sparse_output=True, handle_unknown='ignore', drop='first')

3.3 Target encoding (mean encoding)

Replace category with mean of target in that category:

x_\text{enc} = \frac{\sum_{i: x_i = c} y_i}{|\{i: x_i = c\}|}.

Risk: severe data leakage if done on full training set. Use leave-one-out or k-fold cross-mean encoding to prevent this.

Smoothing (avoid noisy estimates for rare categories):

x_\text{enc} = \frac{n_c \cdot \mu_c + k \cdot \mu_\text{global}}{n_c + k},

where $n_c$ = count, $\mu_c$ = category mean, $\mu_\text{global}$ = global mean, $k$ = smoothing parameter.

3.4 Frequency encoding

Replace category with its frequency in training data. Captures rarity.

3.5 Binary encoding

Encode category as binary number, use each bit as a feature. Reduces dimensionality vs OHE for high-cardinality features.

3.6 Handling high cardinality

Categories with > 50–100 levels: OHE creates too many features → use target encoding, hashing, embeddings (DL).
Group rare categories into "Other" bucket before encoding.
Feature hashing (hashing trick): map categories to a fixed-size vector using a hash function. Compact; collisions are usually tolerable.

4. Missing data

4.1 Types of missingness

Type	Description	Implication
MCAR (Missing Completely At Random)	P(missing) independent of all variables	Simplest case; simple imputation unbiased
MAR (Missing At Random)	P(missing) depends on observed variables	Conditional imputation valid
MNAR (Missing Not At Random)	P(missing) depends on the missing value itself	Hardest; missingness itself is informative

4.2 Simple imputation

from sklearn.impute import SimpleImputer
imp = SimpleImputer(strategy='mean')    # or 'median', 'most_frequent', 'constant'

Mean: good for symmetric distributions without outliers.
Median: robust to skewed distributions and outliers.
Mode (most_frequent): for categorical features.
Constant: mark missing as a specific value (e.g., -999, "MISSING").

4.3 Adding a missingness indicator

Create a binary feature "was feature X missing?" alongside imputed value. Lets the model learn if missingness is informative. Always worth trying when MNAR is plausible.

from sklearn.impute import MissingIndicator
from sklearn.pipeline import FeatureUnion

4.4 Multivariate imputation

KNN Imputer: impute missing values from $k$ nearest neighbors in feature space. Respects correlations but slow for large datasets.

Iterative Imputer (MICE — Multiple Imputation by Chained Equations): model each feature with missing values as a function of others; iterate. sklearn.impute.IterativeImputer. More accurate, more expensive.

from sklearn.impute import KNNImputer, IterativeImputer
imp = IterativeImputer(max_iter=10, random_state=0)

5. Outlier handling

5.1 Detection methods

Z-score: $|z| = |x-\mu|/\sigma > 3$ → outlier (assumes Gaussian).

IQR method: outlier if $x < Q1 - 1.5\cdot\text{IQR}$ or $x > Q3 + 1.5\cdot\text{IQR}$ . More robust.

Isolation Forest: unsupervised tree-based method. Outliers are points that are isolated with few splits. sklearn.ensemble.IsolationForest.

Local Outlier Factor (LOF): compares local density of a point to its neighbors. Good for density-based outliers. sklearn.neighbors.LocalOutlierFactor.

5.2 Handling strategies

Strategy	When
Remove	Confirmed data entry errors
Cap/winsorize	Keep but limit extreme values
Log transform	Compresses heavy tail
Robust model	Use MAE loss, robust scalers
Leave	Tree-based models are largely unaffected

Winsorization: clip feature to $[p_5, p_{95}]$ percentiles (or other bounds).

6. Feature creation and transformation

6.1 Date and time features

From a datetime: extract year, month, day, hour, weekday, is_weekend, is_holiday, days_since_epoch.

Cyclical encoding for periodic features (hour, day of week, month): encode as sine/cosine pair to preserve continuity at period boundaries:

x_\sin = \sin\!\left(\frac{2\pi x}{T}\right), \quad x_\cos = \cos\!\left(\frac{2\pi x}{T}\right).

Example: hour 23 and hour 0 should be close; standard integer encoding makes them far apart.

6.2 Interaction features

$x_1 \cdot x_2$ : captures multiplicative effects. E.g., price × quantity = revenue.

6.3 Aggregation features

In tabular data with groups (e.g., customer_id has many transactions), aggregate: mean, max, min, std, count per group. E.g., "number of purchases by this customer in last 30 days."

6.4 Domain-specific features

Often the most valuable. Examples:

Finance: log returns, rolling volatility, Sharpe ratio.
NLP: TF-IDF, sentence length, character n-grams.
Images: HOG, SIFT (classical), or backbone embeddings (DL).
Medical: BMI = weight / height², lab value ratios.

7. Feature selection

7.1 Why select features?

Reduces overfitting (fewer irrelevant features = lower variance).
Speeds up training.
Improves interpretability.
Removes noise.

7.2 Filter methods (no model needed)

Variance threshold: remove near-constant features.

from sklearn.feature_selection import VarianceThreshold

Univariate statistical tests:

ANOVA F-test / chi-squared for classification.
Pearson/Spearman correlation with target for regression.

from sklearn.feature_selection import SelectKBest, f_classif, mutual_info_classif

Mutual information: $I(X;Y)$ captures nonlinear dependencies.

7.3 Wrapper methods

Recursive Feature Elimination (RFE): fit model, rank features by importance, remove least important, repeat.

from sklearn.feature_selection import RFE
rfe = RFE(estimator=LinearRegression(), n_features_to_select=10)

Forward/backward selection: greedily add/remove features based on validation score. Expensive.

7.4 Embedded methods

Model performs selection during training:

Lasso: sets irrelevant weights to exactly 0.
Tree-based feature importance: split-based or permutation importance.
sklearn.feature_selection.SelectFromModel.

7.5 Permutation importance

For any fitted model: permute values of feature $j$ in validation set; measure drop in performance. No permutation drop = feature unimportant.

from sklearn.inspection import permutation_importance
result = permutation_importance(model, X_val, y_val, n_repeats=10)

Advantages: works for any model, captures actual contribution to performance (not just split counts which are biased toward high-cardinality features in trees).

8. Curse of dimensionality

8.1 The phenomenon

As dimensionality $d$ increases, data becomes exponentially sparse. The volume of a unit hypersphere relative to the enclosing hypercube $\to 0$ as $d \to \infty$ .

Effect on distances: in high dimensions, the ratio of the maximum to minimum pairwise distance among random points → 1. "All points are roughly equally far apart." Distance-based methods (k-NN, k-means, kernel methods with RBF) degrade.

8.2 Mathematical illustration

For $n$ uniform points in $[0,1]^d$ , the expected distance from a query to its nearest neighbor:

\mathbb{E}[\text{NN distance}] \approx 1 - \left(1 - \frac{1}{n}\right)^{1/d} \approx \left(\frac{1}{n}\right)^{1/d}.

To keep the same nearest-neighbor distance as $d$ doubles, you need $n^2$ points — exponential growth in required data.

8.3 Manifestations in ML

k-NN accuracy degrades unless $n \gg k^d$ .
K-means clusters become less meaningful.
Linear classifiers often outperform complex models on high-dimensional sparse data (NLP bag-of-words).
Overfitting worsens as $p/n$ ratio increases.

8.4 Mitigations

Feature selection: remove irrelevant dimensions.
Dimensionality reduction: PCA, LDA (supervised), autoencoders.
Regularization: limits effective model complexity.
Implicit structure: most real high-dimensional data lies on a low-dimensional manifold (images, audio, text have much lower intrinsic dimension than pixel count).

9. Text features (basic)

9.1 Bag of Words (BoW)

Represent document as word count vector. Ignores grammar and order.

from sklearn.feature_extraction.text import CountVectorizer
vec = CountVectorizer(max_features=10_000, ngram_range=(1,2))
X = vec.fit_transform(docs)  # sparse matrix

9.2 TF-IDF (Term Frequency-Inverse Document Frequency)

Down-weights common words:

\text{TF-IDF}(t,d) = \underbrace{\text{count}(t,d)}_\text{TF} \cdot \log\!\frac{N}{\text{df}(t)+1},

where $N$ = total documents, $\text{df}(t)$ = documents containing term $t$ .

from sklearn.feature_extraction.text import TfidfVectorizer

9.3 N-grams

Include bigrams (pairs of consecutive words) to capture some context: "not good" as a unit vs. "not" and "good" separately.

10. Data leakage (critical)

10.1 What is leakage?

Any feature (or preprocessing step) that uses information from the future, or information that would not be available at prediction time.

10.2 Types of leakage

Type	Example
Target leakage	Including a feature that was computed using the label
Temporal leakage	Using data from time T+1 to predict at time T
Preprocessing leakage	Scaling/imputation fit on full dataset including test
Group leakage	Same patient in both train and test

10.3 Prevention checklist

[ ] Fit every preprocessing transformer (scaler, encoder, imputer, feature selector) only on training data.
[ ] Use sklearn.pipeline.Pipeline to enforce this.
[ ] For temporal data: always split by time, never shuffle.
[ ] Audit feature definitions: could this value have been known at prediction time?
[ ] Look for suspiciously high performance (leakage often makes models "too good").

11. Sklearn pipeline

The Pipeline is the correct way to prevent leakage and ensure reproducibility.

from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler, OneHotEncoder
from sklearn.impute import SimpleImputer
from sklearn.compose import ColumnTransformer
from sklearn.ensemble import GradientBoostingClassifier

numeric_features = ['age', 'income', 'score']
categorical_features = ['city', 'gender']

numeric_transformer = Pipeline([
    ('impute', SimpleImputer(strategy='median')),
    ('scale', StandardScaler()),
])

categorical_transformer = Pipeline([
    ('impute', SimpleImputer(strategy='most_frequent')),
    ('ohe', OneHotEncoder(handle_unknown='ignore', sparse_output=False)),
])

preprocessor = ColumnTransformer([
    ('num', numeric_transformer, numeric_features),
    ('cat', categorical_transformer, categorical_features),
])

full_pipeline = Pipeline([
    ('preprocess', preprocessor),
    ('model', GradientBoostingClassifier(n_estimators=200)),
])

# Everything happens correctly inside cross-validation:
from sklearn.model_selection import cross_val_score
scores = cross_val_score(full_pipeline, X, y, cv=5, scoring='roc_auc')
# The preprocessor is fit only on the training fold each time!

*File: notes/06_feature_engineering.md — next: notes/07_classical_algorithms.md*