Feature Engineering: Complete Guide to Feature Engineering

Summary

Feature Engineering is the art and science of transforming raw data into predictive variables optimized for training Machine Learning models. This discipline occupies a central place in any machine learning project: according to the well-known practitioner’s adage, “data is better than the algorithm,” and Feature Engineering is precisely the bridge between imperfect data and exceptional model performance.

This complete guide will cover the fundamental mathematical principles of variable selection (filters, wrappers, embedded methods, and PCA), the practical intuition behind each technique, a detailed Python implementation with scikit-learn, critical hyperparameters, the advantages and limitations of each approach, as well as four concrete use cases illustrating the impact of Feature Engineering in real-world situations.

Mathematical Principle of Feature Engineering and Variable Selection

Feature Engineering rests on two inseparable pillars: the creation of new features from raw data, and the selection of the most relevant variables for the model. Selection relies on four major families of methods, each with its own mathematical foundations.

1. Filter Methods

Filter methods evaluate each variable independently of the final model, using univariate statistical criteria. They are fast and interpretable.

Pearson Correlation — Measures the linear relationship between an explanatory variable X and the target variable Y:

r = cov(X, Y) / (σ_X · σ_Y)

where cov(X, Y) is the covariance between X and Y, σ_X and σ_Y are the respective standard deviations. The coefficient r ∈ [-1, 1]: a value close to ±1 indicates a strong linear correlation, while a value close to 0 suggests the absence of a linear relationship. In Feature Engineering, features with |r| < 0.1 are typically discarded, as they provide little predictive information.

Chi-squared Test (χ²) — Evaluates the independence between two categorical variables:

χ² = Σ (Observed – Expected)² / Expected

where the sum is over all cells of the contingency table. A high χ² value (and a p-value < 0.05) indicates that the explanatory variable and the target are not independent: the feature is therefore informative for classification.

Mutual Information — Measures the general dependence (linear and non-linear) between two variables:

I(X ; Y) = Σ p(x, y) · log(p(x, y) / (p(x) · p(y)))

where p(x, y) is the joint distribution and p(x), p(y) are the marginal distributions. Unlike Pearson correlation, mutual information captures any form of dependence, not just linear relationships. It is a powerful tool for identifying features whose relationship with the target would be invisible to a simple correlation test.

2. Wrapper Methods

Wrapper methods evaluate subsets of variables by actually training the model at each iteration. The most emblematic is Recursive Feature Elimination (RFE):

1. Train the model on the complete set of features.
2. Identify the least important variables (weakest coefficients for a linear model, lowest importances for a tree).
3. Remove these variables and retrain the model.
4. Repeat until the desired number of features is reached.

RFE is computationally expensive (it requires N successive training runs) but generally produces more performant variable subsets than univariate filters, as it accounts for interactions between features.

3. Embedded Methods

Embedded methods integrate variable selection directly into the model training process. The most famous is L1 regularization (Lasso):

L = Error + λ · Σ |w_j|

where Error is the cost function (e.g., mean squared error), λ is the regularization coefficient, and w_j are the model coefficients. The L1 penalty drives some coefficients exactly to zero, thus automatically eliminating irrelevant variables. It is “free” variable selection, integrated into learning.

In comparison, L2 regularization (Ridge) uses Σ w_j² instead of Σ |w_j|: it reduces the magnitude of coefficients without zeroing them, but does not perform selection per se.

4. PCA — Principal Component Analysis

PCA (Principal Component Analysis) projects data onto the axes of greatest variance. Mathematically, we compute the eigenvalues λ_i of the covariance matrix Σ of the data. Each λ_i represents the variance captured by the i-th principal component. By sorting the components by decreasing λ_i and keeping only the first k, we reduce dimensionality while preserving most of the information.

PCA is a linear transformation: the new components are linear combinations of the original variables. This means interpretability is lost — we no longer know which original variable contributes the most — but the gain in dimensionality reduction is often considerable.

Intuition: The Art of Feature Engineering

Feature Engineering is a bit like preparing your ingredients before cooking. A good chef doesn’t throw all the raw vegetables into the pot — they cut them, season them, remove what isn’t good. A good data scientist does exactly the same thing with their variables: they create relevant interactions, encode categories correctly, and keep only the features that really matter.

Let’s take a concrete example. Imagine you’re building a model to predict the price of an apartment. You have the areas of the rooms. Feature Engineering suggests creating a new variable: the total area (sum of all rooms). You could also calculate the room-to-area ratio, or create a binary indicator “has a balcony.” These derived variables didn’t exist in the raw data, but they are often far more predictive than the individual variables.

Variable selection, on the other hand, is like sorting. Among the hundred features available, which ones truly provide information? Which are redundant? Which add noise rather than signal? A model trained on 15 well-chosen features will almost always beat a model trained on 100 features of which 85 are useless — and for three fundamental reasons:

1. Reduction of overfitting: fewer parameters to learn = a more generalizable model.
2. Training speedup: fewer features = fewer computations.
3. Improved interpretability: it’s easier to explain a model with 10 variables than one with 200.

The curse of dimensionality perfectly illustrates this phenomenon: in very high-dimensional spaces, data becomes exponentially sparse, and distances between points lose their meaning. Feature Engineering and variable selection are the direct antidotes to this problem.

Python Implementation With scikit-learn

Here is a complete and progressive implementation of Feature Engineering with the scikit-learn library, covering filter selection, RFE wrapper, categorical encoding, interaction creation, and building reproducible Pipelines.

Filter Selection: SelectKBest

from sklearn.datasets import load_breast_cancer
from sklearn.feature_selection import SelectKBest, f_classif, chi2, mutual_info_classif
from sklearn.model_selection import train_test_split
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import accuracy_score

# Loading data
X, y = load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42
)

# Selecting the k best features with ANOVA F-test
selector_f = SelectKBest(score_func=f_classif, k=10)
X_train_selected = selector_f.fit_transform(X_train, y_train)
X_test_selected = selector_f.transform(X_test)

# Retrieving scores and p-values
scores = selector_f.scores_
pvalues = selector_f.pvalues_

print(f"F-scores of the 10 best features: {scores[:10]}")
print(f"Corresponding p-values: {pvalues[:10]}")

# Training and evaluation
rf = RandomForestClassifier(n_estimators=100, random_state=42)
rf.fit(X_train_selected, y_train)
acc = accuracy_score(y_test, rf.predict(X_test_selected))
print(f"Accuracy with SelectKBest (f_classif, k=10): {acc:.4f}")

You can replace f_classif with chi2 for non-negative categorical data, or with mutual_info_classif to capture non-linear relationships. For a regression problem, you would use f_regression or mutual_info_regression.

Wrapper Selection: RFE with RandomForest

from sklearn.feature_selection import RFE
from sklearn.ensemble import RandomForestClassifier

# RFE with a RandomForest classifier
rf_full = RandomForestClassifier(n_estimators=200, random_state=42)
rfe = RFE(estimator=rf_full, n_features_to_select=10, step=2)
rfe.fit(X_train, y_train)

# Selected features
selected_mask = rfe.support_
selected_indices = [i for i, sel in enumerate(selected_mask) if sel]
selected_features = load_breast_cancer().feature_names
print(f"Features retained by RFE: {[selected_features[i] for i in selected_indices]}")

# Evaluation
X_train_rfe = rfe.transform(X_train)
X_test_rfe = rfe.transform(X_test)
acc_rfe = accuracy_score(y_test, rfe.predict(X_test_rfe))
print(f"Accuracy with RFE (RandomForest, 10 features): {acc_rfe:.4f}")

The step=2 parameter means 2 features are eliminated at each iteration. You can also use step=0.1 to eliminate 10% of the remaining features each round — more gradual, but slower.

Categorical Encoding: OneHotEncoder and TargetEncoder

from sklearn.compose import ColumnTransformer
from sklearn.preprocessing import OneHotEncoder
from sklearn.impute import SimpleImputer

# Example with data containing categorical variables
import pandas as pd
import numpy as np

df = pd.DataFrame({
    'age': [25, 34, 45, 52, 29],
    'city': ['Paris', 'Lyon', 'Paris', 'Marseille', 'Lyon'],
    'education_level': ['Bachelor', 'Master', 'PhD', 'Bachelor', 'Master'],
    'salary': [35000, 42000, 55000, 38000, 47000]
})

# OneHotEncoder for nominal variables
onehot = OneHotEncoder(sparse_output=False, handle_unknown='ignore')
encoded = onehot.fit_transform(df[['city', 'education_level']])
print(f"Variables after one-hot encoding: {onehot.get_feature_names_out().tolist()}")

# ColumnTransformer to apply different treatments per column type
preprocessor = ColumnTransformer(
    transformers=[
        ('num', SimpleImputer(strategy='median'), ['age']),
        ('cat', OneHotEncoder(sparse_output=False, handle_unknown='ignore'), ['city', 'education_level'])
    ],
    remainder='drop'
)

X_processed = preprocessor.fit_transform(df)
print(f"Shape after preprocessing: {X_processed.shape}")

For TargetEncoder (encoding by the mean of the target per category), use the category_encoders library:

import category_encoders as ce

# TargetEncoder: encodes each category by the mean of the target
target_enc = ce.TargetEncoder(cols=['city', 'education_level'])
X_train_encoded = target_enc.fit_transform(X_train_df, y_train)
X_test_encoded = target_enc.transform(X_test_df)

TargetEncoder is particularly useful for high-cardinality categorical variables (more than 50 categories), where OneHotEncoder would produce a dimensional explosion.

Creating Interactions: PolynomialFeatures

from sklearn.preprocessing import PolynomialFeatures

# Creating polynomial features (interactions and powers)
poly = PolynomialFeatures(degree=2, include_bias=False, interaction_only=False)
X_poly = poly.fit_transform(X_train_selected[:, :5])
print(f"Shape after polynomial interaction: {X_poly.shape}")
print(f"New features created: {poly.get_feature_names_out().tolist()[:10]}")

# interaction_only=True creates only cross-products (x1*x2), not squares (x1²)
poly_interactions = PolynomialFeatures(degree=2, include_bias=False, interaction_only=True)
X_inter = poly_interactions.fit_transform(X_train_selected[:, :5])
print(f"Interactions only: {X_inter.shape}")

Polynomial interactions allow the linear model to capture non-linear relationships. Be careful: the number of features grows combinatorially — with 10 variables and degree 2, you go from 10 to 54 features (10² + combinations). This is precisely the curse of dimensionality in action.

Complete and Reproducible Pipeline

from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression

# Complete pipeline: preprocessing → selection → scaling → classification
pipeline = Pipeline([
    ('imputation', SimpleImputer(strategy='median')),
    ('selection', SelectKBest(score_func=f_classif, k=15)),
    ('scaling', StandardScaler()),
    ('classification', LogisticRegression(
        penalty='l1', solver='liblinear', C=1.0, max_iter=1000
    ))
])

pipeline.fit(X_train, y_train)
acc_pipeline = accuracy_score(y_test, pipeline.predict(X_test))
print(f"Accuracy of the complete pipeline: {acc_pipeline:.4f}")

# Which features were retained?
selected_indices = pipeline.named_steps['selection'].get_support()
feature_names = load_breast_cancer().feature_names
selected_names = [name for name, sel in zip(feature_names, selected_indices) if sel]
print(f"Final pipeline features: {selected_names}")

The Pipeline ensures that all transformations are applied in the correct order and prevents data leakage: the fit of SelectKBest only sees the training data, never the test data.

Critical Hyperparameters of Feature Engineering

k (number of features to retain)

This is the most influential hyperparameter. Too few features (k < 5): the model underfits and lacks information. Too many features (k > 50): the risk of overfitting increases, and computation time explodes. The recommended rule of thumb is to start with k between 10 and 30, then adjust via cross-validation.

threshold_variance (variance threshold)

Variance-based selection eliminates features whose variance is below a threshold. By default in scikit-learn, VarianceThreshold(threshold=0.0) only removes constant features. A threshold of 0.01 to 0.05 is reasonable for eliminating quasi-constant variables that provide no discriminative information.

encoding_strategy (encoding strategy)

The choice of encoding method directly impacts performance:

• OneHotEncoder: ideal for nominal variables with low cardinality (< 20 categories). Creates one binary column per category.
• OrdinalEncoder: for ordinal variables (e.g., “small”, “medium”, “large”). Preserves the order.
• TargetEncoder: for high-cardinality variables (> 50 categories). Encodes each category by the mean of the target. Beware of data leakage risk: use with cross-validation.
• BinaryEncoder / HashingEncoder: tradeoffs for intermediate cardinalities. Converts categories to binary or hashed representation.

regularization_alpha (regularization coefficient)

In embedded methods (Lasso, ElasticNet), the C parameter (inverse of α in scikit-learn) controls the penalty intensity:

• C = 10 (low α): gentle penalty, nearly all variables are retained.
• C = 1.0: standard balance.
• C = 0.1 (high α): severe penalty, only the most important features survive.
• C = 0.01: very strong penalty, drastic selection.

The optimal choice is determined by cross-validation on a grid of logarithmic values: C ∈ [0.001, 0.01, 0.1, 1.0, 10.0].

Advantages and Limitations of Feature Engineering

Advantages

1. Significant performance improvement: well-executed Feature Engineering can increase accuracy by 10 to 30% compared to using raw data. It is often the most effective lever before even changing the model.
2. Reduced training time: fewer features means fewer computations. On massive datasets, reducing from 1,000 to 50 features can divide training time by 20.
3. Better interpretability: a model with 10 well-chosen features is much easier to explain to non-technical stakeholders than a model with 500 features.
4. Reduced overfitting: by eliminating noise and redundant variables, the model generalizes better to unseen data.
5. Domain expertise adaptability: Feature Engineering allows injecting expert knowledge into the model (e.g., creating a debt-to-income ratio in credit scoring).

Limitations

1. Time and expertise cost: quality Feature Engineering requires deep domain understanding, meticulous data exploration, and numerous iterations. It is often the most time-consuming step of an ML project.
2. Risk of data leakage: if transformations (normalization, TargetEncoder, etc.) are computed on the entire dataset before the train/test split, the model “cheats” by accessing test information. Using Pipelines is essential to avoid this pitfall.
3. Loss of interpretability with PCA: principal components are obscure linear combinations of the original variables. We no longer know which variables truly contribute.
4. Over-optimization on the validation set: by testing too many feature combinations, we risk adapting to the specific noise of the validation set rather than the general signal.
5. Temporal instability: the relationships between features and target can evolve over time (concept drift). Feature Engineering that is optimal today may become suboptimal in six months.

4 Concrete Use Cases of Feature Engineering

Use Case 1: Credit Scoring — Financial Feature Engineering

In the banking credit domain, raw data includes income, loan amount, job seniority, number of active loans, etc. Feature Engineering creates essential business ratios:

• Debt-to-income ratio = loan monthly payment / monthly income
• Professional stability = years of seniority / age
• Credit diversification = number of different types of credit
• Combined risk score = weighting of past payment incidents

Lasso selection (L1 regularization) then automatically eliminates redundant variables. Typical result: the model goes from 45 raw variables to 12 selected features, with a 15% reduction in the false positive rate.

Use Case 2: Fraud Detection — Temporal Interactions

In bank card fraud detection, raw data consists of transactions with their amount, location, time, and merchant type. Feature Engineering creates temporal and contextual features:

• Transaction frequency: number of transactions in the last 24 hours
• Geographic gap: distance between the current transaction and the previous one
• Amount deviation: difference between the current amount and the average of the last 30 days
• Unusual time: does the transaction occur at a time when the client doesn’t usually transact?
• New merchant: is the merchant unknown in the client’s history?

Mutual information (mutual_info_classif) is particularly effective in this context, because the relationship between these features and fraud is highly non-linear.

Use Case 3: Real Estate Pricing — Geographic Variables and Polynomial Features

To predict the price of a property, raw data includes area, number of rooms, year of construction, neighborhood, and GPS coordinates. Feature Engineering significantly enriches the dataset:

• Price per m² in the neighborhood: average prices of recent sales in the same district
• Distance to transit: kilometers to the nearest metro station
• Renovation index = (current year – year of construction) with bounds
• Polynomial interactions: area × number of rooms (captures average room size)
• Ordinal encoding of standing: “economy”, “standard”, “premium”, “luxury”

A degree-2 PolynomialFeatures on numerical variables, combined with a TargetEncoder for districts, captures complex interactions between location and property characteristics.

Use Case 4: Medical Diagnosis — Biomarker Selection

In AI-assisted medical diagnosis, we have hundreds of blood biomarkers, vital indicators, and demographic data. The challenge is to identify the most discriminating combinations for a given pathology.

Feature Engineering creates composite indices:

• Inflammatory index = combination of CRP, white blood cells, and sedimentation rate
• Metabolic score = glucose + HbA1c + insulinemia
• Hepatic ratio = ASAT / ALAT
• Frailty indicator = albumin + creatinine + body mass index

The RFE method with a Gradient Boosting classifier iteratively selects the 20 most informative biomarkers from the initial 200. This embedded approach accounts for complex interactions between biomarkers that univariate filters would miss.

See Also

• Exploring Dynamic Polynomials with Python: Complete Guide and Practical Applications
• Mastering the Floor Operator in Python: The Revolution of Arithmetic Calculations