Features & feature extraction
A model never sees "a customer," "a sentence," or "a photo." It sees a row of
numbers. A feature is one of those numbers, one measurable property of the
thing. The row is a feature vector, and the whole table is the feature
matrix X of shape (n_samples, n_features).
Feature extraction is the craft of turning raw, messy reality into that clean numeric table. It is, in practice, where most of the real work, and most of the accuracy, lives. "Garbage in, garbage out" is never more true than here.
Don't be confused: the three "feature ___" terms.
- Feature extraction, create numeric features from raw data (pixels → edges, text → TF-IDF, audio → spectrogram). Changes the representation.
- Feature engineering, hand-craft new features from existing ones (
price_per_sqft = price / area). Uses domain knowledge.- Feature selection, pick a subset of existing features and drop the rest (remove redundant or useless columns). Reduces dimensionality.
Deep learning's headline trick is that it does feature extraction for you, the hidden layers learn features instead of you crafting them. That's what "representation learning" means (Chapter 14).
All code below is in code/features.py (listed in full at the end of this chapter) and prints exactly
what's shown.
1. Numeric features, and why you must scale them
Raw numbers come on wildly different scales: age ∈ [0, 100], income ∈ [0,
1,000,000]. Many models (anything using distances, gradients, or regularization)
will let the big-scale feature dominate purely because its numbers are bigger.
Standardization fixes this: per column, subtract the mean and divide by the
standard deviation, giving every feature mean 0 and std 1.
$$ z = \frac{x - \mu}{\sigma} $$
import numpy as np
def standardize(X):
X = np.asarray(X, dtype=float)
return (X - X.mean(axis=0)) / (X.std(axis=0) + 1e-12)
age_income = np.array([[25, 30000.0],
[40, 60000.0],
[55, 90000.0]])
print(standardize(age_income))
Output:
[[-1.225 -1.225]
[ 0. 0. ]
[ 1.225 1.225]]
Now age and income are on the same footing, both range over the same
small spread, so neither bullies the other. Note we used axis=0 (per column /
per feature), exactly as Chapter 2 warned.
Don't be confused: normalization vs standardization. People use these loosely, but the usual meanings are:
- Standardization → z-score: mean 0, std 1. Unbounded output. The default for tabular features.
- Min-max normalization → rescale to a fixed range like [0, 1].
- L2 normalization → scale each row to unit length (for cosine similarity; Chapter 4).
They answer different needs. When someone says "normalize," ask "to unit length, to [0,1], or to mean-0/std-1?"
The cardinal rule: fit the scaler on the training set only, then apply those same numbers to validation/test. Computing the mean over all data leaks test information into training, a classic, silent mistake.
2. Categorical features: one-hot encoding
A category like color ∈ {red, green, blue} is not a number. Encoding it as
red=0, green=1, blue=2 is a trap: it tells the model blue > red and green is
"between" them, which is nonsense. One-hot encoding instead gives each category
its own 0/1 column:
from collections import Counter
import numpy as np
def one_hot_encode(values):
vocab = sorted(set(values))
index = {v: i for i, v in enumerate(vocab)}
out = np.zeros((len(values), len(vocab)))
for row, v in enumerate(values):
out[row, index[v]] = 1.0
return out, vocab
M, vocab = one_hot_encode(["red", "blue", "red", "green"])
print("vocab:", vocab)
print(M)
Output:
vocab: ['blue', 'green', 'red']
[[0. 0. 1.]
[1. 0. 0.]
[0. 0. 1.]
[0. 1. 0.]]
Each row has a single 1 marking its category. No false ordering, no fake
distances.
When not to one-hot: if a category has thousands of values (user IDs, product IDs), one-hot makes an absurdly wide, sparse matrix. There you use an embedding (a learned, dense vector per category), which is exactly the bridge to neural recommenders (Chapter 14). And ordinal categories with a real order (
small < medium < large) can be integer-coded on purpose.
3. Text features: bag-of-words and TF-IDF
Text is the classic feature-extraction problem: variable-length strings must become fixed-length vectors. The oldest, still-useful idea is the bag of words, count how often each vocabulary word appears, ignoring order.
docs = ["the cat sat",
"the dog sat",
"the cat and the dog"]
# vocab (sorted): ['and', 'cat', 'dog', 'sat', 'the']
The count matrix (rows = documents, columns = words):
[[0. 1. 0. 1. 1.] # "the cat sat"
[0. 0. 1. 1. 1.] # "the dog sat"
[1. 1. 1. 0. 2.]] # "the cat and the dog" -> "the" appears twice
The problem: the appears everywhere, so it dominates the counts while carrying
no discriminating signal. TF-IDF (Term Frequency × Inverse Document
Frequency) fixes this by down-weighting words that appear in many documents:
$$ \text{tfidf}(t, d) = \underbrace{\text{count}(t, d)}{\text{term frequency}} \times \underbrace{\left(\log\frac{1 + N}{1 + \text{df}(t)} + 1\right)}{\text{inverse doc. frequency}} $$
where $N$ is the number of documents and $\text{df}(t)$ is how many documents contain term $t$. Rows are then L2-normalized. The result:
[[0. 0.62 0. 0.62 0.481]
[0. 0. 0.62 0.62 0.481]
[0.531 0.404 0.404 0. 0.627]]
Compare the with cat/dog in the first two rows: the counts are equal, but
the scores 0.481 against their 0.62, because the appears in every
document so its IDF is small. (In row 3 the reaches 0.627 only because it
occurs twice there; per occurrence it is still discounted.) TF-IDF is the workhorse behind classic
search ranking and a strong baseline for text classification, and it's the
direct ancestor of the dense embeddings used today.
The feature-extraction checklist
For any new dataset, walk these questions:
- Per column, what type is it? Numeric, categorical, ordinal, text, date, image, ID?
- Numeric → scale it (standardize by default).
- Categorical → one-hot if low-cardinality, embed if high-cardinality.
- Text → TF-IDF for a baseline, sentence-embeddings for semantics.
- Dates → extract
day_of_week,month,is_weekend, time-since-event. - Missing values → decide: drop, fill with mean/median, or add an "is-missing" flag (often the missingness itself is signal).
- Fit transforms on train only, apply to test. Always.
Full source: code/features.py
The complete, runnable program, exactly as verified for this chapter:
"""
features.py — feature extraction from scratch, in pure NumPy + stdlib.
Models never see "a customer" or "a sentence" — they see a row of numbers called
a *feature vector*. This file shows the three workhorse encoders that turn raw
data into those rows:
1. numeric scaling (standardize / min-max)
2. categorical encoding (one-hot)
3. text vectorization (bag-of-words and TF-IDF), built by hand
Run it: python features.py
Only dependency: numpy.
"""
from __future__ import annotations
import math
from collections import Counter
import numpy as np
# --------------------------------------------------------------------------- #
# 1. Numeric features #
# --------------------------------------------------------------------------- #
def standardize(X):
"""z-score each column: (x - mean) / std. Mean 0, std 1 per feature."""
X = np.asarray(X, dtype=float)
return (X - X.mean(axis=0)) / (X.std(axis=0) + 1e-12)
# --------------------------------------------------------------------------- #
# 2. Categorical features #
# --------------------------------------------------------------------------- #
def one_hot_encode(values):
"""List of categories -> (matrix, vocabulary). Order of first appearance."""
vocab = sorted(set(values))
index = {v: i for i, v in enumerate(vocab)}
out = np.zeros((len(values), len(vocab)))
for row, v in enumerate(values):
out[row, index[v]] = 1.0
return out, vocab
# --------------------------------------------------------------------------- #
# 3. Text features: bag-of-words and TF-IDF, by hand #
# --------------------------------------------------------------------------- #
def tokenize(doc):
"""Lowercase and split on whitespace. (Real tokenizers do much more.)"""
return doc.lower().split()
def build_vocab(docs):
vocab = sorted({tok for doc in docs for tok in tokenize(doc)})
return {tok: i for i, tok in enumerate(vocab)}
def bag_of_words(docs, vocab):
"""Count matrix: rows = documents, columns = vocabulary words."""
X = np.zeros((len(docs), len(vocab)))
for r, doc in enumerate(docs):
for tok, count in Counter(tokenize(doc)).items():
if tok in vocab:
X[r, vocab[tok]] = count
return X
def tfidf(docs, vocab):
"""TF-IDF: down-weight words that appear in many documents.
tf = term count in the doc
idf = log( (1 + N) / (1 + df) ) + 1 (smoothed, sklearn-style)
Rows are then L2-normalized, the usual convention.
"""
counts = bag_of_words(docs, vocab)
n_docs = len(docs)
df = np.count_nonzero(counts, axis=0) # docs containing each word
idf = np.log((1 + n_docs) / (1 + df)) + 1.0
tfidf = counts * idf # broadcast idf over columns
norms = np.linalg.norm(tfidf, axis=1, keepdims=True)
return tfidf / (norms + 1e-12)
# --------------------------------------------------------------------------- #
# Demo #
# --------------------------------------------------------------------------- #
def _demo():
np.set_printoptions(precision=3, suppress=True)
print("== 1. Numeric scaling ==")
age_income = np.array([[25, 30000.0],
[40, 60000.0],
[55, 90000.0]])
print("raw:\n", age_income)
print("standardized (note both columns now comparable):\n",
standardize(age_income))
print("\n== 2. One-hot encoding ==")
colors = ["red", "blue", "red", "green"]
M, vocab = one_hot_encode(colors)
print("categories:", colors)
print("vocab :", vocab)
print("one-hot :\n", M)
print("\n== 3. Text: bag-of-words ==")
docs = ["the cat sat",
"the dog sat",
"the cat and the dog"]
vocab = build_vocab(docs)
print("vocab:", list(vocab.keys()))
print("counts:\n", bag_of_words(docs, vocab))
print("\n== 3b. Text: TF-IDF (L2-normalized rows) ==")
T = tfidf(docs, vocab)
print("tfidf:\n", T)
print("note 'the' (in every doc) gets the smallest weight per row")
if __name__ == "__main__":
_demo()
The takeaway
A feature is a number describing one property; feature extraction builds the numeric table the model trains on. Scale numerics, one-hot small categories, embed big ones, TF-IDF text, and never let test data leak into your transforms. Now that we have feature vectors, the next question is the one that powers search, recommendation, and clustering: how do we measure that two vectors are similar? 👉