NLP Pipeline: Complete Guide — Natural Language Processing

Summary

The NLP Pipeline (Natural Language Processing) refers to the set of successive steps that allow a computer to understand, analyze, and generate human language. A typical NLP pipeline includes tokenization, text cleaning, lemmatization or stemming, part-of-speech tagging (POS tagging), named entity extraction (NER), vectorization (TF-IDF, word embeddings), and finally text classification or generation. This guide details each component, the underlying mathematical principles, and provides concrete implementations in Python using the spaCy, NLTK, and scikit-learn libraries. Whether you are a beginner or an advanced practitioner, this tutorial will give you a complete and practical overview of the natural language processing pipeline.

Mathematical Principle

Natural language processing is based on several fundamental mathematical foundations that are essential to understand in order to design robust and efficient systems.

TF-IDF (Term Frequency — Inverse Document Frequency)

The TF-IDF metric measures the importance of a term in a document relative to a corpus. It is broken down into two parts:

• TF(t, d) (term frequency): number of occurrences of term t in document d, normalized by the document length.

tf(t, d) = (number of occurrences of t in d) / (total number of words in d)

• IDF(t) (inverse document frequency): measures how rare the term is in the corpus.

idf(t) = log(N / df(t))

where N represents the total number of documents in the corpus and df(t) the number of documents containing the term t.

The complete formula is written:

tf-idf(t, d) = tf(t, d) × log(N / df(t))

The more frequently a term appears in a document while being rare in the overall corpus, the higher its TF-IDF score. This property makes it an excellent criterion for identifying the keywords of a document.

N-grams

An n-gram is a sequence of k consecutive tokens extracted from a text. Unigrams (k=1) correspond to individual words, bigrams (k=2) to pairs of adjacent words, and trigrams (k=3) to triplets. N-gram modeling allows capturing local context:

P(w_i | w_1...w_{i-1}) ≈ P(w_i | w_{i-k+1}...w_{i-1})

This Markov approximation assumes that the current word depends only on the k-1 previous words, which considerably simplifies probability calculations.

Hidden Markov Models for POS Tagging

Hidden Markov Models (HMM) are one of the first probabilistic approaches for grammatical tagging. The core idea is as follows: grammatical tags (noun, verb, adjective…) form a hidden sequence of states, while the observed words constitute the emissions of these states.

P(t_1...T, w_1...T) = Π P(t_i | t_{i-1}) × P(w_i | t_i)

Viterbi decoding makes it possible to find the most likely tag sequence given the observed word sequence. Each grammatical state emits a word with a certain probability, and the transition between states also follows a probabilistic law. The Viterbi algorithm efficiently explores the space of possible sequences by exploiting the graph structure of the model.

Conditional Random Fields (CRF) for NER

For named entity recognition (NER), CRFs offer a more powerful discriminative approach than HMMs. Unlike generative models, CRFs directly model:

P(y | x) = (1/Z) × exp(Σ λ_k × f_k(y, x))

where f_k are characteristic functions (e.g.: “the word is capitalized”, “the previous word is a verb”), λ_k are weights learned from training data, and Z is the normalization constant ensuring that the probability distribution sums to one.

Language Models

A language model assigns a probability to any sequence of words:

P(w_1...w_n) = Π_{i=1}^{n} P(w_i | w_1...w_{i-1})

Traditional models used n-grams with Kneser-Ney smoothing to avoid zero probabilities on unobserved combinations. Modern models (transformers, recurrent neural networks) capture long-range dependencies well beyond the horizon of classical n-grams, enabling much superior contextual understanding.

Intuition: Understanding NLP Step by Step

Imagine that natural language is a sentence that a translator must analyze word by word before rendering it in another language. The NLP Pipeline does similar work: it decomposes human language into a series of increasingly fine and precise steps.

1. Tokenization — We start by breaking the text into elementary units called tokens. A token can be a word, a punctuation mark, or even a sub-part of a word (like “dé-” and “composition”). It is the equivalent of separating the bricks of a wall to examine them individually.
2. Cleaning and normalization — We remove extraneous characters, convert to lowercase, and remove stop words (articles, prepositions) that do not provide discriminating meaning. This step reduces noise and focuses the analysis on the substantive content.
3. Lemmatization / Stemming — Each word is reduced to its canonical form. “mangeais”, “mangerons”, “mangé” all become “manger” after lemmatization. This step considerably reduces the dimensionality of the term space and improves the statistical consistency of the model.
4. POS Tagging — We identify the grammatical category of each token: proper noun, common noun, verb, adjective, adverb, preposition, conjunction… This information valuable for any subsequent syntactic analysis and allows distinguishing grammatical homonyms.
5. NER (Named Entity Recognition) — We detect and classify named entities: people, organizations, locations, dates, monetary amounts, etc. This step transforms raw text into structured information usable by a knowledge base.
6. Vectorization — The cleaned and annotated text is transformed into a numerical representation (bag of words, TF-IDF, embeddings) usable by machine learning algorithms. This is the bridge between human language and the mathematics of statistical models.
7. Classification / Generation — Finally, the text can be classified (sentiment, spam, thematic category) or used to generate new responses. This is the culmination of the pipeline, where the algorithm produces an operational result.

This sequential approach is powerful because each step enriches the previous one: POS tagging benefits from lemmatization, NER benefits from POS tagging, and the final classification exploits all accumulated annotations. It is a value chain where each link amplifies the next.

Python Implementation

1. Complete pipeline with spaCy

spaCy is the reference library for language processing in production. Its pipeline API groups all steps into a single call:

# -*- coding: utf-8 -*-
import spacy

# Load the French model
nlp = spacy.load("fr_core_news_sm")

# The NLP pipeline in one line
texte = "Emmanuel Macron a rencontré Angela Merkel à Berlin le 14 juillet 2025."
doc = nlp(texte)

# Tokenization
print("=== Tokens ===")
for token in doc:
    print(f"{token.text:15s} | lemme: {token.lemma_:15s} | POS: {token.pos_:10s} | forme: {token.tag_}")

# Named entities (NER)
print("\n=== Entités nommées ===")
for ent in doc.ents:
    print(f"{ent.text:20s} → {ent.label_} ({spacy.explain(ent.label_)})")

# Syntactic dependencies
print("\n=== Dépendances syntaxiques ===")
for token in doc:
    print(f"{token.text:15s} → {token.dep_:10s} → {token.head.text}")

# Display the pipeline
print(f"\nPipeline actif : {nlp.pipe_names}")

The fr_core_news_sm model automatically includes the following components: tok2vec, morphologizer, parser, senter, lemmatizer, ner. Each component enriches the Doc object with additional annotations accessible via token attributes. To process thousands of documents, use nlp.pipe(liste_textes) which automatically parallelizes the processing.

2. TF-IDF with NLTK

To manually compute TF-IDF and understand each step of the process:

# -*- coding: utf-8 -*-
import math

corpus = [
    "le chat mange du poisson dans le jardin",
    "le chien court dans le parc",
    "le poisson est un animal aquatique",
]

# Tokenization and counting
def compute_tf(tokens):
    tf = {}
    total = len(tokens)
    for t in tokens:
        tf[t] = tf.get(t, 0) + 1.0 / total
    return tf

def compute_idf(corpus_tokens):
    N = len(corpus_tokens)
    df = {}
    for tokens in corpus_tokens:
        for t in set(tokens):
            df[t] = df.get(t, 0) + 1
    return {t: math.log(N / count) for t, count in df.items()}

# Prepare the tokenized corpus
tokenized = [doc.split() for doc in corpus]
idf_scores = compute_idf(tokenized)

# Compute TF-IDF for each document
for i, tokens in enumerate(tokenized):
    tf = compute_tf(tokens)
    tfidf = {t: freq * idf_scores.get(t, 0) for t, freq in tf.items()}
    print(f"\nDocument {i+1} — TF-IDF")
    for mot, score in sorted(tfidf.items(), key=lambda x: -x[1]):
        print(f"  {mot:12s} : {score:.4f}")

This manual implementation reveals exactly how TF-IDF works: words frequent in a document but rare in the corpus (like “aquatique”) get high scores, while ubiquitous words (“le”, “dans”) are penalized by IDF. It is this duality that gives the metric its full power.

3. Vectorization and classification with scikit-learn

# -*- coding: utf-8 -*-
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.naive_bayes import MultinomialNB
from sklearn.pipeline import Pipeline
from sklearn.model_selection import train_test_split
from sklearn.metrics import classification_report

# Training corpus
documents = [
    "Ce film est absolument magnifique, une œuvre magistrale",
    "Un chef-d'œuvre du cinéma moderne, bouleversant et profond",
    "Horrible, ennuyeux, je n'ai rien compris à ce navet",
    "Une catastrophe totale, le pire film de l'année",
    "Performance actoriale remarquable, scénario brillant",
    "Insipide et prétentieux, une perte de temps",
]
labels = ["positif", "positif", "négatif", "négatif", "positif", "négatif"]

# Complete text classification pipeline
pipeline = Pipeline([
    ("tfidf", TfidfVectorizer(
        max_features=5000,
        ngram_range=(1, 2),
        min_df=1,
        stop_words="english"
    )),
    ("clf", MultinomialNB(alpha=0.5)),
])

# Training
pipeline.fit(documents, labels)

# Prediction
nouveaux_textes = [
    "Un film extraordinaire, j'ai adoré chaque minute",
    "Le scénario est faible et les acteurs sont mauvais",
]
predictions = pipeline.predict(nouveaux_textes)

for texte, pred in zip(nouveaux_textes, predictions):
    probas = pipeline.predict_proba([texte]).flatten()
    print(f"\n'{texte}'")
    print(f"  → {pred} (confiance: {max(probas):.2%})")

# Display the most important features
feature_names = pipeline.named_steps["tfidf"].get_feature_names_out()
coef = pipeline.named_steps["clf"].feature_log_prob_
print("\n=== Top 10 mots positifs ===")
top_pos = sorted(zip(feature_names, coef[1]), key=lambda x: -x[1])[:10]
for mot, score in top_pos:
    print(f"  {mot:15s} : {score:.4f}")

This pipeline demonstrates how scikit-learn chains TF-IDF vectorization and naive Bayes classification in just a few lines. The ngram_range=(1, 2) parameter captures not only individual words but also compound expressions, which significantly improves accuracy. Training is instantaneous, even on modestly sized corpora.

Critical Hyperparameters

The choice of hyperparameters directly influences the performance of your NLP pipeline. Here are the most important ones to master:

max_features

Controls the maximum size of the vocabulary retained. Limiting to 10,000–50,000 features reduces memory and speeds up training, but risks eliminating rare yet informative terms. In most applications, max_features=10000 offers a good compromise between completeness and computational efficiency.

vectorizer = TfidfVectorizer(max_features=10000)

ngram_range

Defines the range of n-gram sizes. (1, 1) uses only unigrams. (1, 2) includes unigrams and bigrams. (1, 3) adds trigrams, capturing even more complex expressions.

# Capture "pas mal" and "très bon" as semantic units
vectorizer = TfidfVectorizer(ngram_range=(1, 2))

Bigrams are particularly useful for sentiment detection and opinion analysis, where negation (“pas bon” / “not good”) radically changes the meaning compared to individual words (“pas” + “not” and “bon” + “good”) treated separately.

min_df and max_df

• min_df: minimum frequency (as a count or proportion) for a term to be included. min_df=2 ignores terms appearing in only one document, eliminating hapax legomena (unique words) that are often noisy and not generalizable.
• max_df: maximum frequency. max_df=0.95 ignores terms present in more than 95% of documents, too common to be discriminating in classification.

vectorizer = TfidfVectorizer(min_df=2, max_df=0.95)

stop_words

List of words to exclude from the vocabulary: articles, prepositions, conjunctions, and other function words that do not contribute discriminating meaning. Can be a custom list or the built-in lists from NLTK/spaCy.

from nltk.corpus import stopwords
french_stops = set(stopwords.words('french'))
french_stops.update(["faire", "dire", "aller"])  # custom stop words

vectorizer = TfidfVectorizer(stop_words=list(french_stops))

alpha (Naive Bayes Regularization)

The alpha parameter applies Laplace smoothing to avoid zero probabilities. A low value (0.01–0.5) suits large data-rich corpora, while a higher value (1.0–2.0) is preferable for small datasets where the risk of overfitting is greater.

Advantages and Limitations

Advantages of the classic NLP Pipeline

• Modularity — Each component is independent and replaceable. You can improve POS tagging without touching vectorization, or replace the classifier without modifying preprocessing. This flexibility is invaluable in production.
• Transparency — Unlike black-box models (deep transformers), a traditional pipeline is fully interpretable: you know exactly why a document is classified in a given way, which words contribute most to the decision, and how each step transforms the data.
• Computational efficiency — A TF-IDF + Naive Bayes pipeline trains in seconds on millions of documents, compared to hours or days for a BERT-type model. Prediction takes a few milliseconds per text.
• Low data requirements — Classical methods work correctly with a few hundred labeled examples, whereas neural approaches require tens of thousands of quality annotations.
• Fine-grained control — The engineer controls every step: choice of stop words, n-gram size, lemmatization algorithm, vectorization method. This control allows precise tuning to the business domain.

Limitations

• Lack of deep context — TF-IDF treats each word independently of its order and semantic context. “Le chat mange la souris” and “La souris mange le chat” produce strictly identical TF-IDF scores, even though the meaning is radically different.
• No semantic understanding — Traditional methods do not capture synonymy (“voiture” and “automobile” are distinct dimensions in the vector space) or polysemy (“banque” financial vs “banque” riverbank), which limits the quality of text representation.
• Fragility in the face of noise — Spelling mistakes, SMS language, or neologisms create new unknown tokens that the model cannot handle, unlike contextual embeddings that can infer meaning from the surrounding context.
• Language dependency — A pipeline trained in French does not work in Arabic or Chinese without complete reconfiguration of the components (different tokenization, different language models, different morphological characteristics).
• Performance ceiling — Even with optimal hyperparameter tuning, classical pipelines reach a precision ceiling that modern neural architectures (transformers, pretrained language models) systematically surpass on complex tasks.

4 Concrete Use Cases

1. Sentiment Analysis on Customer Reviews

An e-commerce site wants to automatically classify customer reviews into positive, neutral, and negative categories to produce real-time dashboards. The classic NLP pipeline is ideal for this task:

from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.pipeline import Pipeline

# Thousands of collected reviews
avis = ["Produit excellent, livraison rapide !", ...]
labels_sentiments = ["positif", "neutre", "négatif", ...]

pipeline = Pipeline([
    ("tfidf", TfidfVectorizer(ngram_range=(1, 2), max_features=20000)),
    ("clf", LogisticRegression(max_iter=1000)),
])
pipeline.fit(avis, labels_sentiments)

# New review
nouvel_avis = "Déçu par la qualité, rien à voir avec la description"
print(pipeline.predict([nouvel_avis]))

Why a classic pipeline? Customer reviews contain recurring expressions (“très bien” / “very well”, “pas top” / “not great”) that bigrams capture perfectly. Training speed allows daily model re-evaluations. Logistic regression also provides interpretable probabilities that business teams can directly leverage to prioritize corrective actions.

2. Spam and Phishing Detection

Anti-spam filters have historically been based on NLP pipelines. A suspicious email contains precise text signals that the model learns to recognize:

# Signals detected by TF-IDF
# - Excessive urgency: "urgent", "immédiately", "action required"
# - Financial promises: "gagner" (win), "million", "gratuitement" (for free)
# - Suspicious links detected by regex patterns in the email body
# - Unusual sender combined with misleading content

Why a classic pipeline? Spam detection requires real-time processing (a few milliseconds per email). A TF-IDF + Naive Bayes model is fast enough for this constraint while achieving accuracy above 95%. Additionally, the model’s transparency allows security teams to understand why an email was blocked and manually adjust rules if necessary.

3. Information Extraction from Legal Documents

Law firms use NLP to automatically identify clauses, dates, amounts, and involved parties in contracts. This automation saves hundreds of hours of manual work:

import spacy

nlp = spacy.load("fr_core_news_lg")

contrat = """Le présent contrat est conclu entre la société Dupont SARL,
immatriculée au RCS de Paris sous le numéro 123 456 789, et M. Jean Martin,
demeurant au 42 rue de la Paix, 75002 Paris. Le montant du contrat s'élève
à 250 000 euros, payable à compter du 1er janvier 2026."""

doc = nlp(contrat)

# Named entity extraction
entities = [(ent.text, ent.label_) for ent in doc.ents]
for ent_text, ent_label in entities:
    print(f"{ent_text} → {ent_label}")

Typical output: detected entities include company names (ORG), persons (PER), addresses and cities (LOC), amounts (MONEY), and dates (DATE). This structured extraction directly feeds legal databases and contract management tools.

Why a classic pipeline? CRF-based NER is specialized for structured documents and achieves accuracy (F1-score) above 90% on standard entities, with latency well below generative models. For the legal sector, this speed is crucial when processing thousands of pages daily.

4. Automatic Text Summarization (Extractive)

Extracting the most important sentences from an article to produce a concise summary is a classic NLP Pipeline application, particularly useful for competitive intelligence and report summarization:

# -*- coding: utf-8 -*-
from sklearn.feature_extraction.text import TfidfVectorizer
import numpy as np

def resume_extractif(texte, n_phrases=3):
    # Split into sentences
    phrases = [p.strip() for p in texte.split(".") if len(p.strip()) > 20]

    if len(phrases) <= n_phrases:
        return texte

    # TF-IDF on sentences
    vectorizer = TfidfVectorizer(ngram_range=(1, 1), max_features=1000)
    tfidf_matrix = vectorizer.fit_transform(phrases)

    # Score of each sentence: sum of TF-IDF scores normalized
    scores = np.array(tfidf_matrix.sum(axis=1)).flatten()

    # Selection of the top n sentences
    indices_top = np.argsort(scores)[-n_phrases:][::-1]
    resume = ". ".join([phrases[i] for i in sorted(indices_top)])

    return resume

article = """Le traitement automatique du langage naturel, aussi appelé NLP,
est un domaine de l'intelligence artificielle qui se concentre sur l'interaction
entre les ordinateurs et le langage humain. Les applications du NLP sont
nombreuses : traduction automatique, analyse de sentiments, assistants vocaux,
et bien d'autres encore. Les méthodes traditionnelles utilisaient des règles
linguistiques codées à la main, tandis que les approches modernes reposent
sur l'apprentissage profond et des architectures de transformateurs."""

res = resume_extractif(article, n_phrases=2)
print(res)

Why a classical pipeline? Extractive summarization by TF-IDF requires no labeled data (unsupervised approach), works on any thematic domain without retraining, and produces instantly understandable results. It is the ideal method for initial prototyping before investing in more costly neural solutions.

See Also

• Multiplier des chaînes de caractères avec Python
• Maîtrisez l’Art du Reverse Engineering en Python : Guide Complet pour Débutants et Experts