NLP Text Preprocessing: The Complete Guide

Essential Steps:

• Lowercasing: Convert all text to lowercase
• Noise Removal: URLs, HTML tags, special characters
• Contraction Expansion: "can't" → "cannot"
• Spelling Correction: Fix common typos

Python Implementation:

import re
from contractions import fix

def clean_text(text):
    # Lowercase
    text = text.lower()
    # Expand contractions
    text = fix(text)
    # Remove URLs
    text = re.sub(r'https?://\S+|www\.\S+', '', text)
    # Remove HTML tags
    text = re.sub(r'<.*?>', '', text)
    # Remove special chars
    text = re.sub(r'[^\w\s]', '', text)
    return text

sample = "Can't wait for the NLP tutorial at https://example.com!"
print(clean_text(sample))  # "cannot wait for the nlp tutorial at"
        

Tokenization Methods:

Method	Description	Best For
Whitespace	Split on spaces	Basic Western languages
Word-based	Punctuation-aware	Most NLP tasks
Subword	Byte Pair Encoding	Transformers, rare words

Advanced Tokenization:

from transformers import AutoTokenizer
import spacy

# Word tokenization
nlp = spacy.load("en_core_web_sm")
doc = nlp("Apple's stock rose 5% yesterday.")
print([token.text for token in doc])  # ["Apple", "'s", "stock", "rose", "5", "%", "yesterday", "."]

# Subword tokenization (BERT)
tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased")
print(tokenizer.tokenize("unhappiness"))  # ["un", "##happi", "##ness"]
 

Advanced Techniques:

Implementation:

from nltk.stem import PorterStemmer, WordNetLemmatizer
import unicodedata

# Stemming
stemmer = PorterStemmer()
print(stemmer.stem("running"))  # "run"

# Lemmatization
lemmatizer = WordNetLemmatizer()
print(lemmatizer.lemmatize("better", pos="a"))  # "good"

# Unicode normalization
text = "Café"
print(unicodedata.normalize("NFKD", text).encode("ascii", "ignore"))  # b"Cafe"
        

Specialized Methods:

• Custom Regex Tokenizers: Domain-specific patterns
• Spelling Correction: SymSpell, Hunspell
• Text Augmentation: Synonym replacement, backtranslation
• Emoji Handling: Conversion to text

Spelling Correction Example:

from symspellpy import SymSpell

sym_spell = SymSpell(max_dictionary_edit_distance=2)
sym_spell.load_dictionary("frequency_dictionary.txt", term_index=0, count_index=1)

input_term = "NLP is awesom"
suggestions = sym_spell.lookup(input_term, verbosity=2)
print(suggestions[0].term)  # "awesome"