NLP Text Preprocessing Utility Package

---

Table of Contents

• Package Overview
• Key Features
• Architecture Diagram
• Quickstart and Setup
• Pipeline Automation Engine
• Complete API Reference
  • 1. General Feature Extraction
  • 2. Text Cleaning and Normalization
  • 3. Linguistic and Morphological Processing
  • 4. Safety, Moderation and Sentiment
  • 5. Visualization Suite
• Code Examples
  • Basic Text Cleaning
  • Pipeline Engine Usage
  • Pandas DataFrame Integration
  • Batch Feature Extraction
  • WordCloud Generation
• Troubleshooting and Resource Setup
• License and Author

---

Package Overview

nlp_text_preprocessing is a production-grade, modular Python library designed to streamline text cleaning, feature extraction, linguistic analysis, content moderation, and visualization for Natural Language Processing (NLP) pipelines and Machine Learning workflows.

Maintained by Uditya Narayan Tiwari.

It provides over 60 single-purpose functions and an extensible Pipeline Engine with built-in defensive guards for None, NaN, and missing values, making it 100% crash-proof when running over Pandas DataFrames or large raw text datasets.

---

Key Features

• Crash-Proof Pipeline: Automatic None/NaN input sanitation across all functions.
• Pipeline Automation Engine: Chain cleaning steps by name or custom functions via Pipeline(['to_lower_case', 'remove_urls', 'lemmatize']).
• Lazy and Fast Model Caching: Efficient lazy-loading for spaCy (en_core_web_sm) and cached NaiveBayes sentiment analyzers.
• Advanced Cleaning: Remove emojis, phone numbers, PII mask, URLs, emails, HTML, retweets, special symbols, and expand contractions/hashtags.
• Feature & Readability Extraction: Extract word counts, sentence counts, readability scores (Flesch Ease), punctuation density, capital ratios, and full batch statistics.
• Linguistic Utilities: Named Entity Recognition (NER), POS tagging, dependency parsing, language detection, keyword extraction, synonyms, antonyms, and n-grams.
• Safety & Moderation: Profanity detection/censoring, toxicity scores, and spam signal detection.
• Visualization: Generate and save custom WordClouds to disk or display interactively.

---

Architecture Diagram

The package integrates spaCy, NLTK, TextBlob, BeautifulSoup4, and WordCloud through a unified API surface:

---

Quickstart and Setup

1. Installation

pip install nlp-text-preprocessing

2. Downloading Required Corpora and Models

To automatically download all required NLTK corpora (stopwords, movie_reviews, brown) and spaCy models (en_core_web_sm), run:

import nlp_text_preprocessing as tp

# One-time automated setup for NLTK and spaCy resources
tp.download_nltk_packages()

---

Pipeline Automation Engine

The Pipeline class allows chaining cleaning and transformation steps cleanly:

import nlp_text_preprocessing as tp

# 1. Create default zero-config pipeline
pipe = tp.Pipeline.default()

# 2. Or build a custom step pipeline
pipe = tp.Pipeline([
    'to_lower_case',
    'expand_hashtags',
    'remove_urls',
    'remove_emails',
    'remove_extra_whitespace',
    'lemmatize'
])

# Process single text
clean_result = pipe.run("Check out #MachineLearning at https://example.com!")

# Process batch list or Pandas Series
clean_batch = pipe.run_batch(["Text 1", "Text 2"])
clean_series = pipe.run_series(df['text_column'])

# Generate diff report of changes
report = tp.diff_report(raw_texts, clean_batch)

---

Complete API Reference

Below is the complete reference guide for all functions available in the library:

1. General Feature Extraction

Function	Parameters	Return Type	Description
word_count(x)	x: str	int	Returns total whitespace-separated word count.
char_count(x)	x: str	int	Returns total characters excluding spaces.
avg_word_len(x)	x: str	float	Returns average word length (0.0 for empty inputs).
stop_words_count(x)	x: str	int	Returns count of stop words (case-insensitive).
sentence_count(x)	x: str	int	Returns total number of sentences.
avg_sentence_len(x)	x: str	float	Returns average sentence length in words.
punctuation_density(x)	x: str	float	Returns proportion of punctuation characters in text.
capital_ratio(x)	x: str	float	Returns proportion of uppercase words relative to total words.
unique_word_ratio(x)	x: str	float	Returns lexical diversity ratio (unique words / total words).
readability_score(x)	x: str	float	Calculates Flesch Reading Ease score.
hashtags_count(x)	x: str	int	Returns count of hashtag tokens (#tag).
mentions_count(x)	x: str	int	Returns count of user handle mentions (@user).
numerics_count(x)	x: str	int	Returns count of numeric tokens.
upper_case_count(x)	x: str	int	Returns count of uppercase words.
text_stats(x)	x: str	dict	Returns a dictionary containing all extracted feature statistics at once.
text_stats_batch(df, col)	df, col	DataFrame	Returns a DataFrame of stats for a pandas DataFrame column.

---

2. Text Cleaning and Normalization

Function	Parameters	Return Type	Description
to_lower_case(x)	x: str	str	Converts input text to lowercase.
contraction_to_expansion(x)	x: str	str	Expands contractions using contractions dictionary.
remove_emails(x)	x: str	str	Removes email addresses from text.
count_emails(x)	x: str	int	Counts email addresses present in text.
remove_urls(x)	x: str	str	Removes HTTP/HTTPS links and www URLs.
count_urls(x)	x: str	int	Counts URLs present in text.
remove_rt(x)	x: str	str	Removes retweet headers (RT @user).
count_rt(x)	x: str	int	Counts retweet occurrences in text.
remove_html_tag(x)	x: str	str	Strips HTML/XML tags using BeautifulSoup.
remove_accented_chars(x)	x: str	str	Normalizes accented characters (NFKD -> ASCII).
remove_mentions(x)	x: str	str	Removes user handle mentions (@username).
remove_special_chars(x)	x: str	str	Strips punctuation and special symbols.
remove_repeated_chars(x)	x: str	str	Truncates repeated characters beyond 2 consecutive occurrences.
remove_stop_words(x)	x: str	str	Case-insensitive removal of stop words while preserving word case.
remove_emojis(x)	x: str	str	Strips emoji characters from text.
extract_emojis(x)	x: str	list[str]	Returns list of emojis found in text.
emoji_to_text(x)	x: str	str	Converts emojis to descriptive text (e.g. 😂 -> face_with_tears_of_joy).
remove_phone_numbers(x)	x: str	str	Strips phone numbers from text.
count_phone_numbers(x)	x: str	int	Counts phone numbers present in text.
mask_pii(x)	x: str	str	Redacts emails, phone numbers, and card numbers with [REDACTED].
remove_extra_whitespace(x)	x: str	str	Collapses consecutive spaces, tabs, and newlines to a single space.
expand_hashtags(x)	x: str	str	Splits camelCase/PascalCase hashtags (#MachineLearning -> Machine Learning).
normalize_unicode_punctuation(x)	x: str	str	Replaces smart quotes, em-dashes, and ellipsis with ASCII equivalents.
remove_numbers(x)	x: str	str	Strips all numeric digits.
remove_currency_symbols(x)	x: str	str	Strips currency symbols ($, €, ₹, £, ¥).
normalize_whitespace_and_casing(x)	x: str	str	Canonical combination of lowercasing and whitespace collapse.
clean_text(text)	text: str	str	Runs end-to-end cleaning pipeline.

---

3. Linguistic and Morphological Processing

Function	Parameters	Return Type	Description
convert_to_base(x)	x: str	str	Lemmatizes nouns and verbs via spaCy while keeping other POS tags.
lemmatize(x)	x: str	str	Performs full lemmatization across all tokens via spaCy.
extract_entities(x)	x: str	list[tuple]	Extracts named entities (text, label) using spaCy.
extract_pos_tags(x)	x: str	list[tuple]	Extracts full POS tag breakdown (token, pos, tag) using spaCy.
get_dependency_tree(x)	x: str	list[tuple]	Extracts dependency parse tree (token, dep, head) using spaCy.
detect_language(x)	x: str	str	Detects language code ('en', 'fr', etc.) using TextBlob.
keyword_extraction(x, top_n)	x: str, top_n: int	list[str]	Extracts top N keywords by term frequency.
text_similarity(x1, x2)	x1, x2	float	Calculates token Jaccard similarity between two texts (0.0 to 1.0).
get_synonyms(word)	word: str	list[str]	Returns list of synonyms from WordNet.
get_antonyms(word)	word: str	list[str]	Returns list of antonyms from WordNet.
chunk_noun_phrases(x)	x: str	list[dict]	Extracts noun phrases with character start/end spans.
correct_spelling(x)	x: str	str	Corrects word spelling using TextBlob model.
get_noun_phrase(x)	x: str	list[str]	Extracts noun phrases from text.
n_grams(x, n=2)	x: str, n: int	list	Generates word-level n-grams.
singularize_words(x)	x: str	str	Converts plural nouns (NNS) into singular forms.
pluralize_words(x)	x: str	str	Converts singular nouns (NN) into plural forms.

---

4. Safety, Moderation and Sentiment

Function	Parameters	Return Type	Description
detect_profanity(x)	x: str	bool	Returns True if text contains profanity words.
censor_profanity(x)	x: str	str	Censors profanity words with asterisks (****).
detect_toxicity(x)	x: str	float	Calculates baseline toxicity score (0.0 to 1.0).
detect_spam_signals(x)	x: str	dict	Returns spam flags for excessive caps, punctuation, and URL density.
sentiment_analysis(x)	x: str	str	Returns sentiment classification ('pos' / 'neg') using cached NaiveBayes model.
polarity_subjectivity(x)	x: str	dict	Returns fast pattern-based sentiment polarity (-1.0 to 1.0) and subjectivity.
emotion_detection(x)	x: str	dict	Calculates lexicon-based emotion counts (joy, anger, sadness, fear).

---

5. Visualization Suite

Function	Parameters	Return Type	Description
get_wordcloud(x, save_path=None, show=True)	x: str, save_path: str, show: bool	WordCloud	Generates a WordCloud object, with optional file saving and GUI toggle options.

---

Code Examples

1. Basic Text Cleaning

import nlp_text_preprocessing as tp

raw_text = "Check out https://example.com! Contact support@company.org or RT @user 'I'm loving #NLP'."
cleaned = tp.clean_text(raw_text)

print(cleaned)
# Output: check out contact support company org or i am love nlp

2. Pipeline Engine Usage

import nlp_text_preprocessing as tp

# Build custom pipeline
pipe = tp.Pipeline(['to_lower_case', 'expand_hashtags', 'remove_urls', 'remove_extra_whitespace'])

text = "Check out #MachineLearning at https://example.com!"
print(pipe.run(text))
# Output: check out machine learning at!

3. Pandas DataFrame Integration

All functions seamlessly support Pandas .apply() and handle missing values (NaN/None):

import pandas as pd
import nlp_text_preprocessing as tp

df = pd.DataFrame({
    'text': [
        "I'm loving this NLP package! #awesome",
        "Contact me at info@test.com or visit https://test.com",
        None
    ]
})

# Apply end-to-end text cleaning
df['clean_text'] = df['text'].apply(tp.clean_text)

# Extract word counts safely
df['word_count'] = df['text'].apply(tp.word_count)

4. Batch Feature Extraction

Extract full feature matrices directly into Pandas columns:

import pandas as pd
import nlp_text_preprocessing as tp

df = pd.DataFrame({'text': ["Hello #world! Visit https://example.com", "Contact user@test.com"]})

# Extract all summary stats into a new DataFrame
stats_df = tp.text_stats_batch(df, 'text')
print(stats_df)

5. WordCloud Generation

import nlp_text_preprocessing as tp

text = "python natural language processing machine learning text mining data science spaCy nltk textblob"

# Save directly to disk without opening interactive GUI window
tp.get_wordcloud(text, save_path="wordcloud.png", show=False)

---

Troubleshooting and Resource Setup

If any NLTK corpus or spaCy model throws a missing resource error during runtime, simply run:

import nlp_text_preprocessing as tp
tp.download_nltk_packages()

Or manually download spaCy model via terminal:

python -m spacy download en_core_web_sm

---

License and Author

• Author: Uditya Narayan Tiwari
• License: MIT License (LICENSE)