Profile customer datasets for quality and PII exposure

Introduction

When you run a client discovery workshop, proposing a promising AI use case only to discover the underlying data cannot support it wastes weeks of scoping effort. Data readiness assessment provides an objective, repeatable way to evaluate whether a client's dataset has the schema completeness, null rates, cardinality distribution, volume, and text quality needed before any model work begins. This lesson teaches you to build a DataReadinessProfiler using pandas and Pydantic that produces a structured readiness report — including PII detection — so you can give workshop stakeholders quantified evidence, not guesswork.

Key Terminology

• Null Rate — the fraction of missing values in a column, computed as series.isnull().mean(); a high null rate on a field the proposed use case depends on is a readiness blocker that must be surfaced before any model work begins.
• Cardinality Ratio — the number of unique values in a column divided by total row count, computed as series.nunique() / len(series); a ratio near 1.0 on a text field (e.g., thousands of unique department names in a 10,000-row dataset) signals inconsistent naming conventions that degrade downstream classifiers.
• ColumnProfile — a Pydantic model that captures per-column statistics including null_rate, unique_count, cardinality_ratio, and up to five sample_values; type-checking at instantiation ensures incomplete profiles fail fast rather than silently propagating bad data.
• DataReadinessReport — the top-level Pydantic model that assembles all column_profiles alongside aggregate assessments: schema_completeness_score, volume_assessment, embedding_suitability, pii_findings, and overall_readiness; this structured object is the deliverable stakeholders receive in place of verbal guesswork.
• Embedding Suitability — a rating within DataReadinessReport that evaluates whether a dataset's text columns are appropriate for vector embedding, considering factors such as average text length, vocabulary diversity, and language consistency.
• PII Detection — the process of scanning sample text from client data using Presidio's AnalyzerEngine, which returns RecognizerResult objects carrying an entity_type, character span, and confidence score; results populate pii_findings in the readiness report and make data governance obligations concrete before a use case advances to scoping.

Concepts

Why Assessment Must Precede Model Work

The core problem data readiness assessment solves is costly late discovery: a use case that looks promising in a whiteboard session can collapse when the underlying dataset has 40% null rates on its most important field, or when the client's text exports turn out to contain thousands of unredacted SSNs. Running a structured profiler at the start of a discovery workshop makes those blockers visible while scope is still negotiable, not after a sprint of model work has already been spent.

The lesson's DataReadinessProfiler turns this into a repeatable, evidence-based process. Instead of asking a client "how complete is your data?", you run profile_column across their export and hand back a DataReadinessReport with actual null rates, cardinality distributions, and PII findings. Stakeholders who receive numbers can act on them; stakeholders who receive verbal assessments often do not.

Five Dimensions, One Score

Data readiness is not a single thing — it is five orthogonal properties that each block a different class of AI project. Schema completeness asks whether the required fields exist at all. Null rate measures how much of any given field is actually populated. Cardinality reveals distribution anomalies: too many unique values in a categorical field means dirty data; too few in a text field means insufficient variance. Volume determines whether there are enough records for the chosen approach (fine-tuning needs thousands of labeled examples; RAG can work with far fewer documents). Embedding suitability evaluates whether text content — its length, vocabulary richness, and language consistency — will produce meaningful vector representations.

The DataReadinessReport model captures all five dimensions, which means every field in its schema maps directly to one of these concerns (see Code Walkthrough).

PII as a Readiness Blocker, Not an Afterthought

PII detection belongs inside the readiness assessment rather than in a separate compliance review, because clients frequently do not know their own data contains personal information. A customer notes column that appears to hold support ticket text may contain embedded SSNs, email addresses, or phone numbers entered by agents over years. Discovering this after a use case has been scoped — or worse, after data has been shared with a model provider — is a project-stopping event.

Presidio's AnalyzerEngine makes detection cheap enough to run during a workshop. Scanning a sample of each text column and populating pii_findings with typed, confidence-scored RecognizerResult objects converts an invisible risk into a documented deliverable. "Column customer_notes contains 12 high-confidence PERSON + EMAIL co-occurrences" is an actionable finding a client can bring to their data governance team immediately.

Code Walkthrough

Now that you understand the five readiness dimensions — schema completeness, null rate, cardinality, volume, and embedding suitability — the code below translates each into a measurable, structured output.

The DataReadinessProfiler anchors its output in two Pydantic models. ColumnProfile captures per-column statistics: null rate, unique count, cardinality ratio, and a short list of representative sample values. DataReadinessReport assembles those profiles into a full assessment, adding a schema_completeness_score, a volume_assessment verdict, an embedding_suitability rating, and a pii_findings list. Using Pydantic ensures every field is type-checked at instantiation, so an incomplete report fails fast rather than silently propagating bad data into downstream scoring. The profile_column function is the workhorse: it computes null rate as isnull().mean(), the cardinality ratio as unique-count divided by total rows, and collects up to five non-null sample values for manual spot-checking.

Code snippetpython
1import pandas as pd
2from pydantic import BaseModel
3from typing import List
4
5class ColumnProfile(BaseModel):
6    name: str
7    dtype: str
8    null_rate: float
9    unique_count: int
10    cardinality_ratio: float
11    sample_values: List[str]
12
13class DataReadinessReport(BaseModel):
14    total_rows: int
15    total_columns: int
16    column_profiles: List[ColumnProfile]
17    schema_completeness_score: float
18    volume_assessment: str
19    embedding_suitability: str
20    pii_findings: List[dict]
21    overall_readiness: str
22
23def profile_column(df: pd.DataFrame, col: str) -> ColumnProfile:
24    series = df[col]
25    return ColumnProfile(
26        name=col,
27        dtype=str(series.dtype),
28        null_rate=float(series.isnull().mean()),
29        unique_count=int(series.nunique()),
30        cardinality_ratio=float(series.nunique() / len(series)),
31        sample_values=[str(v) for v in series.dropna().head(5).tolist()]
32    )

Running profile_column across every column in a client's export produces the column_profiles list that feeds the final DataReadinessReport. A cardinality_ratio near 1.0 on a text field signals potential data quality issues — for example, thousands of unique department names in a 10,000-row dataset likely indicates inconsistent naming conventions that will degrade any classifier built on top of it.

Customer datasets frequently contain PII that clients are unaware of. Before any dataset enters an AI pipeline, Presidio surfaces hidden exposure:

Code snippetpython
1from presidio_analyzer import AnalyzerEngine
2
3analyzer = AnalyzerEngine()
4results = analyzer.analyze(
5    text="Contact John Smith at john.smith@acme.com or 555-0123",
6    language="en",
7    entities=["PERSON", "EMAIL_ADDRESS", "PHONE_NUMBER", "US_SSN"]
8)
9for r in results:
10    print(r.entity_type, r.score)

AnalyzerEngine.analyze() returns a list of RecognizerResult objects, each identifying an entity type, its character span within the source text, and a confidence score. During a workshop, scanning a sample of the client's text columns populates pii_findings in the DataReadinessReport — giving stakeholders concrete evidence of data governance obligations before the use case advances to scoping.

Confirm that profile_column applied to a sample DataFrame returns a ColumnProfile whose null_rate falls between 0.0 and 1.0, and that AnalyzerEngine.analyze() returns at least one result flagging the email address in the test string above.

Do's and Don'ts

Having walked through the material above, the following Do's and Don'ts distill it into practice.

Do's

1. ✓Do anchor your readiness report in Pydantic models (ColumnProfile and DataReadinessReport) — type enforcement at instantiation means an incomplete or malformed report fails immediately rather than silently propagating bad null rates or missing pii_findings into downstream use-case scoring.
2. ✓Do interpret cardinality_ratio in column context, not in isolation — a ratio near 1.0 on a text field (e.g., thousands of unique department names across 10,000 rows) signals inconsistent naming conventions that will degrade any classifier built on that column, even when the null rate looks acceptable.
3. ✓Do run AnalyzerEngine.analyze() on a sample of the client's text columns before any use case advances to scoping — Presidio surfaces PII exposure clients are unaware of, and populating pii_findings in the DataReadinessReport gives stakeholders concrete evidence of data governance obligations rather than abstract warnings.

Don'ts

1. ✗Don't substitute plain dicts for ColumnProfile and DataReadinessReport when assembling the profiler output — without Pydantic's type-checked instantiation, a missing schema_completeness_score or a null_rate stored as a string passes silently and corrupts every readiness verdict derived from it.
2. ✗Don't compute cardinality ratio using row count from a filtered or sampled slice — profile_column divides series.nunique() by len(series) (the full column length), so passing a pre-filtered DataFrame inflates the ratio and misrepresents the true distribution, producing false "high-cardinality" flags on clean categorical fields.
3. ✗Don't defer the Presidio PII scan to post-scoping or treat it as optional cleanup — running AnalyzerEngine.analyze() after a use case is already resourced means stakeholders commit to a data pipeline before learning the dataset carries EMAIL_ADDRESS, PERSON, or US_SSN entities that require governance review or anonymization, reversing weeks of scoping work.