Implement Presidio regex and NER-based PII detection

Introduction

When you ingest free-form user text into a GenAI data pipeline, every document might carry a Social Security number, a patient record ID, or a personal email address you never intended to forward to a model provider — and skipping a fast first-pass scrubber is how teams end up with regulated PII in third-party logs. This lesson shows you how to implement PII detection in a GenAI data pipeline using Microsoft Presidio's combination of regex-based recognizers and spaCy named entity recognition. By the end you'll be able to configure the AnalyzerEngine and register custom PatternRecognizer instances for domain-specific identifiers so the regex/NER stage can serve as the fast first layer of detection before more expensive methods.

Key Terminology

• AnalyzerEngine: Presidio's orchestrator that runs the registered recognizers over input text and returns a list of RecognizerResult objects with entity type, character offsets, and a confidence score.
• PatternRecognizer: A regex-driven recognizer that maps one or more Pattern objects to a single entity type and optionally boosts the score when configured context words appear near the match.
• NlpEngineProvider: The wrapper that loads the spaCy model (for example en_core_web_lg) behind the analyzer so built-in NER entities such as PERSON and LOCATION can be detected alongside regex matches.

Concepts

Presidio splits PII detection into two complementary mechanisms: deterministic regex patterns for structured identifiers (SSN, email, phone, credit card, MRN) and spaCy NER models for unstructured entities such as person names and organizations. Each recognizer reports a confidence score between 0 and 1, and context words near a match can lift that score above the baseline so downstream filtering can apply a single threshold across heterogeneous detectors. Because the regex/NER stage runs on CPU with predictable latency, it is positioned as the first layer of a pipeline — catching the bulk of structured PII cheaply before more expensive methods run.

The AnalyzerEngine and Built-in Recognizers

The AnalyzerEngine is the orchestrator that holds a registry of recognizers and routes input text through them. Out of the box, the registry includes ~180 built-in recognizers covering common entity types — SSN, EMAIL_ADDRESS, PHONE_NUMBER, CREDIT_CARD, PERSON, LOCATION — using a mix of regex, checksum validators, and spaCy NER. Configuring the engine means picking the spaCy model (typically en_core_web_lg for production-quality NER), constraining the supported languages so unused models stay out of memory, and optionally passing an entities=[...] list at analyze time to restrict detection to the types you care about (see Code Walkthrough).

Custom PatternRecognizers and Context Boosting

Built-in recognizers don't cover domain-specific identifiers like medical record numbers, internal employee IDs, or proprietary account formats. For those you build a PatternRecognizer that maps one or more Pattern objects to a custom entity type and register it on the analyzer's registry at startup. The context field on a PatternRecognizer lists cue words — "patient", "chart", "record" near an MRN match — that lift the confidence score above the regex baseline, so a downstream threshold filter can distinguish a true MRN from an incidental 8-digit number (see Code Walkthrough).

Code Walkthrough

The two snippets below put the concepts from the previous section into running code: first the AnalyzerEngine wired up with the spaCy NLP engine and a request for three built-in entity types, then a PatternRecognizer that adds a custom MRN entity with context boosting and registers it on the live analyzer.

Configuring the Presidio Analyzer

The AnalyzerEngine is the central class that orchestrates all recognizers. You instantiate it with a spaCy NLP engine and then call analyze() on text to receive a list of RecognizerResult objects, each containing the entity type, start and end character positions, and a confidence score:

1from presidio_analyzer import AnalyzerEngine, PatternRecognizer, Pattern
2from presidio_analyzer.nlp_engine import NlpEngineProvider
3
4nlp_provider = NlpEngineProvider(nlp_configuration={
5    "nlp_engine_name": "spacy",
6    "models": [{"lang_code": "en", "model_name": "en_core_web_lg"}],
7})
8
9analyzer = AnalyzerEngine(
10    nlp_engine=nlp_provider.create_engine(),
11    supported_languages=["en"],
12)
13
14text = "Contact John Smith at john.smith@acme.com or 555-867-5309."
15results = analyzer.analyze(
16    text=text,
17    language="en",
18    entities=["PERSON", "EMAIL_ADDRESS", "PHONE_NUMBER"],
19)
20
21for result in results:
22    print(f"{result.entity_type}: '{text[result.start:result.end]}' "
23          f"(score={result.score:.2f})")

Building Custom Recognizers for Domain-Specific Patterns

Presidio's built-in recognizers cover common PII types, but production pipelines frequently encounter domain-specific identifiers that require custom detection. Medical record numbers, internal employee IDs, and proprietary account formats all follow patterns unique to the organization. You create a PatternRecognizer with one or more regex patterns and register it with the analyzer:

1mrn_pattern = Pattern(
2    name="medical_record_number",
3    regex=r"\bMRN[-: ]?\d{7,10}\b",
4    score=0.85,
5)
6
7mrn_recognizer = PatternRecognizer(
8    supported_entity="MEDICAL_RECORD_NUMBER",
9    patterns=[mrn_pattern],
10    supported_language="en",
11    context=["patient", "medical", "record", "chart"],
12)
13
14analyzer.registry.add_recognizer(mrn_recognizer)
15
16results = analyzer.analyze(
17    text="Patient chart MRN-4829103 shows allergies to penicillin.",
18    language="en",
19    entities=["MEDICAL_RECORD_NUMBER"],
20)

You'll know it works when the first snippet prints one RecognizerResult line per detected entity (PERSON, EMAIL_ADDRESS, PHONE_NUMBER) with a non-zero score, and the second snippet returns a MEDICAL_RECORD_NUMBER result for MRN-4829103 whose score is higher than the 0.85 baseline because the word "Patient" sits inside the context window.

Do's and Don'ts

Do's

1. ✓Pin the exact en_core_web_lg spaCy model version in production so NER detections do not drift between deployments.
2. ✓Provide a context word list on every custom PatternRecognizer so nearby cue words (for example "patient" near an MRN) lift the score above the regex baseline.
3. ✓Measure precision and recall per entity type against a labeled test set and set entity-specific thresholds (for example require recall ≥ 0.95 on SSN before promoting a change to production).

Don'ts

1. ✗Don't omit the entities filter during development tuning — running all 180+ built-in recognizers makes it harder to attribute false positives to a specific pattern; in production, drop the filter so built-in and custom recognizers both run.
2. ✗Don't rely on Presidio alone for unstructured names in informal text — the regex/NER layer is the fast first pass, so escalate low-score or context-poor matches to a downstream detector instead of treating spaCy NER output as final.
3. ✗Don't hand-tune a single global score threshold across all entity types — structured identifiers (email, SSN) and free-text entities (PERSON) need different cutoffs derived from the per-type benchmark.