Implement content quality scoring with NeMo Curator filters

Introduction

When you ingest documents from the open web, a measurable fraction will be navigation chrome, duplicated template blocks, or under-50-word stubs — and if you embed and index them unfiltered, your retriever surfaces garbage whenever a query brushes the boilerplate. Teams that skip a quality-scoring layer pay for it in retrieval precision and human-review backlog downstream. By the end of this lesson you'll be able to wire NeMo Curator's heuristic filters into a pipeline, add a TF-IDF-based boilerplate detector, and combine the resulting signals into a configurable policy that emits a PASS / FLAG / QUARANTINE / REJECT action per document.

Key Terminology

• Heuristic filter — a deterministic, rule-based check (word count, repeated-paragraph ratio, language ID) applied per document to drop or flag obvious junk before more expensive scoring runs.
• boilerplate_ratio — the fraction of a document's TF-IDF weight concentrated in low-information terms (navigation, footers, templated phrases); high values flag documents dominated by non-informational content rather than topical signal.
• QualityPolicy — the dataclass that externalises every threshold (min_word_count, max_boilerplate_ratio, min_language_confidence, target_languages) so operators can tune behaviour without modifying pipeline code.
• QualityAction — the enum (PASS, FLAG, QUARANTINE, REJECT) the scorer emits per document so downstream stages know whether to index, review, or drop it.
• Composite quality score — a value in [0, 1] computed by deducting penalties from a baseline of 1.0; the scorer maps band ranges of this score onto a QualityAction.

Concepts

Content quality scoring rests on three layered ideas that build toward a single per-document action.

Cheap signals first, expensive signals last. Quality assessment is a funnel. Deterministic heuristic filters — word count, repeated-paragraph ratio, language confidence — are O(1) per document and catch the bulk of obvious junk (stubs, scraper duplication, wrong-language pages). Run them before the TF-IDF boilerplate detector, which must vectorize the document, so you never pay vectorization cost on a document a word-count check would have dropped. Order the pipeline cheapest-to-costliest.

Boilerplate is a distributional signal, not a keyword list. Navigation chrome, footers, and templated phrases recur across many documents, so a TfidfVectorizer fit on a representative corpus assigns them low TF-IDF weight while topic-specific terms score high. boilerplate_ratio is the share of a document's total TF-IDF weight concentrated in those low-information terms — a high ratio means the document is dominated by template, not content. The vectorizer is fit once on a domain-representative sample and reused; refitting per document destroys the cross-document comparison the score depends on.

Separate policy from mechanism. Thresholds belong in a QualityPolicy dataclass, not inline in the scorer. The QualityScorer combines signals into a composite score in [0, 1] by deducting penalties from a baseline of 1.0, then maps score bands onto a QualityAction (PASS / FLAG / QUARANTINE / REJECT). Because every threshold is externalized, operators retune behaviour through configuration without touching pipeline code, and each emitted action carries enough provenance (doc_id, issues, quality_score) for downstream stages to index, review, or auditably drop the document.

Code Walkthrough

Extracting Quality Signals

Set up NeMo Curator's heuristic filters and the TF-IDF boilerplate detector — the two signal sources the policy will later combine:

Code snippetpython
1from nemo_curator.filters import (
2    WordCountFilter,
3    RepeatedParagraphsFilter,
4)
5from nemo_curator.datasets import DocumentDataset
6from sklearn.feature_extraction.text import TfidfVectorizer
7import numpy as np
8import pandas as pd
9
10def apply_heuristic_filters(
11    documents: list[dict],
12    min_words: int = 50,
13    max_words: int = 100000,
14    max_repeated_ratio: float = 0.3,
15) -> tuple[list[dict], list[dict]]:
16    """Apply NeMo Curator heuristic filters, returning passed and failed docs."""
17    df = pd.DataFrame(documents)
18    dataset = DocumentDataset(df)
19
20    word_filter = WordCountFilter(
21        min_words=min_words,
22        max_words=max_words,
23        text_field="text",
24    )
25    repeated_filter = RepeatedParagraphsFilter(
26        max_repeated_fraction=max_repeated_ratio,
27        text_field="text",
28    )
29
30    passed = word_filter(dataset)
31    passed = repeated_filter(passed)
32
33    passed_ids = set(passed.df["doc_id"].tolist())
34    passed_docs = [d for d in documents if d["doc_id"] in passed_ids]
35    failed_docs = [d for d in documents if d["doc_id"] not in passed_ids]
36    return passed_docs, failed_docs
37
38class BoilerplateDetector:
39    def __init__(self, corpus_sample: list[str], threshold: float = 0.15):
40        self.threshold = threshold
41        self.vectorizer = TfidfVectorizer(
42            max_features=10000,
43            ngram_range=(1, 2),
44            stop_words="english",
45        )
46        self.vectorizer.fit(corpus_sample)
47
48    def score_document(self, text: str) -> dict:
49        doc_vector = self.vectorizer.transform([text]).toarray()[0]
50        low_tfidf_mask = doc_vector < self.threshold
51        boilerplate_ratio = np.sum(low_tfidf_mask * doc_vector) / max(
52            np.sum(doc_vector), 1e-9
53        )
54        return {
55            "boilerplate_ratio": float(boilerplate_ratio),
56            "unique_term_count": int(np.sum(doc_vector > 0)),
57            "is_boilerplate_heavy": boilerplate_ratio > 0.5,
58        }

• apply_heuristic_filters: WordCountFilter rejects documents shorter than min_words (too thin to retrieve meaningfully) or longer than max_words (likely concatenation errors). RepeatedParagraphsFilter drops documents where more than max_repeated_fraction of content is duplicated blocks — a common scraper artifact.
• BoilerplateDetector.__init__: TfidfVectorizer learns term importance from a corpus sample. Frequent cross-document terms (navigation, footers) get low TF-IDF scores; topic-specific terms get high scores. Fit once on a representative corpus and reuse — never refit per document.
• BoilerplateDetector.score_document: For each document, sums the TF-IDF weight attributable to low-scoring terms and divides by total weight to produce boilerplate_ratio. A ratio above 0.5 indicates the document is dominated by repetitive non-informational content.

Defining Quality Threshold Policies

Combine individual quality signals into a configurable policy system:

1from dataclasses import dataclass
2from enum import Enum
3
4class QualityAction(str, Enum):
5    PASS = "pass"
6    FLAG = "flag"
7    QUARANTINE = "quarantine"
8    REJECT = "reject"
9
10@dataclass
11class QualityPolicy:
12    min_word_count: int = 50
13    max_boilerplate_ratio: float = 0.6
14    min_language_confidence: float = 0.8
15    target_languages: list[str] = None
16
17    def __post_init__(self):
18        if self.target_languages is None:
19            self.target_languages = ["en"]
20
21class QualityScorer:
22    def __init__(self, policy: QualityPolicy, boilerplate_detector: BoilerplateDetector):
23        self.policy = policy
24        self.detector = boilerplate_detector
25
26    def score(self, doc: dict) -> dict:
27        text = doc.get("text", "")
28        word_count = len(text.split())
29        boilerplate = self.detector.score_document(text)
30
31        language = doc.get("language", "en")
32        language_confidence = doc.get("language_confidence", 1.0)
33
34        issues = []
35        if word_count < self.policy.min_word_count:
36            issues.append("too_short")
37        if boilerplate["boilerplate_ratio"] > self.policy.max_boilerplate_ratio:
38            issues.append("high_boilerplate")
39        if language not in self.policy.target_languages:
40            issues.append("off_target_language")
41        if language_confidence < self.policy.min_language_confidence:
42            issues.append("low_language_confidence")
43
44        quality_score = 1.0
45        quality_score -= 0.3 * min(boilerplate["boilerplate_ratio"], 1.0)
46        if word_count < self.policy.min_word_count:
47            quality_score -= 0.4
48        if language not in self.policy.target_languages:
49            quality_score -= 0.4
50        if language_confidence < self.policy.min_language_confidence:
51            quality_score -= 0.2 * (
52                self.policy.min_language_confidence - language_confidence
53            )
54
55        action = QualityAction.PASS
56        if quality_score < 0.3:
57            action = QualityAction.REJECT
58        elif quality_score < 0.5:
59            action = QualityAction.QUARANTINE
60        elif quality_score < 0.7:
61            action = QualityAction.FLAG
62
63        return {
64            "doc_id": doc.get("doc_id"),
65            "quality_score": round(max(quality_score, 0.0), 3),
66            "action": action.value,
67            "issues": issues,
68            "boilerplate_ratio": boilerplate["boilerplate_ratio"],
69            "word_count": word_count,
70        }

This three-layer approach — NeMo Curator heuristic filters for fast bulk checks, custom boilerplate detection for domain-specific quality, and configurable policy scoring for action routing — gives you a comprehensive quality assessment pipeline that adapts to different corpus profiles through configuration rather than code changes.

Do's and Don'ts

Now that you have worked through the implementation, the practices below separate a durable approach from a fragile one.

Do's

1. ✓Run cheap heuristic filters (word count, repeated-paragraph ratio) before expensive TF-IDF boilerplate scoring so you discard obvious junk before paying for vectorization.
2. ✓Fit the TfidfVectorizer on a representative corpus sample drawn from the same domain you intend to score — boilerplate signatures are domain-specific.
3. ✓Externalize every threshold (min_word_count, max_boilerplate_ratio, min_language_confidence) on QualityPolicy so operators can retune the PASS/FLAG/QUARANTINE/REJECT bands without redeploying pipeline code.

Don'ts

1. ✗Don't reuse a TfidfVectorizer fitted on one corpus to score documents from a different domain; common-vs-rare term distributions shift and boilerplate detection collapses.
2. ✗Don't hard-code quality thresholds inside QualityScorer.score — that defeats the policy object and forces a code change for every threshold tweak.
3. ✗Don't treat REJECT as silent deletion; persist the doc_id, quality_score, and issues list so rejections are auditable when downstream retrieval gaps surface.