Prerequisites

This chapter assumes you have completed the course-level setup for Enterprise LLM Customization, including a working Python 3.11+ environment with pip and venv configured. You should be comfortable writing Python functions, working with file I/O, and using type hints. Familiarity with JSON and basic command-line operations is expected. No prior experience with Pydantic, Instructor, or ETL frameworks is required—each tool is introduced from first principles within the chapter. Access to a Google AI Studio API key for Gemini is needed for the code execution sections.

Learning Goals

1. Extract and PII-scrub training pairs from enterprise documents, proprietary f...

   • Extract and PII-scrub training pairs from enterprise documents, proprietary formats, and bulk data sourcesMaster the end-to-end process of converting raw enterprise content into clean, privacy-compliant training data suitable for fine-tuning large language models. This goal addresses the most labor-intensive and legally sensitive phase of any enterprise LLM customization project: transforming messy, real-world documents into structured instruction-response pairs while ensuring that no personally identifiable information leaks into the training set. Senior engineers must understand that the quality ceiling of any fine-tuned model is permanently bounded by the quality of its training data, and that PII contamination in training data can expose the organization to regulatory penalties under GDPR, CCPA, HIPAA, and sector-specific data protection frameworks. You will build extraction pipelines that handle the heterogeneous document landscape found in real enterprises, where a single department might store knowledge across PDFs, Confluence wikis, Slack threads, JIRA tickets, internal Git repositories, and proprietary XML formats that predate modern tooling.

   • Design document-type-aware extraction strategies that preserve semantic structure while discarding formatting artifacts. Enterprise documents are not uniform blobs of text. A 200-page PDF generated from a legacy SAP system carries fundamentally different structural signals than a Markdown runbook in an internal wiki. Your extraction layer must dispatch documents to type-specific parsers that understand headers, tables, code blocks, nested lists, footnotes, and cross-references within each format. For PDF extraction, you will use libraries such as pdfplumber and PyMuPDF to differentiate between text-layer PDFs and scanned image PDFs, routing the latter through OCR pipelines built on Tesseract or cloud-based document AI services. For proprietary formats like .docx, .pptx, and legacy .xls, you will use python-docx, python-pptx, and openpyxl respectively, mapping each format's internal XML structure to a unified intermediate representation. The intermediate representation should be a flat list of typed content blocks—paragraph, heading, table, code, list-item—each annotated with its hierarchical depth and source location, so that downstream training-pair extraction can reason about which blocks belong together semantically. A critical mistake that junior engineers make is stripping all formatting and concatenating raw text, which destroys the paragraph-to-heading relationships that are essential for generating high-quality instruction-response pairs. For bulk ingestion scenarios involving thousands of documents, you will implement a fan-out architecture where a manifest file lists every source document with its MIME type, byte size, and expected parser, and a pool of workers processes documents in parallel with per-document timeout guards to prevent a single malformed file from blocking the entire pipeline. Each worker writes its output to a per-document JSONL shard, and a final merge step concatenates shards into the canonical training file while deduplicating content blocks that appear in multiple source documents.

   • Implement multi-layer PII detection and scrubbing that satisfies legal review for regulated industries. PII scrubbing is not a single regex pass—it is a layered defense system that combines rule-based pattern matching, named-entity recognition, and context-aware heuristics. The first layer applies deterministic regex patterns for structured PII: Social Security numbers, credit card numbers (with Luhn validation to reduce false positives), phone numbers in international formats, email addresses, and IP addresses. The second layer runs a named-entity recognition model—typically spaCy's en_core_web_trf transformer pipeline or a dedicated PII model like Presidio from Microsoft—to detect names, organizations, dates of birth, and physical addresses that cannot be captured by simple patterns. The third layer applies context-aware heuristics: for example, a nine-digit number following the phrase "patient ID" should be scrubbed even if it does not match the SSN format, because it constitutes a quasi-identifier in a healthcare context. You must decide between replacement strategies: naive redaction (replacing PII with [REDACTED]) destroys sentence structure and teaches the model to generate redaction tokens, while entity-type replacement (replacing "John Smith" with [PERSON_1] and "Acme Corp" with [ORG_1]) preserves grammatical structure but leaks entity counts. The most robust approach for training data is synthetic replacement, where detected names are replaced with plausible synthetic names from a locale-appropriate name generator, detected addresses are replaced with synthetic addresses, and detected dates are shifted by a consistent random offset per document to preserve temporal relationships. You will implement a PiiScrubber class that chains all three layers, logs every detection with its confidence score and replacement value to an audit trail, and exposes a configurable confidence threshold below which detections are flagged for human review rather than automatically scrubbed. For bulk pipelines processing millions of records, you will track scrubbing statistics—detection counts by entity type, false-positive rates estimated from a manually reviewed sample, and processing latency per document—to provide the legal and compliance team with quantitative evidence that the scrubbing pipeline meets the organization's risk tolerance.

   • Build training-pair extraction logic that generates instruction-response pairs aligned with the target fine-tuning format. Raw text blocks, even after cleaning and PII scrubbing, are not training data. They must be transformed into structured pairs that match the conversational or instruction-following format expected by the fine-tuning framework. You will implement extraction strategies for three common enterprise patterns. First, question-answer extraction from FAQ documents and support tickets, where the subject line or question field becomes the instruction and the accepted answer becomes the response, with a quality filter that discards pairs where the response is shorter than 50 tokens or where the question is ambiguous without additional context. Second, summarization extraction from long-form documents, where each section becomes a response and a synthetically generated instruction such as "Explain the company's policy on [section heading]" becomes the prompt, using a frontier model like Gemini or GPT-4 to generate diverse instruction phrasings that avoid repetitive templates. Third, code-explanation extraction from internal repositories, where function docstrings or inline comments are paired with the code they describe, formatted as "Explain what this function does" → docstring or "Write a function that [docstring]" → code. Each extraction strategy outputs records in JSONL format with a consistent schema containing fields for instruction, response, source_document, extraction_method, and quality_score. The quality score is a heuristic composite of response length, lexical diversity measured by type-token ratio, and semantic coherence measured by embedding similarity between instruction and response. You will implement deduplication at the training-pair level using MinHash locality-sensitive hashing to detect near-duplicate responses that would cause the fine-tuned model to memorize specific phrasings rather than learning generalizable patterns, setting a Jaccard similarity threshold of 0.85 above which the shorter duplicate is discarded.

   • Handle proprietary and legacy formats through adapter patterns that isolate format-specific complexity. Many enterprises maintain critical knowledge in formats that no open-source parser handles well: legacy Lotus Notes databases, mainframe EBCDIC-encoded flat files, custom XML schemas from in-house tools built decades ago, or exported data from discontinued SaaS products. Rather than writing monolithic extraction code that tangles format parsing with business logic, you will implement an adapter pattern where each proprietary format gets a dedicated FormatAdapter class that implements a common interface with a single extract_blocks(file_path: str) -> list[ContentBlock] method. The adapter encapsulates all format-specific dependencies, error handling, and encoding conversions, so the rest of the pipeline operates exclusively on the unified ContentBlock intermediate representation. This pattern is essential for maintainability because proprietary formats are the most likely to require updates when the source system changes versions, and isolating that volatility behind a stable interface prevents ripple effects through the pipeline. For formats where no Python library exists, you will implement shell-out adapters that invoke command-line conversion tools—such as LibreOffice in headless mode for obscure Office variants or Pandoc for markup format conversions—capturing stdout and parsing the converted output. Each adapter must implement robust error handling that distinguishes between recoverable errors (a single corrupted page in an otherwise valid PDF) and fatal errors (an entirely unreadable file), logging the former as warnings and raising the latter as exceptions that the orchestration layer can route to a dead-letter queue for manual inspection.

2. Validate data quality with Pydantic schemas

   • Validate data quality with Pydantic schemas enforcing structural, semantic, and statistical constraintsImplement a comprehensive data validation layer using Pydantic that catches malformed, incomplete, and low-quality training records before they contaminate the fine-tuning dataset. Data validation in enterprise ML pipelines goes far beyond checking that JSON parses correctly. At the L5-L6 level, you must design validation schemas that encode your organization's data quality standards as executable code, ensuring that every training record meets structural requirements (correct field types and formats), semantic requirements (meaningful content that will actually improve model performance), and statistical requirements (distributions that match expected patterns and flag anomalies). Pydantic is the validation backbone because its declarative model syntax doubles as living documentation of your data contract, its error messages are precise enough to drive automated remediation, and its integration with the broader Python type-checking ecosystem catches errors at development time rather than at pipeline runtime. You will build validation schemas that evolve alongside your training data requirements, using Pydantic's model versioning capabilities to handle schema migrations without breaking existing pipeline stages.

   • Design hierarchical Pydantic models that validate training records at field, record, and batch levels. A single flat Pydantic model that validates individual fields is insufficient for production training data. You need a three-tier validation hierarchy. The field level uses Pydantic's built-in validators and custom types to enforce constraints on individual values: instructions must be between 10 and 500 tokens, responses must be between 50 and 4096 tokens, the extraction_method field must be one of a defined enum set, and the quality_score must be a float between 0.0 and 1.0. The record level uses Pydantic's model_validator decorator to enforce cross-field constraints that cannot be expressed on individual fields: the response must not be a substring of the instruction (which would indicate a copy-paste extraction error), the source_document field must reference a file that exists in the document registry, and the token count of the instruction plus response must not exceed the context window of the target fine-tuning model. The batch level uses a separate BatchValidator model that accepts a list of records and enforces statistical constraints across the entire dataset: the distribution of extraction_method values must include at least three distinct methods (to prevent the model from overfitting to a single document type), the mean quality score must exceed 0.7, and no single source document may contribute more than 5% of total records (to prevent the model from memorizing one document's style). Each validation tier produces structured error reports that identify the exact field, record index, and constraint that failed, enabling automated remediation pipelines to route records to the appropriate fix-up logic rather than simply discarding them.

   • Implement custom Pydantic validators that detect subtle data quality issues specific to LLM training data. Generic type-checking misses the data quality issues that actually degrade fine-tuning outcomes. You will implement custom validators for five critical quality dimensions. First, a language-consistency validator that uses langdetect or fasttext to verify that instruction and response are in the same language and match the target training language, catching records where a multilingual source document was partially translated. Second, a repetition detector that flags responses containing repeated phrases or sentences, which is a common artifact of poor extraction from documents with headers, footers, or boilerplate that was not properly stripped. Third, a toxicity screener that runs responses through a lightweight classifier to flag content that could cause the fine-tuned model to generate harmful outputs, with configurable thresholds per deployment context (a medical model has different safety requirements than a code-generation model). Fourth, a format-compliance validator that ensures responses follow the expected output format—if the fine-tuning task expects Markdown-formatted responses, the validator checks for proper heading syntax, code fence closure, and list formatting. Fifth, a semantic-drift validator that computes embedding similarity between each record's instruction-response pair and a set of reference examples from the target domain, flagging records whose embeddings fall outside the expected cluster, which indicates that the extraction pipeline pulled content from an irrelevant section of the source document. Each custom validator is implemented as a reusable Pydantic validator function decorated with @field_validator or @model_validator, composable across multiple schema versions, and documented with examples of the specific failure modes it detects.

   • Build JSONL serialization and deserialization pipelines with schema-versioned validation checkpoints. Production training data pipelines process data through multiple stages—extraction, scrubbing, validation, augmentation, deduplication, and final formatting—and each stage may modify the record schema. You will implement a schema-versioned JSONL pipeline where each record carries a schema_version field, and each pipeline stage declares which schema versions it accepts as input and which version it produces as output. The TrainingRecordV1 model contains core fields from extraction, TrainingRecordV2 adds quality metrics from the validation stage, and TrainingRecordV3 adds augmentation metadata from the data augmentation stage. Pydantic's model inheritance makes this natural: each version extends the previous one, and a RecordRouter function inspects the schema_version field and instantiates the appropriate model class for validation. For JSONL serialization, you will implement streaming readers and writers that process files line-by-line rather than loading entire datasets into memory, using Python generators to maintain constant memory usage regardless of dataset size. Each pipeline checkpoint writes a validated JSONL file with an accompanying manifest that records the total record count, the schema version, validation pass/fail statistics, and a SHA-256 checksum of the file contents for integrity verification. When a downstream stage detects a checksum mismatch or an unexpected schema version, it raises a PipelineIntegrityError rather than silently processing corrupted data. This checkpoint architecture enables incremental pipeline reruns: if the augmentation stage fails, you can restart from the last valid checkpoint rather than reprocessing from raw documents, saving hours of compute time on large datasets.

3. Orchestrate ETL pipelines with scheduling, incremental ingestion, and state t...

   • Orchestrate ETL pipelines with scheduling, incremental ingestion, and persistent state trackingDesign and implement production-grade ETL orchestration that transforms the individual extraction, validation, and formatting components into a reliable, observable, and incrementally updatable pipeline system. Moving from script-level data processing to enterprise ETL orchestration requires solving three fundamental problems that do not exist in one-shot batch jobs: scheduling (when does each pipeline stage run and what triggers it), incremental ingestion (how do you process only new or changed documents without reprocessing the entire corpus), and state tracking (how do you know which documents have been processed, which failed, and what the current state of the training dataset is at any point in time). At the senior engineer level, you must make principled architectural decisions about whether to adopt a full-featured orchestration framework like Airflow, Prefect, or Dagster, or whether to build lightweight orchestration using Python's native scheduling capabilities with a persistent state store. The answer depends on your organization's existing infrastructure, team expertise, and the expected scale of the pipeline. This goal teaches you to build orchestration that is correct regardless of scale, then layer on framework-specific optimizations when the operational requirements justify the added complexity.

   • Implement a DAG-based pipeline executor that manages stage dependencies, parallelism, and failure recovery. A training data pipeline is a directed acyclic graph where each node is a processing stage (extraction, PII scrubbing, validation, deduplication, formatting) and edges represent data dependencies. You will implement a PipelineDAG class that registers stages as callable objects with declared input and output types, resolves the execution order via topological sort, and executes independent stages in parallel using Python's concurrent.futures.ProcessPoolExecutor. Each stage receives its input from the previous stage's output directory and writes its output to a dedicated directory, with the DAG executor managing the handoff. Critical to production reliability is the failure recovery model: when a stage fails, the executor must distinguish between transient failures (network timeout when calling a cloud PII detection API) and permanent failures (a source document that consistently causes the parser to crash). Transient failures trigger automatic retries with exponential backoff, capped at a configurable maximum retry count. Permanent failures route the offending record to a dead-letter queue and continue processing the remaining records, so that one bad document does not block an entire pipeline run. The executor logs every stage transition—started, completed, failed, retried—to a structured event log that serves as the pipeline's audit trail. After each complete pipeline run, the executor generates a run summary that reports the total records processed, records failed, records routed to dead-letter, wall-clock time per stage, and the output file paths, providing operators with a single artifact to assess pipeline health. You will also implement a dry-run mode that validates the DAG structure, checks that all input files exist, and estimates processing time based on historical stage durations, without actually executing any stage—this is essential for pre-deployment validation in CI/CD pipelines.

   • Build incremental ingestion logic that tracks document-level processing state to avoid redundant reprocessing. Enterprise document corpora are not static. New documents are added daily, existing documents are updated, and occasionally documents are deleted or retracted. A production pipeline must process only the delta—new and changed documents—without reprocessing the entire corpus, which could take hours or days at enterprise scale. You will implement a DocumentRegistry backed by a SQLite database that records, for each source document, its file path, content hash (SHA-256 of the file contents), last-processed timestamp, processing status (pending, completed, failed), and the schema version of the output records it produced. At the start of each pipeline run, the ingestion stage scans the source document directory, computes the content hash of each file, and compares it against the registry. Documents whose hash has not changed since the last successful processing are skipped entirely. Documents whose hash has changed are marked for reprocessing, and their previous output records are removed from the training dataset before the new version is processed. New documents (present on disk but absent from the registry) are added with pending status. Deleted documents (present in the registry but absent from disk) are flagged, and their output records are removed from the training dataset. This content-hash approach is strictly more reliable than timestamp-based change detection, which breaks when files are copied, restored from backup, or modified by tools that do not update modification times. The registry also tracks cross-document dependencies: if document A references document B (for example, a FAQ document that references a policy document for context), and document B is updated, document A is also marked for reprocessing because its extracted training pairs may depend on stale context. You will implement registry queries that generate processing manifests—ordered lists of documents to process in the current run—and post-processing updates that atomically mark all successfully processed documents as completed within a single database transaction, ensuring that a crash during processing does not leave the registry in an inconsistent state.

   • Design scheduling and monitoring infrastructure that provides operational visibility into pipeline health. A pipeline that runs correctly but cannot be monitored or scheduled is not production-ready. You will implement scheduling at two levels: time-based scheduling for regular corpus updates (for example, processing new documents every night at 2 AM using a cron expression or APScheduler) and event-based triggering for on-demand runs (for example, triggering a pipeline run when a new batch of documents is uploaded to a watched directory or cloud storage bucket). The scheduler must enforce mutual exclusion: if a pipeline run is already in progress when a new trigger fires, the new run is queued rather than started concurrently, preventing race conditions on the document registry and output files. For monitoring, you will implement a health-check endpoint or status file that reports the current pipeline state: idle, running (with progress percentage), failed (with the error details from the last run), or degraded (completed with records routed to the dead-letter queue). Pipeline metrics—records processed per second, stage durations, error rates, dead-letter queue depth—are emitted in a format compatible with Prometheus or CloudWatch, enabling operators to set up alerts for anomalous pipeline behavior. You will also implement a pipeline audit log that records every run with its trigger source (scheduled, manual, event), start time, end time, input document count, output record count, and final status, stored in append-only format for compliance and debugging. For debugging failed runs, the audit log cross-references with the per-stage event log, enabling operators to trace a failed record from the final error back through every pipeline stage it traversed, identifying the exact point and cause of failure.

   • Implement state checkpointing and pipeline resumability for long-running ingestion jobs. Enterprise pipelines processing tens of thousands of documents can run for hours, and any interruption—a node restart, a memory limit, a network partition—should not require starting from scratch. You will implement stage-level checkpointing where each stage periodically writes its progress to a checkpoint file: the index of the last successfully processed record, the cumulative output statistics, and a reference to the partial output file. When the pipeline executor detects a checkpoint file for a stage that was previously interrupted, it resumes from the checkpointed position rather than restarting the stage. The checkpoint mechanism must be atomic: checkpoints are written to a temporary file and renamed to the final path using os.replace, which is an atomic operation on POSIX filesystems, preventing corrupted checkpoint files if the process crashes during the write. For stages that call external APIs (such as a cloud-based PII detection service or a frontier model for synthetic instruction generation), you will implement idempotency keys based on the record's content hash, so that resuming a stage does not result in duplicate API calls for records that were successfully processed before the interruption. The checkpoint files also serve as progress indicators: the monitoring system reads the checkpoint to report the percentage completion of a running stage, providing real-time visibility into long-running pipeline jobs. You will test the resumability mechanism by implementing a chaos-testing mode that randomly terminates stages mid-execution and verifies that the resumed pipeline produces identical output to an uninterrupted run, ensuring that the checkpointing logic does not introduce subtle data corruption or record reordering.

4. Build distillation pipeline from frontier to small model

   • Build a distillation pipeline from frontier model to small model using structured outputs and quality filteringConstruct an end-to-end knowledge distillation pipeline that uses a frontier model such as Gemini or GPT-4 as a teacher to generate high-quality training data, validates and filters the generated data using the Instructor library with Pydantic schemas, and formats the output for fine-tuning a smaller, deployable student model. Knowledge distillation is the most cost-effective strategy for enterprise LLM customization because it allows organizations to capture the reasoning capabilities of expensive frontier models in smaller models that can be deployed on-premises or at a fraction of the inference cost. The key insight is that frontier models, when prompted with domain-specific context from enterprise documents, can generate training examples of higher quality and greater diversity than manual annotation, but only if the generation pipeline includes rigorous structural validation, semantic quality filtering, and deduplication. You will use the Instructor library—a Python framework that patches LLM API clients to return validated Pydantic objects instead of raw text—to enforce that every generated training example conforms to your schema before it enters the training set, eliminating the parsing failures and format inconsistencies that plague naive generation pipelines. Gemini's Code Execution capability adds a unique dimension: you can generate training examples that include executable code, run that code in Gemini's sandboxed environment to verify correctness, and include only verified examples in the training set, producing code-generation training data with a correctness guarantee that no other approach can match.

   • Configure the Instructor library with Pydantic response models to guarantee structured output from frontier model API calls. The Instructor library transforms the unreliable process of prompting an LLM for structured output into a type-safe, validated pipeline. Without Instructor, generating training data from a frontier model requires writing fragile parsing code that extracts JSON from markdown code fences, handles partial responses, retries on malformed output, and manually validates every field—a process that fails silently at scale and produces datasets with subtle corruption. With Instructor, you define a Pydantic model representing the desired output schema, pass it to the patched API client, and receive a validated Python object or a clear validation error. You will configure Instructor with the Gemini API client using the instructor.from_gemini patch, setting mode=instructor.Mode.GEMINI_JSON to use Gemini's native JSON mode for maximum reliability. For each training example type—question-answer pairs, code-explanation pairs, summarization pairs—you will define a dedicated Pydantic response model with field-level validators that enforce quality constraints at generation time: the response field must contain at least 100 tokens, the instruction field must end with a question mark for QA pairs, and the difficulty rating must be an integer between 1 and 5. When Instructor detects a validation failure, it automatically retries the API call with the validation error appended to the prompt, giving the frontier model a chance to self-correct—a process that succeeds on the retry approximately 85% of the time for well-designed schemas. You will implement a DistillationConfig dataclass that parameterizes the generation pipeline: the teacher model name, temperature, maximum retries, batch size, rate-limiting parameters to stay within API quotas, and the target number of examples per category, allowing operators to tune the pipeline without modifying code.

   • Leverage Gemini Code Execution to generate and verify executable code training examples. Gemini's Code Execution feature allows the model to write Python code and execute it within a sandboxed environment, returning both the code and its output. This capability is transformative for generating code-related training data because it enables a generate-and-verify loop: Gemini generates a code example in response to an instruction, executes it to verify that it runs without errors and produces the expected output, and the pipeline includes only verified examples in the training set. You will implement a CodeDistillationPipeline that sends coding instructions to Gemini with code execution enabled, inspects the execution result for success or failure, and routes successful examples to the training set while routing failures back through a repair loop where Gemini is prompted to fix the error based on the traceback. The repair loop runs for a configurable maximum number of iterations (typically three), after which persistently failing examples are discarded and logged for manual review. For each verified code example, the pipeline records the instruction, the final working code, the execution output, and the number of repair iterations required, providing metadata that downstream quality filters can use to prioritize examples that succeeded on the first attempt. You will also implement output-based verification where the instruction specifies an expected output (for example, "Write a function that returns the Fibonacci sequence up to n=10") and the pipeline automatically compares Gemini's execution output against the expected value, providing a ground-truth correctness signal that is stronger than any heuristic quality metric. This verified-code approach produces training data that, when used to fine-tune a smaller code-generation model, measurably reduces the rate of syntax errors and runtime exceptions in the student model's outputs compared to training on unverified code examples.

   • Implement quality filtering and diversity sampling to curate the final distillation dataset. Generating thousands of training examples from a frontier model does not guarantee a high-quality training set. The distillation pipeline must include a curation stage that filters out low-quality examples and samples for diversity to maximize the fine-tuning signal per training token. You will implement quality filtering along four dimensions. First, a length filter that removes examples where the response is disproportionately short relative to the instruction complexity, using a learned length-ratio threshold calibrated on a small set of manually rated examples. Second, a self-consistency filter that generates multiple responses for the same instruction (typically three), computes pairwise semantic similarity using embedding cosine distance, and retains only instructions where at least two of three responses are highly similar—this filters out ambiguous instructions where the frontier model itself is uncertain. Third, a novelty filter that compares each generated example against the existing training set using embedding similarity, discarding examples that are too similar to existing records (below a cosine distance threshold of 0.15) to prevent redundancy. Fourth, a difficulty-distribution filter that ensures the final dataset contains a balanced mix of easy, medium, and hard examples as rated by the frontier model during generation, preventing the fine-tuned model from being biased toward any single difficulty level. After filtering, you will implement stratified sampling that selects examples to match a target distribution across categories (document types, task types, difficulty levels), ensuring that the final training set is representative of the full range of enterprise use cases rather than dominated by whichever category was easiest for the frontier model to generate. The curation stage outputs a final JSONL file accompanied by a dataset card—a metadata document recording the total examples generated, the filter pass rates, the final category distribution, and the estimated dataset quality score—providing full traceability from the raw generation to the curated training set.

   • Design the teacher-to-student training data format conversion with tokenizer-aware truncation and chat template formatting. The final pipeline stage converts the curated distillation dataset into the exact format required by the student model's fine-tuning framework, accounting for the student model's tokenizer, context window, and chat template. You will implement a FormatConverter class that accepts a target model identifier (for example, mistral-7b-instruct-v0.2 or llama-3-8b-instruct), loads the corresponding tokenizer from HuggingFace, and applies the model's chat template to each training example. The chat template wraps instruction-response pairs in the special tokens and role markers that the student model expects—for example, Llama-3 uses <|begin_of_text|><|start_header_id|>user<|end_header_id|> while Mistral uses [INST] and [/INST] delimiters. Applying the wrong template produces training data that the student model cannot learn from effectively, because the special tokens serve as control signals that the model uses to distinguish between user input and assistant output. The converter must also implement tokenizer-aware truncation: when an instruction-response pair exceeds the student model's context window (typically 4096 or 8192 tokens for fine-tuning), the converter must decide whether to truncate the response (acceptable for long-form generation tasks), split the example into multiple shorter examples (acceptable for document summarization), or discard the example entirely (necessary when truncation would remove critical information). You will implement truncation strategies as pluggable policies, allowing operators to select the appropriate strategy per task type. The converter outputs training data in the format expected by popular fine-tuning frameworks: JSONL with messages arrays for OpenAI-compatible frameworks, JSONL with text fields containing the fully templated string for HuggingFace TRL, and Alpaca format with instruction, input, and output fields for legacy frameworks. Each output file includes a header comment or sidecar metadata file recording the student model, tokenizer version, chat template hash, and truncation statistics, ensuring that the fine-tuning engineer can verify the data was formatted for the correct target model before launching a potentially expensive training run.

Key Terminology