GenAI Safety & Evaluation Engineering

Curate, version, and maintain evaluation datasets for hosted LLMs.

Implement Technical Requirements for EU AI Act Risk Categories with Current Enforcement Timeline

Define a Pydantic-based schema for evaluation test cases, including enums for task categories and difficulty levels, and build a schema validator that loads JSONL datasets and reports validation errors with row-level detail.

Build a TestCaseGenerator that creates evaluation test cases spanning five task categories and three difficulty levels using template-based patterns, then analyze the resulting dataset for stratification balance and coverage gaps.

Build a DatasetBuilder class that loads JSONL evaluation datasets, validates entries with Pydantic, computes coverage statistics including category counts, difficulty distribution, and average token counts, then generates a CoverageReporter that formats results as tables and identifies balance…

Build a ContaminationDetector class that analyzes recorded probe transcripts - the expected answer for each evaluation test case paired with the text a model returned for it - and decides deterministically which cases look contaminated. You implement text normalization, n-gram overlap scoring…

Implement ROUGE-L scoring from scratch using the longest common subsequence algorithm to measure text overlap between model outputs and expected answers, then build a batch scorer that computes contamination rates across an evaluation dataset.

Build a QuarantineWorkflow that takes per-(case, model) contamination signals, isolates burned evaluation test cases with evidence records, and generates an aggregate contamination report with per-model and per-category rates - all with deterministic grouping, filtering, and arithmetic that runs…

Build a ComplianceTracker over the four EU AI Act enforcement milestones (Feb 2025, Aug 2025, Feb 2026, Aug 2026). Compute the signed days remaining to each deadline relative to an explicit reference date, classify every milestone as complete, overdue, due-soon (within a configurable window), or…

Build an AutomatedComplianceChecker that registers a verification predicate for each compliance requirement, runs one or all checks against a system-state snapshot, treats a predicate error as a failed check rather than a crash, and aggregates the pass/fail results into a compliance summary with…

Build a GapReportGenerator that identifies non-compliant and partially compliant requirements, prioritizes them by combining enforcement deadline proximity with risk severity into a single priority score, generates deterministic remediation recommendations, and produces a formatted, ranked gap…