Classify project risks with DSPy-optimized prompts

Introduction

When you scope an AI project without a structured risk process, critical threats — model accuracy degradation, data pipeline failures, and stakeholder misalignment — surface late and derail delivery. Manual risk registers are slow to build and easy to overlook under deadline pressure. This lesson shows you how to automate risk assessment across five categories: technical, data, integration, organizational, and timeline. By the end, you'll be able to build a DSPy-powered risk classification pipeline and generate structured mitigation strategies that quantify effort and assign ownership.

Key Terminology

• Risk category taxonomy — The five-dimension classification scheme (technical, data, integration, organizational, timeline) that sorts every identified AI project threat into a distinct failure-mode bucket, encoded as the valid values of the category output field in RiskClassifier.
• dspy.Signature — A DSPy class that declares a model's input/output contract as typed, described fields rather than a hand-crafted prompt; RiskClassifier extends it to specify that a project_spec string must map to five structured risk fields including severity, likelihood, and mitigation.
• ChainOfThought — A DSPy predictor that wraps a Signature and instructs the language model to produce intermediate reasoning before committing to each output field, reducing hallucinated severity scores and vague mitigation strings in RiskAssessmentModule.
• Severity scoring — A compound rating produced by combining a risk's likelihood (very_likely → unlikely) with its impact level, yielding a severity value (critical → low) that determines mitigation priority and can block project kickoff when rated critical.
• RiskMitigation — A Pydantic model that anchors each classified risk to a concrete response plan by capturing effort_story_points, timeline_impact_days, residual_severity, and owner_role, so mitigations feed directly into scope documents rather than living in a separate checklist.
• Residual severity — The severity level that remains after a mitigation strategy is applied, stored in RiskMitigation.residual_severity; a critical risk that drops to medium after remediation justifies the mitigation cost, while one that stays critical signals a genuine project blocker requiring escalation before scoping continues.

Concepts

From Ad Hoc Risk Lists to Declarative Classification

Traditional risk registers are built in kick-off workshops and filed away. They capture the threats that were top-of-mind on the day they were written, not the systemic risks that emerge from careful reading of a project specification. For AI projects this gap is particularly costly: model accuracy degradation, data pipeline failures, and stakeholder misalignment tend to surface late because anticipating them requires domain knowledge that a rushed checklist exercise does not surface reliably.

Automating risk assessment with a declarative pipeline changes the dynamic in two ways. First, the same analysis runs consistently on every specification — the classifier does not skip a category because a client's slide deck looked polished. Second, the output is structured from the start: each risk arrives with a category, severity, likelihood, and a mitigation stub ready to feed directly into scope documents and effort estimates (see Code Walkthrough). The DSPy Signature approach also separates the what (the risk schema) from the how (prompt tuning), so the classifier can be improved later with labeled examples instead of hand-edited prompts.

The Five-Category Taxonomy and Severity Scoring

The classifier assigns every identified risk to one of five categories, each mapping to a distinct AI project failure mode. Technical risks cover model accuracy, latency, context limits, and API stability — failure modes that are architectural and largely invisible to non-technical stakeholders until production. Data risks address volume, label quality, distribution shift, and pipeline reliability — the most common cause of AI projects that work in demo but fail in deployment. Integration risks capture auth compatibility, data format mismatches, and API versioning conflicts between the new system and the client's existing stack. Organizational risks reflect human factors: stakeholder alignment, change management readiness, and competing priorities that stall decision-making. Timeline risks flag external dependencies — procurement, regulatory approval, seasonal blackout windows — that no amount of engineering effort can compress.

Severity is a compound score, not a single gut-feel rating. The classifier assigns both a likelihood (very_likely → unlikely) and an implied impact level; the severity matrix combines them. A risk that is very likely but carries only low impact scores medium — not critical. A risk that is only possible but carries critical impact scores high, because low-probability catastrophes demand real mitigation investment. This two-axis structure prevents the common mistake of treating "unlikely" as synonymous with "safe to ignore."

Mitigations as First-Class Scope Artifacts

The RiskMitigation model is designed to make risk responses visible inside the project plan, not hidden in an appendix. By quantifying each mitigation in effort_story_points and timeline_impact_days, the output of risk assessment flows directly into the effort estimation tooling built elsewhere in this chapter — there is no manual hand-off step where mitigations get trimmed when deadline pressure mounts. The owner_role field enforces accountability at authoring time: an action item without an owner is not an action item, it is a hope.

residual_severity closes the loop by making the value of each mitigation explicit. If a proposed control reduces a critical risk to medium, the effort_story_points investment is justified. If residual_severity stays critical after the proposed strategy, the mitigation is insufficient and the project faces a genuine blocker that should be surfaced during scoping — not discovered mid-execution.

Code Walkthrough

Now that you understand the five risk categories and the severity scoring matrix, the next step is turning that taxonomy into runnable code that classifies risks automatically.

The DSPy-based approach uses a declarative Signature to define what the model must produce — category, description, severity, likelihood, and mitigation — without hand-crafting a prompt. A ChainOfThought predictor wraps the signature so the model reasons step-by-step before committing to each output field. This structure means the classifier can later be optimized with labeled examples instead of manual prompt iteration.

Code snippetpython
1import dspy
2
3class RiskClassifier(dspy.Signature):
4    """Classify project risks from specification text."""
5    project_spec: str = dspy.InputField(desc="Project specification and scope")
6    category: str = dspy.OutputField(desc="Risk category: technical|data|integration|organizational|timeline")
7    risk_description: str = dspy.OutputField(desc="Specific risk identified")
8    severity: str = dspy.OutputField(desc="Severity: critical|high|medium|low")
9    likelihood: str = dspy.OutputField(desc="Likelihood: very_likely|likely|possible|unlikely")
10    mitigation: str = dspy.OutputField(desc="Recommended mitigation strategy")
11
12class RiskAssessmentModule(dspy.Module):
13    """Multi-step risk assessment with chain of thought."""
14
15    def __init__(self):
16        self.classifier = dspy.ChainOfThought(RiskClassifier)
17
18    def forward(self, project_spec: str):
19        return self.classifier(project_spec=project_spec)

RiskClassifier extends dspy.Signature to establish the input/output contract: the model receives a project_spec string and must populate five output fields that map directly onto the risk category taxonomy covered in the Concepts section. The docstring acts as the task instruction for the underlying language model. RiskAssessmentModule wraps this signature with ChainOfThought, prompting the model to produce intermediate reasoning before each output — reducing hallucinated severity scores and vague mitigation strings.

Once risks are classified, each finding is wrapped in a structured mitigation record. The RiskMitigation model links a risk identifier to a concrete response plan, capturing effort cost in story points, schedule impact in days, residual severity after action, and the role responsible for execution.

Code snippetpython
1from pydantic import BaseModel
2
3class RiskMitigation(BaseModel):
4    """Mitigation strategy for an identified risk."""
5    risk_id: str
6    strategy: str
7    effort_story_points: int
8    timeline_impact_days: int
9    residual_severity: str
10    owner_role: str

effort_story_points and timeline_impact_days feed directly into the effort estimation tooling built later in this chapter, ensuring that risk mitigations are reflected in the project's final scope document rather than treated as a separate checklist. owner_role prevents orphaned action items by anchoring each mitigation to a specific accountable party — addressing the organizational risk dimension where unassigned action items stall during change management.

Confirm that RiskAssessmentModule().forward(project_spec="...") returns a prediction with non-empty category, severity, and mitigation fields, and that a RiskMitigation instance can be instantiated with those values without raising a Pydantic validation error.

Do's and Don'ts

Having walked through the material above, the following Do's and Don'ts distill it into practice.

Do's

1. ✓Do declare all five output fields (category, severity, likelihood, risk_description, mitigation) on RiskClassifier — the DSPy Signature contract binds the model to produce every field in one forward pass, so a missing output field silently collapses that dimension from the assessment and breaks downstream RiskMitigation instantiation.
2. ✓Do wrap RiskClassifier with dspy.ChainOfThought in RiskAssessmentModule — intermediate reasoning before each output field reduces hallucinated severity scores and vague mitigation strings that a direct dspy.Predict call produces when distinguishing "critical" from "high" on ambiguous specs.
3. ✓Do populate effort_story_points, timeline_impact_days, and owner_role on every RiskMitigation instance — these fields feed the effort estimation tooling built later in this chapter; leaving them as zero or None disconnects risk findings from the final scope document and leaves action items without an accountable party.

Don'ts

1. ✗Don't hand-craft prompt strings instead of using dspy.Signature fields with desc= annotations — bypassing the declarative contract means the classifier can't be optimized later with labeled examples and you lose the structured output guarantees that prevent RiskMitigation from failing Pydantic validation on malformed severity strings.
2. ✗Don't collapse the five risk categories (technical, data, integration, organizational, timeline) into a single freeform label — the category field's constrained vocabulary is what lets the effort estimation layer downstream route mitigations correctly; unconstrained output causes category mismatches that silently exclude organizational or data risks from the scope document.
3. ✗Don't leave owner_role empty or assign it to a team name like "engineering" — RiskMitigation.owner_role exists specifically to address the organizational risk dimension where unassigned action items stall during change management; a role-less mitigation record is indistinguishable from an orphaned checklist item.