Generate training quizzes with DSPy-optimized prompts

Introduction

Engineers often spend hours manually writing quiz questions from runbooks and technical documentation — a slow, inconsistent process that rarely keeps pace with how fast that content changes. When a product update ships, the training material is fresh but the assessment lags behind, and customer engineering teams end up tested on outdated concepts. By the end of this lesson, you'll be able to build an automated quiz generator that uses DSPy to produce structured, multi-difficulty questions with plausible distractors directly from your team's documentation.

Key Terminology

• DSPy Signature — A Python class that inherits from dspy.Signature and declares the typed input/output contract an LLM call must satisfy; in this lesson, QuestionGenerator maps documentation, topic, and difficulty inputs to a question, correct_answer, distractors, and explanation.
• dspy.InputField / dspy.OutputField — Field descriptors attached to a Signature class that label each slot as either data the caller provides or data the LLM must produce; the desc string on each field guides the model on what that slot represents and how to fill it.
• dspy.ChainOfThought — A DSPy module that wraps a Signature and instructs the LLM to reason step-by-step before committing to each output field, producing more consistent answers than a direct single-shot prompt would.
• Distractor — A plausible-but-incorrect answer option in a multiple-choice question; effective distractors represent misconceptions a partially-informed engineer might genuinely hold, and they are the primary signal of whether a quiz measures real understanding or just reading speed.
• Difficulty progression — The ordered sequence of "easy", "medium", and "hard" values passed as an input to QuestionGenerator, driving build_quiz to produce one question per tier so a quiz covers a concept at multiple depths.

Concepts

Signatures as a Typed Contract for LLM Outputs

A DSPy Signature is not a prompt string — it is a class declaration that separates what an LLM call must produce from how it produces it. By listing input and output fields with descriptive labels, you give DSPy enough information to construct, validate, and swap prompt strategies without you rewriting them. This is the core architectural shift the lesson makes: instead of crafting a bespoke prompt for a quiz generator and then hand-parsing its response, you declare a contract in code (QuestionGenerator) and let DSPy enforce that contract at every call site.

The practical benefit for training automation is reproducibility. Every call through QuestionGenerator returns the same four output slots regardless of the underlying model or prompt variant. Downstream code like build_quiz can confidently access result.question and result.correct_answer without defensive string parsing, and the structure serializes cleanly to dicts that a front-end renderer can consume directly (see Code Walkthrough).

Why Chain-of-Thought Improves Distractor Quality

Generating a plausible wrong answer is harder than generating a correct one. A good distractor must be related to the topic, sound credible to a partially-informed reader, and be definitively incorrect — three constraints that pull in different directions. A direct prompt that asks for all four output fields simultaneously tends to produce distractors that are either too obviously wrong (trivially rejected) or subtly restate the correct answer with minor wording changes.

dspy.ChainOfThought addresses this by inserting a reasoning step before each output field is filled. The model articulates why an answer is correct before committing to the correct_answer field, and that reasoning context is in scope when the distractors field is generated next. The result is wrong answers that are wrong for a specific, articulable reason — the kind of wrong that tests understanding rather than reading speed.

Topic Specificity as the Key Quality Lever

The topic input field is the most important dial for distractor quality. A broad topic like "Kubernetes" gives the model too much freedom — distractors can wander into unrelated sub-concepts and end up incoherent. A narrow topic like "Kubernetes pod scheduling preemption" constrains the space so that both the correct answer and every distractor must stay within that concept boundary, making each wrong option feel genuinely plausible to someone who almost understands the mechanism.

build_quiz passes the same topic string across all three difficulty levels, which keeps the question set coherent: easy, medium, and hard questions probe the same concept from different angles rather than drifting into unrelated territory. When the returned distractors feel vague or obviously wrong, narrowing the topic string is the first fix to try — before adjusting the model, the Signature, or the documentation chunk.

Code Walkthrough

Now that you understand how DSPy Signatures define structured contracts for LLM outputs, the next step is putting that structure to work in a quiz generator. The QuestionGenerator signature below formalizes the contract between a documentation input and the four outputs every quiz question needs: a question, a correct answer, a JSON list of distractors, and an explanation.

Code snippetpython
1import dspy
2
3class QuestionGenerator(dspy.Signature):
4    """Generate a quiz question from documentation."""
5
6    documentation: str = dspy.InputField(
7        desc="Technical documentation content"
8    )
9    topic: str = dspy.InputField(
10        desc="Specific topic to test"
11    )
12    difficulty: str = dspy.InputField(
13        desc="easy, medium, or hard"
14    )
15    question: str = dspy.OutputField(
16        desc="Clear question text"
17    )
18    correct_answer: str = dspy.OutputField(
19        desc="The correct answer"
20    )
21    distractors: str = dspy.OutputField(
22        desc="Three plausible wrong answers as JSON list"
23    )
24    explanation: str = dspy.OutputField(
25        desc="Why the correct answer is right"
26    )

With the signature in place, dspy.ChainOfThought wraps it into a callable module that reasons step-by-step before committing to each output field. The builder below drives that module across an easy-to-hard difficulty progression and collects results as plain dicts, making them straightforward to serialize or pass to a front-end renderer.

Code snippetpython
1import json
2import dspy
3
4def build_quiz(docs: list[str], topic: str) -> list[dict]:
5    """Generate one question per difficulty level from documentation."""
6    difficulties = ["easy", "medium", "hard"]
7    generator = dspy.ChainOfThought(QuestionGenerator)
8    questions = []
9    for i, difficulty in enumerate(difficulties):
10        result = generator(
11            documentation=docs[i % len(docs)],
12            topic=topic,
13            difficulty=difficulty,
14        )
15        distractors = json.loads(result.distractors)
16        questions.append({
17            "question": result.question,
18            "correct": result.correct_answer,
19            "options": [result.correct_answer] + distractors,
20            "explanation": result.explanation,
21            "difficulty": difficulty,
22        })
23    return questions

Each call to generator(...) sends a documentation chunk, the topic, and a difficulty level to the LLM. DSPy's ChainOfThought module instructs the model to reason before filling each output field, which produces more consistent distractor quality than a direct prompt would. The distractors field comes back as a JSON string; json.loads converts it to a Python list before the question dict is assembled. The options list places the correct answer first — your rendering layer should shuffle positions before displaying to learners.

Distractor quality determines whether a quiz measures genuine understanding or just reading speed. A well-formed distractor represents a misconception a partially-informed engineer might genuinely hold. If the LLM returns distractors that are obviously wrong or unrelated to the topic, narrow the topic input string and re-run — more specific topics produce tighter, more plausible wrong answers.

Verify by calling build_quiz(["Your documentation text here"], topic="Kubernetes pod scheduling") and confirming that each dict in the returned list contains a non-empty question, a correct answer, a four-item options list, and a non-empty explanation.

Do's and Don'ts

Having walked through the material above, the following Do's and Don'ts distill it into practice.

Do's

1. ✓Do use dspy.ChainOfThought to wrap QuestionGenerator — ChainOfThought instructs the model to reason before committing to each output field, which produces more consistent distractor plausibility than dspy.Predict or a raw prompt call would; skipping it degrades wrong-answer quality across the easy-to-hard progression.
2. ✓Do keep the topic input string narrow and specific — build_quiz feeds topic directly into every generator(...) call, and a broad topic (e.g., "Kubernetes") yields loosely related distractors that test vocabulary rather than understanding; a tight topic (e.g., "Kubernetes pod scheduling") forces the model to construct misconceptions an informed engineer could plausibly hold.
3. ✓Do shuffle options in your rendering layer before displaying questions to learners — build_quiz always places correct_answer at index 0 of the options list, so un-shuffled output leaks the answer by position regardless of how well-formed the distractors are.

Don'ts

1. ✗Don't treat result.distractors as a ready-to-use Python list without calling json.loads — DSPy returns the distractors output field as a raw JSON string; concatenating it directly into options embeds a bracket-laden string as a single list element, silently breaking the three-distractor structure that QuestionGenerator's output contract promises.
2. ✗Don't reuse the same documentation chunk for every difficulty level if you have multiple docs — build_quiz uses docs[i % len(docs)] to rotate through available chunks; passing a single-item list works but feeds identical context to easy, medium, and hard calls, making the model's only lever for difficulty the word "hard" rather than genuinely deeper source material.
3. ✗Don't define QuestionGenerator output fields without desc strings — DSPy uses field descriptions to shape what the model writes into each slot; omitting desc on distractors in particular removes the "Three plausible wrong answers as JSON list" instruction that tells the model both the count and the serialization format, causing unpredictable output shapes that break json.loads.