Build a code validator with Gemini ToolCodeExecution

Introduction

When an LLM streams Python code into a chat response, you have no guarantee that the snippet actually runs — syntactically valid code still throws ImportError, references undefined names, or hangs on an infinite loop. Teams that ship unverified blocks to learners erode trust the moment a paste-and-run example breaks in front of a user. By the end of this lesson you'll be able to validate extracted Python code blocks using Gemini 2.5 Flash's native ToolCodeExecution sandbox, so every Python block your pipeline emits has been proven runnable server-side before it reaches the client.

Key Terminology

• ToolCodeExecution: The Gemini SDK tool declaration that enables server-side execution of Python code inside an isolated sandbox managed by Google, returning structured ExecutableCode and CodeExecutionResult parts in the model response.
• CodeValidator: The stateless class in processing/code_validator.py that wraps a google.genai.Client, constructs the execution prompt, invokes Gemini with the ToolCodeExecution tool, and parses the response into a ValidationResult.
• ValidationResult: The Pydantic model returned by CodeValidator.validate, carrying a ValidationStatus (SUCCESS, ERROR, or TIMEOUT), captured stdout, an error message, and the exact code that was executed.

Concepts

Practical Considerations for Production Deployments

Rate limiting and batching. Gemini API calls for code execution consume the same quota as standard generation requests. If a single LLM response contains five or more Python blocks, sequential validation creates noticeable latency. Use asyncio.gather with the async variant of the client (await client.aio.models.generate_content(...)) to validate blocks concurrently. The google-genai SDK exposes client.aio for async operations, which maps directly onto the same ToolCodeExecution configuration shown above.

Sandbox limitations. Gemini's code execution sandbox supports standard library modules and a curated set of third-party packages (NumPy, pandas, matplotlib, and others in the scientific Python stack). It does not support network access, file system writes, or arbitrary pip installs. Code blocks that import unsupported modules will return OUTCOME_FAILED. Your pipeline should treat these failures as inconclusive rather than invalid — the code might be perfectly correct but simply untestable in the sandbox.

Error classification. Not all execution errors indicate bad code. Distinguish between syntax errors (which ast.parse from another goal should have already caught), runtime errors (import failures, type errors, assertion failures), and resource errors (timeout, memory exceeded). Map each category to a different UI treatment — syntax errors warrant a red indicator, runtime errors an orange warning, and resource errors a gray "could not verify" badge.

Security boundaries. Although Gemini's sandbox is isolated, the prompt you send contains user-influenced content. Sanitize the code string to remove prompt-injection attempts — for example, text that tries to instruct Gemini to ignore the execution directive and instead return fabricated success output. The temperature=0.0 setting and explicit prompt phrasing reduce but do not eliminate this risk. For high-security deployments, validate that the ExecutableCode part in the response matches the code you submitted, which the _parse_response method already supports by comparing executed against original_code.

Code Walkthrough

Architecture: Where Code Validation Fits

The following diagram shows how a Python code block flows from initial extraction through sandbox validation and back into the content assembly pipeline. Notice that the CodeValidator operates as a synchronous gate — downstream SSE emission waits for the validation verdict before including the code block in the streamed response.

Loading diagram...

This design keeps code execution entirely off your own servers. The CodeValidator class is stateless — it holds a configured google.genai.Client and a model identifier, but no execution state. Each validation call is independent, which means you can run multiple validations concurrently using asyncio.gather when a single LLM response contains several Python blocks.

Building the CodeValidator Class

The CodeValidator class lives in processing/code_validator.py and serves as the single integration point with Gemini's code execution API. It accepts a CodeBlock Pydantic model (produced by the Instructor extraction phase from another goal), constructs a prompt that instructs Gemini to execute the code, and parses the response parts into a ValidationResult model. The class uses google.genai.Client from the google-genai SDK rather than the older google-generativeai package, and it configures ToolCodeExecution as part of the tools parameter in the generate_content call. The constructor accepts an optional timeout parameter that defaults to 30 seconds — long enough for reasonable code but short enough to prevent runaway execution from blocking the pipeline.

Code snippetpython
1from google import genai
2from google.genai import types
3from pydantic import BaseModel, Field
4from enum import Enum
5
6class ValidationStatus(str, Enum):
7    SUCCESS = "success"
8    ERROR = "error"
9    TIMEOUT = "timeout"
10
11class ValidationResult(BaseModel):
12    status: ValidationStatus
13    stdout: str = Field(default="")
14    error_message: str = Field(default="")
15    executed_code: str = Field(default="")
16
17class CodeValidator:
18    def __init__(self, model: str = "gemini-2.5-flash", timeout: int = 30):
19        self.client = genai.Client()
20        self.model = model
21        self.timeout = timeout
22        self.code_execution_tool = types.Tool(
23            code_execution=types.ToolCodeExecution()
24        )
25
26    def validate(self, code: str) -> ValidationResult:
27        fence = "```"
28        prompt = (
29            "Execute the following Python code and report the result. "
30            "Do not modify the code. Run it exactly as provided.\n\n"
31            f"{fence}python\n{code}\n{fence}"
32        )
33        try:
34            response = self.client.models.generate_content(
35                model=self.model,
36                contents=prompt,
37                config=types.GenerateContentConfig(
38                    tools=[self.code_execution_tool],
39                    temperature=0.0,
40                ),
41            )
42            return self._parse_response(response, code)
43        except Exception as exc:
44            return ValidationResult(
45                status=ValidationStatus.TIMEOUT,
46                error_message=str(exc),
47                executed_code=code,
48            )
49
50    def _parse_response(
51        self, response: types.GenerateContentResponse, original_code: str
52    ) -> ValidationResult:
53        stdout_parts = []
54        error_parts = []
55        executed = original_code
56
57        for part in response.candidates[0].content.parts:
58            if hasattr(part, "executable_code") and part.executable_code:
59                executed = part.executable_code.code or executed
60
61            if hasattr(part, "code_execution_result") and part.code_execution_result:
62                result = part.code_execution_result
63                if result.outcome == types.Outcome.OUTCOME_OK:
64                    stdout_parts.append(result.output or "")
65                else:
66                    error_parts.append(result.output or "No details provided")
67
68        if error_parts:
69            return ValidationResult(
70                status=ValidationStatus.ERROR,
71                error_message="\n".join(error_parts),
72                executed_code=executed,
73            )
74        return ValidationResult(
75            status=ValidationStatus.SUCCESS,
76            stdout="\n".join(stdout_parts),
77            executed_code=executed,
78        )

• Imports (lines 1-4): Pull in the google.genai client, the types module for tool declarations, Pydantic's BaseModel/Field for structured output, and Enum for the validation status states.
• ValidationStatus enum (lines 6-9): Three outcomes — SUCCESS when code runs cleanly, ERROR when execution raises an exception, and TIMEOUT when the API call itself fails or exceeds limits.
• ValidationResult model (lines 11-15): Captures everything downstream consumers need: the status, any captured stdout, an error message if execution failed, and the exact code that was executed (useful for audit logging).
• __init__ (lines 18-24): Initializes a genai.Client (which reads GOOGLE_API_KEY or application default credentials from the environment), stores the model identifier, sets the timeout, and pre-builds the ToolCodeExecution tool declaration so it is reused across calls.
• validate prompt construction (lines 27-31): Deliberately explicit — tells Gemini to execute the code without modification. Without this guard, Gemini might "improve" the code or add error handling, defeating the purpose of validating the original snippet.
• validate API call (lines 32-38): Passes code_execution_tool in the config and sets temperature to 0.0 for deterministic execution behavior. The response object contains structured parts that _parse_response interprets.
• validate error guard (lines 39-44): The outer try/except catches network errors, quota exhaustion, and SDK failures, wrapping them in a TIMEOUT status so callers always receive a valid ValidationResult instead of an unhandled exception.
• _parse_response iteration (lines 52-62): Walks the parts in the first candidate's content. When an executable_code part is found, the .code attribute (with an or executed fallback for None) records what Gemini actually ran. When a code_execution_result part is found, the outcome enum routes the output: OUTCOME_OK into stdout_parts, anything else (including OUTCOME_DEADLINE_EXCEEDED and OUTCOME_FAILED) into error_parts.
• _parse_response verdict (lines 64-74): If any error parts were collected, the status is ERROR regardless of whether stdout was also captured. This conservative rule ensures that partially-failing code (e.g., prints output before raising an exception) is still flagged invalid.

Integrating Validation Into the Pipeline

With the CodeValidator built, integration into the broader message processing pipeline follows a straightforward pattern. After Instructor extracts CodeBlock models from the LLM response and the markdown parser confirms syntax validity via ast.parse, each Python block passes through CodeValidator.validate. The validation result attaches to the block before it enters the SSE streaming assembly phase. The following snippet shows how the orchestration layer calls the validator and annotates the block, using the ValidationResult to set a validated flag and optionally attach execution output to the content model that gets serialized into the SSE event stream.

1from processing.code_validator import CodeValidator, ValidationStatus
2
3def process_code_blocks(blocks: list, validator: CodeValidator) -> list:
4    results = []
5    for block in blocks:
6        if block.language != "python":
7            block.validated = False
8            block.validation_note = "non-python: skipped"
9            results.append(block)
10            continue
11
12        vr = validator.validate(block.source)
13        block.executed_code = vr.executed_code
14        block.validated = vr.status == ValidationStatus.SUCCESS
15        block.validation_note = (
16            vr.stdout if vr.status == ValidationStatus.SUCCESS
17            else f"FAILED: {vr.error_message}"
18        )
19        results.append(block)
20    return results

Do's and Don'ts

Do's

1. ✓Do configure ToolCodeExecution as a types.Tool object and pass it through the tools parameter inside GenerateContentConfig — omitting either the types.Tool wrapper or the config argument causes Gemini to ignore the tool entirely and return a plain text response instead of executing the code, so the _parse_response loop never finds a code_execution_result part.
2. ✓Do iterate over response.candidates[0].content.parts and check both the executable_code and code_execution_result attributes on each part — Gemini interleaves these part types in a single response, and skipping the executable_code check means executed_code silently falls back to the original string, hiding any pre-execution normalization Gemini applied before the OUTCOME_OK or OUTCOME_ERROR verdict.
3. ✓Do keep CodeValidator stateless — only a shared google.genai.Client and model identifier, no per-call execution state — statelessness is what makes it safe to fan out concurrent validations with asyncio.gather when a single LLM response contains multiple Python blocks, bounding SSE emission latency to the slowest block rather than the cumulative sum.

Don'ts

1. ✗Don't import from the google-generativeai package; use from google import genai (the google-genai SDK) — types.ToolCodeExecution, types.Outcome, and types.GenerateContentConfig live only in the newer SDK's namespace; the older package lacks these types entirely, causing AttributeError at the generate_content call site with no fallback behavior.
2. ✗Don't set temperature above 0.0 in GenerateContentConfig when invoking the code execution tool — any non-zero temperature gives Gemini latitude to rewrite the snippet before executing it, so an OUTCOME_OK verdict reflects a modified version of the code rather than the original extracted block, breaking the pipeline's guarantee that only the emitted-as-extracted code has been proven runnable.
3. ✗Don't omit the timeout parameter or leave exception handling to the caller — a Python block with an infinite loop or blocking I/O will stall CodeValidator.validate indefinitely, freezing the synchronous SSE gate and preventing any downstream content from reaching the client; the 30-second default and the broad except Exception catch that maps to ValidationStatus.TIMEOUT are what make the validator safe to insert as a blocking pipeline stage.