Deploy FastAPI to Cloud Run with auto-scaling

Introduction

When you ship a FastAPI GenAI service to production, the choice of compute target shapes your cost curve, latency profile, and on-call burden for the lifetime of the application. Teams that default to GKE for stateless inference workloads routinely overpay by 3x and inherit cluster-management toil they do not need; teams that move to Cloud Run without tuning concurrency for LLM latency suffer OOM kills and runaway scaling bills. By the end of this lesson you will be able to deploy a FastAPI application to Cloud Run with a Cloud Build CI/CD pipeline, configure a Serverless VPC Access connector for private backend access, and tune concurrency, memory, and minimum-instance settings to balance cold-start latency against scale-to-zero economics.

Key Terminology

• Cloud Run: Google Cloud's fully managed serverless container runtime that auto-scales stateless HTTP services from zero to many instances based on request load.
• Serverless VPC Access connector: A managed resource that lets Cloud Run instances reach private VPC resources (Cloud SQL, Memorystore, internal services) without exposing them to the public internet.
• Concurrency: The maximum number of in-flight requests Cloud Run routes to a single container instance before spinning up another — the primary tuning knob for LLM-bound FastAPI workloads.
• Cold start: The latency penalty incurred when Cloud Run starts a new container instance to handle a request, dominated by container init plus dependency and model client loading.
• Cloud Build: Google Cloud's managed CI/CD service that builds container images, pushes them to Artifact Registry, and deploys revisions to Cloud Run via declarative build steps.
• Revision: An immutable snapshot of a Cloud Run service's container image and configuration; traffic is split across revisions to enable canary rollouts and instant rollback.

Concepts

Scale-to-Zero Economics and Concurrency Tuning

Scale-to-zero is Cloud Run's defining economic advantage over GKE for GenAI workloads with variable traffic. When no requests arrive, Cloud Run terminates all instances and billing drops to zero. However, the cold-start penalty—container initialization plus model/dependency loading—introduces latency that you must quantify and mitigate.

For a FastAPI GenAI application with typical dependencies (LangChain, Anthropic SDK, SQLAlchemy), cold-start adds 3-8 seconds to the first request. Three strategies reduce this impact:

• Minimum instances: Setting --min-instances 1 keeps one warm instance permanently, costing approximately $0.0864/hour for a 2-vCPU instance. For production services, this $62/month eliminates cold starts for the first burst of requests up to your concurrency limit.
• CPU allocation during startup: The --cpu-boost flag doubles CPU allocation during container startup, reducing initialization time by 30-50%. This is true cost optimization—the boosted CPU billing lasts only seconds while it halves perceived latency.
• Lazy initialization: Defer heavy imports and client instantiation to the first request using FastAPI's lifespan events rather than module-level initialization. This moves work from container startup (which blocks the readiness check) to first-request handling (which Cloud Run reports as normal latency).

The concurrency setting deserves careful analysis. Cloud Run's default of 80 concurrent requests per instance assumes fast I/O-bound handlers. For GenAI applications, each request occupies an async coroutine for 2-15 seconds while waiting for an LLM API response. Setting concurrency to 40 means each instance handles 40 simultaneous LLM-bound requests. If each request consumes approximately 50MB of memory (prompt context, response buffers, intermediate chain state), 40 concurrent requests require 2GB—exactly matching the --memory 2Gi setting in the deployment configuration shown in the Code Walkthrough section below. Setting concurrency too high causes OOM kills; setting it too low wastes CPU cycles and triggers unnecessary scaling.

The cost comparison against GKE Autopilot is decisive for most workloads. A Cloud Run service handling 100,000 requests/day with an average latency of 5 seconds costs approximately $45/month (2 vCPU, 2 GiB, billed per-request-second). An equivalent GKE Autopilot deployment running a minimum of 2 pods continuously costs approximately $140/month regardless of traffic. The breakeven point where GKE becomes cheaper is approximately 500,000 requests/day with sustained concurrency above 30—at which point you are also likely hitting Cloud Run's 1000-instance limit and should migrate to GKE for other reasons.

Traffic Migration and Canary Deployment

The deployment configuration shown in the Code Walkthrough section below uses --no-traffic deliberately. After deploying a canary-tagged revision, you validate it before migrating traffic. The canary revision is accessible at https://canary---SERVICE-HASH.a.run.app while production traffic continues hitting the previous revision. Once health checks pass and you confirm the canary handles test requests correctly, you migrate traffic incrementally:

1. Route 5% of traffic to canary, monitor error rates for 10 minutes
2. Increase to 25%, monitor p99 latency against baseline
3. Promote to 100% if no regressions appear

This graduated rollout is critical for GenAI services because LLM prompt template changes, updated system prompts, or new tool definitions can cause subtle behavioral regressions that unit tests miss. A 5% canary catches these issues with real user traffic before they affect the full user base.

When integrating with Cloud Build triggers, connect the trigger to your repository's main branch push event. The trigger executes the build configuration, which deploys the --no-traffic canary. A separate Cloud Function or Cloud Workflow then runs integration tests against the canary URL and, on success, executes gcloud run services update-traffic to promote the revision. This separation ensures that the build pipeline never directly routes untested code to production—a guardrail that prevents the most common class of GenAI deployment incidents: shipping a broken prompt template that passes CI but fails on real inputs.

Code Walkthrough

Cloud Run Architecture for GenAI Services

Before writing any deployment configuration, you must understand how Cloud Run's concurrency model interacts with LLM inference latency. A typical REST API handles requests in single-digit milliseconds, making a concurrency setting of 80-250 reasonable per container instance. GenAI applications calling hosted LLM APIs experience 1-15 second response times, which means each concurrent request holds a thread (or coroutine) for orders of magnitude longer. This fundamentally changes your scaling arithmetic.

Loading diagram...

This diagram illustrates the production topology. Cloud Run instances connect to private VPC resources through a Serverless VPC Access connector, keeping all LLM API traffic, database queries, and cache operations off the public internet. Secret Manager injects API keys at container startup without baking them into images. Artifact Registry stores versioned container images that Cloud Build produces.

Cloud Build CI/CD Pipeline Configuration

The CI/CD pipeline must accomplish three tasks atomically: build the container image with layer caching from the previous build, push it to Artifact Registry, and deploy a new Cloud Run revision with traffic migration. The following Python script defines a programmatic Cloud Build configuration using the google.cloud.devtools.cloudbuild_v1 client library. Rather than maintaining a static cloudbuild.yaml, generating the configuration programmatically lets you inject environment-specific values—like the VPC connector name, minimum instance count, and concurrency limits—from your deployment orchestration layer. The function build_cloudbuild_config constructs the three-step pipeline, while submit_build handles asynchronous submission and polling. Pay particular attention to the --cache-from flag in the build step, which pulls the previous image to reuse cached layers for dependency installation—saving 2-4 minutes on builds where only application code changed.

1from google.cloud.devtools import cloudbuild_v1
2from google.protobuf import duration_pb2
3import os
4
5def build_cloudbuild_config(
6    project_id: str,
7    region: str = "us-central1",
8    service_name: str = "genai-api",
9    vpc_connector: str = "genai-connector",
10    max_instances: int = 10,
11    concurrency: int = 40,
12) -> cloudbuild_v1.Build:
13    image_uri = f"{region}-docker.pkg.dev/{project_id}/genai/{service_name}:latest"
14    
15    build_step = cloudbuild_v1.BuildStep(
16        name="gcr.io/cloud-builders/docker",
17        args=[
18            "build",
19            "--cache-from", image_uri,
20            "-t", image_uri,
21            "-f", "Dockerfile.prod",
22            "--build-arg", "PYTHON_VERSION=3.12",
23            "--build-arg", f"BUILD_HASH=$SHORT_SHA",
24            ".",
25        ],
26        id="build-image",
27    )
28    
29    push_step = cloudbuild_v1.BuildStep(
30        name="gcr.io/cloud-builders/docker",
31        args=["push", image_uri],
32        id="push-image",
33        wait_for=["build-image"],
34    )
35    
36    deploy_args = [
37        "run", "deploy", service_name,
38        "--image", image_uri,
39        "--region", region,
40        "--platform", "managed",
41        "--concurrency", str(concurrency),
42        "--max-instances", str(max_instances),
43        "--min-instances", "0",
44        "--cpu", "2",
45        "--memory", "2Gi",
46        "--timeout", "300s",
47        "--vpc-connector", vpc_connector,
48        "--vpc-connector-egress-settings", "private-ranges-only",
49        "--set-secrets", "ANTHROPIC_API_KEY=anthropic-key:latest",
50        "--no-traffic",
51        "--tag", "canary",
52    ]
53    
54    deploy_step = cloudbuild_v1.BuildStep(
55        name="gcr.io/cloud-builders/gcloud",
56        args=deploy_args,
57        id="deploy-revision",
58        wait_for=["push-image"],
59    )
60    
61    return cloudbuild_v1.Build(
62        steps=[build_step, push_step, deploy_step],
63        timeout=duration_pb2.Duration(seconds=1200),
64        options=cloudbuild_v1.BuildOptions(
65            logging=cloudbuild_v1.BuildOptions.LoggingMode.CLOUD_LOGGING_ONLY,
66            machine_type=cloudbuild_v1.BuildOptions.MachineType.E2_HIGHCPU_8,
67        ),
68    )

VPC Connector and Private Network Configuration

Cloud Run services run outside your VPC by default. Without a Serverless VPC Access connector, every call to Cloud SQL, Memorystore, or a private Vertex AI endpoint would require public IP exposure—violating security baselines at any serious organization. The connector creates a subnet of /28 (16 IPs) that Cloud Run instances use for egress into your VPC. The following configuration function uses the google.cloud.vpcaccess_v1 client to programmatically create the connector with appropriate throughput settings. The create_vpc_connector function accepts an ip_range parameter for the connector subnet and a max_throughput value in Mbps. In production GenAI workloads, throughput sizing matters because LLM responses containing long completions can generate sustained egress bursts of 10-50 Mbps across all concurrent instances.

1from google.cloud import vpcaccess_v1
2from google.api_core import operation as gac_operation
3import logging
4
5logger = logging.getLogger(__name__)
6
7def create_vpc_connector(
8    project_id: str,
9    region: str,
10    connector_name: str = "genai-connector",
11    network: str = "default",
12    ip_range: str = "10.8.0.0/28",
13    min_throughput: int = 200,
14    max_throughput: int = 1000,
15) -> vpcaccess_v1.Connector:
16    client = vpcaccess_v1.VpcAccessServiceClient()
17    parent = f"projects/{project_id}/locations/{region}"
18    
19    connector = vpcaccess_v1.Connector(
20        name=f"{parent}/connectors/{connector_name}",
21        network=network,
22        ip_cidr_range=ip_range,
23        min_throughput=min_throughput,
24        max_throughput=max_throughput,
25    )
26    
27    try:
28        operation = client.create_connector(
29            parent=parent,
30            connector_id=connector_name,
31            connector=connector,
32        )
33        result = operation.result(timeout=300)
34        logger.info(
35            "Connector %s created: state=%s, throughput=%d-%d Mbps",
36            result.name, result.state.name,
37            result.min_throughput, result.max_throughput,
38        )
39        return result
40    except Exception as exc:
41        if "already exists" in str(exc):
42            logger.warning("Connector %s already exists, fetching.", connector_name)
43            return client.get_connector(
44                name=f"{parent}/connectors/{connector_name}"
45            )
46        raise

Do's and Don'ts

Do's

1. ✓Do set concurrency to 40 (or lower) for FastAPI services that call hosted LLM APIs — GenAI requests hold a thread or coroutine for 1–15 seconds, not single-digit milliseconds, so the default Cloud Run concurrency of 80–250 causes runaway horizontal scaling and memory exhaustion under real inference latency; the lesson's build_cloudbuild_config defaults to 40 for exactly this reason.
2. ✓Do pass --cache-from <image_uri> in the Cloud Build build_step pointing at the previous Artifact Registry image — this reuses cached dependency layers and cuts 2–4 minutes from builds where only application code (not requirements.txt) changed, which is the common case in a fast-moving GenAI service.
3. ✓Do set --vpc-connector-egress-settings private-ranges-only when attaching a Serverless VPC Access connector — this routes only traffic destined for private CIDR ranges (Redis Memorystore, Cloud SQL, Vertex AI Private Service Connect) through the connector while letting Secret Manager and Artifact Registry calls continue over Google's public APIs, avoiding unnecessary connector bandwidth charges.

Don'ts

1. ✗Don't shift traffic to a new Cloud Run revision immediately by omitting --no-traffic — the lesson's deploy_args deploys with --no-traffic and --tag canary so the revision can be validated before any live requests reach it; skipping this means a misconfigured concurrency setting or a broken secret reference instantly affects 100% of production traffic with no rollback window.
2. ✗Don't inject API keys via --build-arg or bake them into the Dockerfile.prod — the lesson uses --set-secrets ANTHROPIC_API_KEY=anthropic-key:latest to pull from Secret Manager at container startup; embedding secrets in image layers means they persist in Artifact Registry history indefinitely and are readable by anyone with registry pull access.
3. ✗Don't default to GKE for stateless FastAPI GenAI inference services — teams that do so routinely overpay by 3x and inherit cluster-management toil that Cloud Run's scale-to-zero with tuned min-instances, concurrency, and --memory 2Gi settings already handles; GKE is reserved for workloads with stateful, GPU, or networking requirements that Cloud Run's managed platform cannot satisfy.