Build automated maintenance workflows with Argo

Introduction

When you let graph maintenance run manually, tasks get skipped during busy periods and human error silently corrupts production data. As the knowledge graph grows, snapshot creation, orphan cleanup, validation, and metric collection multiply — and the moment one is missed, you discover the lapse weeks later through a downstream query failure. Argo Workflows turns these scattered maintenance jobs into a single scheduled DAG that runs on a cron, retries transient failures, and surfaces persistent problems before they cascade. By the end of this lesson you'll be able to define a CronWorkflow manifest that chains maintenance tasks in dependency order, set retry policies and timeouts per step, and deploy the pipeline so it runs unattended every night.

Key Terminology

• CronWorkflow — an Argo resource that runs a Workflow on a cron schedule; it's how you turn an ad-hoc maintenance DAG into a recurring job without writing your own scheduler.
• DAG template — Argo's directed-acyclic-graph workflow type where each task declares which tasks it depends on; dependency edges determine execution order so that snapshots run before cleanup runs before metrics.
• Retry strategy — per-task policy (retryStrategy.limit) that controls how many times Argo re-runs a failed step before declaring it failed; essential for tolerating transient image-pull or network errors in maintenance jobs.
• activeDeadlineSeconds — per-task timeout that kills a step if it runs too long; protects the pipeline from a stuck cleanup job blocking the rest of the maintenance schedule.

Concepts

The maintenance pipeline rests on three ideas: declarative scheduling, dependency-ordered execution, and per-step resilience. Together they replace ad-hoc maintenance scripts with a workflow that runs predictably under Kubernetes.

Declarative scheduling with CronWorkflow

Argo's CronWorkflow resource pairs a cron expression with a workflowSpec. The Argo controller watches CronWorkflow objects and submits a fresh Workflow instance each time the schedule fires. There's no external scheduler, no cron container, no glue script — the cluster itself runs the maintenance jobs. A single YAML manifest replaces the brittle "someone configured a cron on the bastion host" pattern (see Code Walkthrough).

Dependency-ordered DAGs

Maintenance tasks have natural ordering: snapshot before validation, validation before cleanup, cleanup before metrics. The DAG template encodes that ordering by listing each task's dependencies. Argo computes the partial order and runs tasks in parallel when their dependencies are satisfied, sequentially when they're not. Wrong ordering is the most common maintenance bug — cleanup running before snapshot means a corrupted graph has no recovery point.

Per-step resilience

Each task carries its own retryStrategy.limit and activeDeadlineSeconds. Retries absorb transient failures — an image pull that times out, a brief network blip while contacting the graph store. Deadlines bound the blast radius of a stuck task: if cleanup hangs, it dies after ten minutes instead of blocking the 3 a.m. metric push for hours. Resilience is per-step because the right values differ — snapshot may tolerate one retry, metrics collection three.

Code Walkthrough

The snippet below combines the three concepts from the previous section — declarative scheduling, the dependency DAG, and per-step retries — into a single builder that emits a CronWorkflow manifest.

Code snippetpython
1from dataclasses import dataclass, field
2
3@dataclass
4class MaintenanceTask:
5    name: str
6    image: str
7    command: list[str]
8    dependencies: list[str] = field(default_factory=list)
9    retry_limit: int = 2
10    timeout_seconds: int = 600
11
12class MaintenanceWorkflowBuilder:
13    def __init__(self, namespace: str, schedule: str):
14        self.namespace = namespace
15        self.schedule = schedule
16        self.tasks: list[MaintenanceTask] = []
17
18    def add_task(self, task: MaintenanceTask):
19        self.tasks.append(task)
20        return self
21
22    def build_cron_workflow(self) -> dict:
23        templates = []
24        dag_tasks = []
25        for task in self.tasks:
26            templates.append({
27                "name": task.name,
28                "container": {
29                    "image": task.image,
30                    "command": task.command,
31                },
32                "retryStrategy": {"limit": task.retry_limit},
33                "activeDeadlineSeconds": task.timeout_seconds,
34            })
35            dag_tasks.append({
36                "name": task.name,
37                "template": task.name,
38                "dependencies": task.dependencies,
39            })
40
41        return {
42            "apiVersion": "argoproj.io/v1alpha1",
43            "kind": "CronWorkflow",
44            "metadata": {
45                "name": "graph-maintenance",
46                "namespace": self.namespace,
47            },
48            "spec": {
49                "schedule": self.schedule,
50                "workflowSpec": {
51                    "entrypoint": "maintenance-dag",
52                    "templates": [
53                        {
54                            "name": "maintenance-dag",
55                            "dag": {"tasks": dag_tasks},
56                        },
57                        *templates,
58                    ],
59                },
60            },
61        }
62
63def create_maintenance_pipeline(namespace: str) -> dict:
64    builder = MaintenanceWorkflowBuilder(namespace, "0 2 * * *")
65    builder.add_task(MaintenanceTask(
66        name="create-snapshot",
67        image="graph-tools:latest",
68        command=["python", "-m", "maintenance.snapshot"],
69    ))
70    builder.add_task(MaintenanceTask(
71        name="run-validation",
72        image="graph-tools:latest",
73        command=["python", "-m", "maintenance.validate"],
74        dependencies=["create-snapshot"],
75    ))
76    builder.add_task(MaintenanceTask(
77        name="cleanup-orphans",
78        image="graph-tools:latest",
79        command=["python", "-m", "maintenance.cleanup"],
80        dependencies=["run-validation"],
81    ))
82    builder.add_task(MaintenanceTask(
83        name="collect-metrics",
84        image="graph-tools:latest",
85        command=["python", "-m", "maintenance.metrics"],
86        dependencies=["cleanup-orphans"],
87    ))
88    return builder.build_cron_workflow()

• MaintenanceTask carries the four per-step knobs that matter — image, command, dependencies, retry limit, and timeout. Defaults of two retries and a ten-minute deadline cover most maintenance operations.
• MaintenanceWorkflowBuilder.add_task appends tasks and returns self for fluent chaining; build_cron_workflow emits both DAG entries (which encode dependencies) and templates (which encode container, retries, and timeout) — the two halves of an Argo workflow spec.
• create_maintenance_pipeline wires snapshot → validation → cleanup → metrics. Snapshot runs first as a recovery point; validation runs against the pre-cleanup state; metrics run last so they capture the post-maintenance graph.

You'll know it works when kubectl apply -f <generated-manifest>.yaml registers the CronWorkflow, kubectl get cronworkflow graph-maintenance shows the next scheduled time, and at 02:00 a child Workflow appears in kubectl get workflows running each step in dependency order.

Do's and Don'ts

Now that you have worked through the implementation, the practices below separate a durable approach from a fragile one.

Do's

1. ✓Set per-task retry limits and timeouts — transient errors are common in maintenance jobs, and a stuck step should not block the rest of the DAG.
2. ✓Order tasks by dependency, not by convenience — snapshot before any destructive task so you always have a recovery point.
3. ✓Use a CronWorkflow instead of an external scheduler — keeping the schedule in the cluster eliminates a separate piece of infrastructure to monitor.

Don'ts

1. ✗Don't run cleanup before validation — you lose the ability to correlate validation failures with the pre-cleanup graph state.
2. ✗Don't omit activeDeadlineSeconds — a hung step with no deadline can block every subsequent maintenance run.
3. ✗Don't share one retry limit across all tasks — snapshot and metrics tolerate different retry counts, and one global number is always wrong somewhere.